Slashdot Mirror
Enraged Verizon FiOS Customer Seemingly Demonstrates Netflix Throttling
MojoKid (1002251) writes The ongoing battle between Netflix and ISPs that can't seem to handle the streaming video service's traffic, boiled over to an infuriating level for Colin Nederkoon, a startup CEO who resides in New York City. Rather than accept excuses and finger pointing from either side, Nederkoon did a little investigating into why he was receiving such slow Netflix streams on his Verizon FiOS connection. What he discovered is that there appears to be a clear culprit. Nederkoon pays for Internet service that promises 75Mbps downstream and 35Mbps upstream through his FiOS connection. However, his Netflix video streams were limping along at just 375kbps (0.375mbps), equivalent to 0.5 percent of the speed he's paying for. On a hunch, he decided to connect to a VPN service, which in theory should actually make things slower since it's adding extra hops. Speeds didn't get slower, they got much faster. After connecting to VyprVPN, his Netflix connection suddenly jumped to 3000kbps, the fastest the streaming service allows and around 10 times faster than when connecting directly with Verizon. Verizon may have a different explanation as to why Nederkoon's Netflix streams suddenly sped up, but in the meantime, it would appear that throttling shenanigans are taking place. It seems that by using a VPN, Verizon simply doesn't know which packets to throttle, hence the gross disparity in speed.
Now they'll just throttle VPN traffic too.
It is also possible the the VPN packets are transiting a different upstream peer from Verizon and bypassing the peering bottleneck at issue. Assuming that Verizon is performing inspection of packets and throttling only Netflix packets is quite a leap.
Routing traffic via the VPN changes the path the traffic flows over, possibly avoiding routes that are saturated and (who knows) pending upgrade.
It's tempting to imagine the internet as a giant blob of fungible bandwidth, but in reality it's just a big mess of cables some of which are higher capacity than others. Assuming malice is fun, but there isn't enough data here to say one way or another.
Without enough transit from the backbone providers and without direct peering, there's going to be congestion. Providers want to get paid by both sides. Unfortunately Netflix has foolishly given in once. Now everybody wants a piece of the pie. Anyone who has a lot of data that users want should refuse to pay for peering with anyone who has the users who want the data, and vice versa. If they can't reach a peering agreement, both need to buy sufficient transit from third parties, so there's the incentive.
You can also see this between FIOS and VerizonWireless 4G--Fios is slower for Netflix connections. So they don't throttle uniformly.
I have to wonder, what would happen if customers were to start throttling the payment of ISP's?
"You will get your payment when you actually fulfil your end of our contract, but not before."
Couldn't the same data be interpreted as Netflix penalises Verizon IPs, and the vpn connection could not be identified by Netflix as a Verizon IP on the server side.
Both Verizon(http://publicpolicy.verizon.com/blog/entry/why-is-netflix-buffering-dispelling-the-congestion-myth) and Level3(blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/) have documented that their peering links are flooded. Verizon wants paid peering. Level3 wants unpaid peering.
Obviously no amount of encryption is going to speed up a flooded link. So obviously the traffic through the VPN is taking a different path.
This is a dupe of Verizon's Accidental Mea Culpa.
It's based on the same blog post by Colin Nederkoon. The explanation in the earlier story seemed much more convincing.
Verizon is too big, and our government does not care.
The only answer is to actively work to destroy Verizon until they acquiesce or no longer exist.
This is VERY stupid for Verizon. For a company that advertises a great network, and Fios as a premier very high speed internet connection.
Throttling Netflix is acceptable for a cheap DSL connection, from ATT. When someone chooses to sign up for Fios, they are choosing to pay lots of money for a premium internet connection, and EVERYTHING the internet has to offer, including Netflix. Throttling Netflix devalues the Fios brand. Very stupid from a marketing standpoint.
vpn throttling, here we come
My ism, it's full of beliefs.
Might you consider a civil action instead?
Verizon isn't throttling packets, but rather, it is deliberately limiting the number of channels (and therefore bandwidth) at the routers that interconnect between Verizon's network and ISP that serve Netflix such as Level 3 Communications. That way they can say they are not "throttling", but they are accomplishing the same thing. There is a nice explanation of this by the CEO of Level 3 floating around on the net somewhere. Google it.
Verizon doesn't care. They own RedBox Instant; they last thing they want is customers using Netflix. We're not gonna get net neutrality out of the FCC (the public comments are a sham; the FCC only care about the businesses involved in the decision); so this is not going to get fixed. If Netflix uses Level3; they were cripple all level3 connectivity.
You win the most absurd analogy ever award.
I recently upgraded my FIOS service and they used Netflix streaming as one of the reasons that I should do it. After going from 25/5 to 50/25 I still get downgraded quality when watching flix.
The difference between Canada and the USA is that in Canada healthcare is a right and gun ownership is a privilege.
The netflix cdn box is worth tens of thousands depending on spec and the isp has to set aside around 8tb for sync. This had in places reduced netflix traffic beyond the isp to below 10% in some cases. PROVIDERS can't have it both ways. Capping fibre to 300k is beyond being cuntish, round here you'd get stabbed for cheating someone as hard
You win the most absurd analogy ever award.
Hey, nobody told me this was a contest ...
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
375k!?
No, I get your point... just...
First off, I assert that whether Verizon is actively throttling packets, or simply not providing sufficient peering to get to Netflix, they are committing fraud by advertising high speeds and not delivering them.
However, to *really* convince people, more rigorous experiment has to be performed: find a VPN (or set up your own with a colo) that's connected as closely to Verizon as possible, as close to their peering with Netflix as possible. That way the route between Verizon and your VPN/colo is as similar as possible to the Verizon<>Netflix route. You can then measure Netflix bandwidth to your VPN/colo, and the resulting full-path bandwidth.
I *strongly* suspect you'll see the exact same behavior, but by doing that you've proven beyond a shadow of a doubt that Verizon is absolutely to blame. It still doesn't separate the packet-throttling scenario from the insufficient-peering scenario, because even though your Verizon ingress point is ideally the same router, Netflix is *supposed* to peer to that router through dedicated lines (e.g. trunked 10G to the next room over where Netflix's router is).
Of course, since Netflix has offered to both purchase and install the 10G cards and wires on their own dime, that scenario is absolutely no different than packet-throttling. Except that in order to do packet throttling, Verizon had to spend *more* money on hardware than they would have to just add more capacity. Now *there's* a bit of research to do: $ to throttle vs $ to add capacity.....
GStreamer - The only way to stream!
This is not news. Verizon, Netflix and customers all already know what the problem is. Routing around the bottleneck (saturated interconnect) by using a VPN will obviously avoid the bottleneck... I don't understand why this was posted as a story.
Enraged Verizon FiOS Customer Seemingly Demonstrates Netflix Throttling
Ummm...NO. And TFA got it right early on, but not by the end. As did the blog post TFA didn't link.
It seems absurd to me that adding another hop via a VPN actually improves streaming speed. Clearly it's not Netflix that doesn't have the capacity
This is not provably throttling.
It is a form of control. It is a form of underprovisioning. It may be a... financial or "business level decision" form of throttling through under budgeting. But isn't provably, conclusively, or even... probably fucking network throttling. It's bad traffic rules at the ...worst.
Without demonstrating identical entry and termination points on the VPN path, or at least that VPN traffic exiting to the same peer from the VZW network flows faster, it doesn't fucking prove anything, or even hint or "cast doubt"...
And the people that don't realize it aren't fit to be a tier 2 helpdesk tech, much less touch a fucking network.
Verizon should have to provide the capacity they sell, and provide that to a majority of the internet, including any popular, reasonably well connected site. They've failed.
That doesn't make it throttling or a net neutrality issue. It makes it a provisioning issue. Get over it. This isn't your red-state/blue-state/political/gay orphan baby seal issue of the day. It's just an ISP playing the old telco underprovisioned/oversold game where they can't hold up to sold utilization.
My colleague and I can confirm Comcast does this as well. Put on the VPN and your streaming rates improve
This shit is just getting silly.
Seems like many peering conflicts are related to excessive vertical integration of some providers. Verizon is currently both tier 1 provider and consumer ISP. Suppose it would be split to Verizon1, tier 1 provider, and Verizon 2, tier 2 consumer ISP. Verizon1 would be one of several upstream providers to Verizon2. What wold happen in such case if there is such kind of peering conflict between Verizon1 and Lever3? Verizon2 would immediately change its routing in a way that traffic between Level3 and Verizon2 would go through another upsream provider of Verizon2, which would threaten potential revenues of Verizon1 from Verizon2, therefore Verizon1 would have much stronger incentive to promptly solve the dispute.
my wife: "Not that we're surprised, but it's nice to have concrete facts to back it up."
please... let me sleep... a little more... yay, no longer annonmyous coward.
and (who knows) pending upgrade.
I bet Verizon knows. ;) Pending an upgrade....Ya right, more like pending a nice fat check from Netlfix or L3
Uh, the traffic is _from_ Netflix, i.e. outside the Verizon network. Thus it certainly isn't Verizon who is choosing the congested route, the sender chooses the route, not the receiver.
Netflix is also a middleman. There are alternate ways to get that entertainment.
How did this makes the frontpage weeks after it happened, and after you've already posted the Verizon/L3 blog exchange?
I've never seen Verizon seriously promote RedBox instead of Netflix. I doubt they care as much as you think they do, especially since the profit margins are much lower on video services (due to the content fees).
What we need is more competition and communication of just how abusive companies are.
If people have alternatives, then there is no reward for shenanigans.
Get thee glass eyes, and, like a scurvy politician, seem to see things thou dost not.--King Lear
Whatever Verizon is doing with their networks is also causing serious issues for owners of Nintendo systems.
Often times the only solution for downloading a large game or sometimes even for just updating the Wii U or 3DS is to take it to a different location that isn't Verizon.
if it's that easy to bypass throttling, then have at it, i say - the more the better.
there has to come a point when derision and cronies will be forced to admit they have been raping yet another dead horse.
when dealing with bullies it pays to be light on your feet, i find.
This story is almost two weeks old. People here need to get on the ball.
More likely... Verizon sold me unlimited access to the fridge (bandwidth), Netflix (content provider) brings in trucks of beer, but Verizon (service provider) will only take one six pack every week to restock. Netflix even offered to supply a free fridge if Verizon would make space and plug it into an outlet.
Since she didn't use a car to break penises and scrotums, I don't understand what you're trying to get at.
"speed tests" (the consumer oriented public sort, eg: speedtest.net) are a joke because Oookla (et.al) encourage the ISPs to run a local copy of it .. and naturally the configuration favors the *most local* one by default .. so unless you're paying attention or use a geek-oriented test (of which there is no shortage, but not quite as easy to use) you are just testing the "local loop" portion of your connection.
I should point out here that the CIR (bandwidth you are promised) is generally only stated in terms of the local loop, since normally that's all that your access-ISP has control over.
FWIW you can download the Oookla (speedtest.net) client to run on your own network, which is rather handy if you want to test out that fancy 802.11ac wifi setup (or whatever).
The problem is that the laws that already exist aren't enforced. It's a secondary problem that they are so written that it's relatively easy to weasel around them, but even the existing laws aren't enforced, so adding new laws (that aren't enforced) wouldn't do any good.
I think we've pushed this "anyone can grow up to be president" thing too far.
I'm lucky enough to live in a building in NYC that has both FIOS and RCN available. I switched to RCN and now get excellent streaming from Netflix--FIOS sucked; both connections were advertised as 50Mbps download.
The FCC should make Comcast and Verizon and all these other companies chose... They can be content creation companies or bandwidth providers, but not both. This is all about making it easier and more convenient to pay them for their video services than to attempt to get it from a third party it's monopolistic and anti-competitive behavior...
"GET / HTTP/1.0" 200 51230 "-" "Mozilla/4.0 (compatible; Setec Astronomy)"
Heaven forbid they just upgrade the equipment and peering agreements to provide a better service to their customers. After all, $5 billion in net income in one quarter certainly isn't enough. I'm all for making money, but how much is enough (rhetorical of course)?
"The carrier’s net income totaled $5.07 billion in the fourth quarter."
http://bgr.com/2014/01/21/verizon-earnings-q4-2013/
If you've paid any attention to what's been going on, Netflix streams into the networks over network connectivity provided by level 3 communications. The heart of this is a "peering dispute" between level 3 and verizon. Verizon wants level 3 to pay them money to light up more connections between their networks, because the traffic is imbalanced. Level 3 says Verizon's claims are inherently unfair, as verizon's network is inherently imbalanced, so generates more down than up. By connecting a VPN, the guy probably routed netflix into verizon's network across a different backbone provider, and avoided the congested links which are at issue.
In short, he didn't demonstrate throttling at all, only that some links are congested and other links are not.
Try testmy.net, they don't colocate with ISPs.
I will not go back to cable no matter how crappy you make competitors streaming services. It is still better than cable. In fact, turning off the internet and watching a blank screen is still better than cable. Really.
Love,
Mike
I understand just fine. That's why I know better than most that what you are saying is a load of complete and utter crap deserving of contempt. Data coming from level 3 is being requested by Verizon Customers. Verizon isn't acting as a transit network unless they want to, so peering is just about getting Verizon customers the data they are requesting. Level 3 would gladly peer with Verizon in every metropolitan area in which they operate at very high speeds. Verizon is choosing not to.
This! Exactly this.
This is probably what is hapenning. However it doesn't mean there is no malice involved.
Sounds like they try to blackmail Netflix to pay extra to connect directly to their network. A la Comcast.
I have Verizon FIOS and I have done similar tests myself, and it isn't just netflix. The webcomic modestmedusa.com and the site acidsquirrel.com load very slowly, and this has only been for the past week or so. If I go through a VPN or ssh X forwarding, it is lightning fast. I too assumed it was due to throttling of some sort, until I found this:
http://forums.verizon.com/t5/F...
People claimed it was a DNS issue. I thought that was nonsense until I tried it. When I use OpenDNS, it solves the problem: both modestmedusa.com and acidsquirrel.com load lightning fast. I don't use Netflix, but maybe someone could try it and see if it fixes that problem?
More peering links won't rectify the imbalance. Your claims of knowing anything about the subject fall quite flat.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
Sounds like what comcast has been doing... When you connect to anything over the internet, for the first few seconds your bandwidth allotment is huge. Websites load blazingly fast. But try to actually do anything serious that doesn't involve just transferring a few kilobytes around, like say playing games or watching a video, and suddenly your bandwidth drops through the floor and all sorts of trouble shows up.
Any metric can be gamed...
Someone realized the Verizon/Level3 bottleneck is true!
Who cares about the imbalance? It's the congestion we want fixed.
Glad my ISP doesn't do any of this shit. No caps, no throttling, no filtering, no monitoring. It's just a straight connection with no restrictions.
If this kid is in NYC, then he must have alternatives. Perhaps not "up to 75mbps", which is a lie anyway, but still something decent. Here in LA, I pay $39/month for my true 25mbps + unlimited phone service.
Same exact issue I've been having on & off with Easynews.
Can I just pay based on the performance i'm getting Verizon? I have a 50/25 connection.
This is a Verizon vs Cogent issue. Verizon and Cogent have SERIOUS bottlenecks at their various peering points. It has been like this for well over a year. My company has various circuits and carriers around the US, including some with Cogent and Verizon 1GB ethernet service. The only consistant bandwidth issues we have been between the Cogent and the Verizon connected circuits between those sites. Some of their peering locations are better than others but some just flat out underperform. I have done troublshooting with Cogent from the point just before the handoff to Verizon, we can achive full line speed. Our tests into Verizon and on to our site from there sometimes maxes out well under 1mbit/sec for long periods of time. I've seen 50kbit/sec for periods of time too. We had to stop trying to use our Cogent circuit for that data and now route it out our other carrier in that office to our Verizon sites.
In my opinion, Verizon is not blocking or restricting Netflix at all, they are just sitting there looking at a peering interface with Cogent that is way too small and probably sitting at 100% utilization for most of the day. This guys VPN is just avoiding the bottlenecked Cogent-Verizon peering point and going around it another way. As inefficent it is to use a VPN for streaming Netflix, it is still better than trying to put your data through those peering points. This is networking 101.
In case you've ever noticed, every version of the SpeedTest browser page and mobile apps allow you to choose specific servers to test against. And "most local" actually means the one with the lowest ping times which isn't necessarily geographically closest.
For example, our Brisbane office uses TPG. Bizarrely, TPG routes *all* of their traffic to Sydney first and from there it gets routed to the rest of the internet. (Any TPG user in Brisbane can confirm this with a traceroute.) SpeedTest gives "ace" results when it selects a Sydney server (which it does by default) but comparatively crap results when it selects a Brisbane server - which is physically much closer, but actually twice the distance by internet routing.
Netflix either buys bandwidth or convinces ISPs to give it to them for free as peering. Its peers / transit providers are either big enough to peer with Verizon for free, or buy bandwidth from Verizon, and they don't have fat enough connections to Verizon to carry all of Netflix's traffic to this customer, at least at the peering point that Netflix's traffic uses to reach that customer (Netflix may have enough bandwidth to reach VZ customers in the other part of the country, e.g. LA peering is full while Seattle or DC isn't.)
The customer's VPN provider may very well provide free/cheap connectivity to Netflix (or at least fast enough), maybe even one of the providers Netflix buys from, because they're cheap. And they may also get it from somebody who has good bandwidth to Verizon. But peering's just peering - it's not transit.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
The problem isn't from Verizon's backbone to the customer - otherwise the VPN connection would have also been slow. It's the pipes from Netflix and their peering/transit providers to Verizon, which aren't big enough to handle all the Netflix customers on Verizon, at least in that region.
Also, what do you mean "has to set aside 8tb for sync"? Do you mean the ISP has to provide 8TB of some kind of storage hardware, or 8 terabits per second of traffic from somewhere to somewhere else? That's fairly huge, considering that most data connections aren't much bigger than 10 Gbps per wavelength (some carriers use 40, but it's not usually price-competitive.)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
The reason the VPN connection went fast isn't that EEEVILLL Verizon was throttling the customer's Netflix connections by doing deep packet routing and didn't do that to the VPN. The pipes Netflix bought to deliver movies to their paying customers who use Verizon weren't big enough to carry all the demand, at least at the peering* point that customer's traffic went through, while the pipes they bought or peering they got for free were big enough to reach the VPN endpoint, and the VPN endpoint had bought enough bandwidth from their ISPs to get from there to their peering point with Verizon, so there was enough bandwidth on the whole route to carry the movie that way.
That's not to say that there aren't ISPs harassing particular content (there was at least one well-publicized case a few years ago of some telco ISP blocking VOIP, and of course most of the cable modem and some DSL providers block home web servers), but this ain't one of them.
(*Peering unfortunately means two different things here - it's giving each other service for free, and it's having BGP-managed interconnections, usually at the big internet exchange locations, to pass traffic, not necessarily for free.)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Verizon and Level-3.
Good, then complain to Netflix. They can fix it.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
So you are saying that Verizon is too stupid to balance connections and packets over multiple routes? You can even buy multi-WAN routers for your home that do this for you. Verizon can't figure it out?
Go with a smaller ISP, I live in Toronto and we have 0 throttling, I get 5 MB/s at all times.
Sound going in and out and more lag than usual. I guess this is part of Verizon's new "network optimization".
Back when I was in the process of switching providers, I bought a subscription to a VPN service so I could have a secure connection and routable IP through public internet access points. Later one of the things I noticed with my new AT&T U-Verse service was that *all* access was faster in either latency or throughput using the VPN to tunnel through U-Verse to half way across the US including things like DNS. Some things were a little bit faster and some things were an order of magnitude faster.
Mod parent up...heavily.
How exactly would Verizon be "screwed" to allow L3 to provide Verizon Customers with the content they want at higher speeds?
Also how is it that L3 can deliver the data to Verizon, but Verizon is just somehow incapable of passing it along to their customers? The answer is that it is not a techn
Not a technical decision, but greed driven decision.
Just see all the Tek Syndicate videos on this, especially Comcast, Verizon, Tom Wheeler ones, and see how long this has been going on.
Then ask your Congressman why they are not doing anything about fraud, or how much the most hated corporations on EARTH let alone 'merica, are paying the "Congress Critter"
changing your routes changes the interconnects
changing your routes changes the interconnects
changing your routes changes the interconnects
changing your routes changes the interconnects
changing your routes changes the interconnects
Seriously, folks, changing your routes changes the interconnects.
His VPN provider probably has a much better route back to Verizon. Yes, Verizon is being somewhat dickish to not acknowledge that Netflix is a big driver for their higher speed plans and giving Netflix's bandwidth carriers a bit of a price break for that reason. No, this is no proof at all of throttling.
Is it evidence suggesting throttling? Well, yeah. Proof? Not even close. It's entirely consistent with what Verizon already said about an imbalanced interconnect that needs more hardware.
Thanks to recent FCC regulation, get used to this. Throttling is legal, net neutrality, gone.
Correct, and god forbid the CONSUMER actually expends a tiny bit of EFFORT and educates themselves. Very few places left where you don't have more than one choice for getting your Internet.
Of course the lazy folks, who want someone else to do their thinking for them, want somebody else to do all the work... Despite the fact that this always turns out the same, the politicians and the business folks get rich, the consumer gets screwed. Doesn't matter which party is in charge.
So get out there and vote - with your wallet. It's the most powerful vote you have.
Murphy was an optimist
It looks like you've stumbled onto one of the features of CDNs: Local copies. CDNs distribute content to a more local link, so that the same data isn't being re-transmitted through backbone links. Part of how this often works, is DNS. Each ISP operates their own DNS server, and has modified entries for CDN content. These point to the local CDN server, rather than the original source. The result is (supposed to be) a much faster and reliable connection.
When you use an independent or alternate DNS server, you do not connect to the local CDN. You get someone else's CDN, and the data still has to cross the internet again. It may, however, take a very different route, one that is not as congested.
Assuming Verizon wanted the traffic to come in a different route (e.g. LimeLight, etc), I believe they can update the BGP route they advertise to change that.
Otherwise, if Verizon just asks nicely, I can promise that Netflix would adjust the routing tables on their end to send it through another link. Just like they can expand the link with Level 3.
L3 cannot deliver the data to Verizon since there is not enough connectivity between L3 and Verizon to hand the data over at the interfaces where L3 is attempting to do so.
Verizon does not want to put all their bandwidth eggs in L3's basket just to accommodate Netflix so they want Netflix to either peer directly or force L3 and its other CDNs to re-route traffic through other Verizon peers.
Depending too heavily on a single upstream provider is not sound business practice and Verizon wants to avoid getting tied up in that sort of relationship with L3 mostly due to Netflix.