Ask Slashdot: How Can I Prove My ISP Slows Certain Traffic?

Long-time Slashdot reader GerryGilmore is "a basically pretty knowledgeable Linux guy totally comfortable with the command line." But unfortunately, he lives in north Georgia, "where we have a monopoly ISP provider...whose service overall could charitably be described as iffy." Sometimes, I have noticed that certain services like Netflix and/or HBONow will be ridiculously slow, but -- when I run an internet speed test from my Linux laptop -- the basic throughput is what it's supposed to be for my DSL service. That is, about 3Mbps due to my distance from the nearest CO. Other basic web browsing seems to be fine... I don't know enough about network tracing to be able to identify where/why such severe slowdowns in certain circumstances are occurring.
Slashdot reader darkharlequin has also noticed a speed decrease on Comcast "that magickally resolves when I run internet speed tests." But if the original submitter's ultimate goal is delivering evidence to his local legislators so they can pressure on his ISP -- what evidence is there? Leave your best answers in the comments. How can he prove his ISP is slowing certain traffic?

Comments

probably not slowing it specificly

[locopuyo]

They might not be slowing down specific traffic but instead just have a poor connection to those popular services and it gets saturated.

Re:probably not slowing it specificly

[baomike]

I think you have that right. I try to download files from GB prior to 1800 PDT as after that time the rate drops like a rock. What takes 5 minutes at 1600 takes 30-60 minutes at 2000. These are not netflix files but it would seem that in the evening netflixs flood the system.

You can't really

[Njovich]

Traffic slowed by your ISP, and traffic slowed further down the chain - for instance by poor peering - are indistinguishable. With some help from hops further along like Netflix (or within the ISP) you may be able to pinpoint the exact problem. However, given that so many providers are capable of routing Netflix at acceptable speeds it also doesn't matter, it's obviously your ISP's responsibility.

Re:You can't really

[behrooz0az]

he can try fiddling with ttl values and send duplicate packets and measure their roundtrip.
not easy, i know

Re:You can't really

[Greyfox]

Back in the day you'd fire up a tool like traceroute (or mtr) and see which hops are causing the problem. For me, it was always a sprint border router in Atlanta dropping all my packets on the floor. Seems like a lot of Linux distributions don't bundle traceroute by default anymore. In these days of generally-installing-everything-properly ubuntu (et al) distributions, you don't really need to be a sysadmin with a lot of networking know-how to run a networked Linux machine. Just get the wifi-enabled cable-modem from your cable provider and kick off your Ubuntu install. It's just like running windows -- wine will run most of your steam games and most MMOs flawlessly.

No QoS 4 U

[Anonymous Coward]

Until the government ensures a quality service on ISPs to make them similar to land-line phones (common-carrier), it doesn't really matter if you complain about small things you find. I have had VOIP blocked by the ISP due to their competing service, and have had speed issues that looked better with the speed test, but it really can't get resolved until the government cares about the Internet as critical infrastructure which can easily damage the economy.

Paper insulated

[AHuxley]

wireline might be a problem. Before the ISP side of the network.
Streaming services will often consider a users connection and change their streaming service to what they think will offer the best experience.

Long term support any innovative new community broadband network and see how the different networks do networking?
Freedom of choice.
Support a better new network.

Alternatively:

[Anonymous Coward]

Find a free vpn service, like vpngate, and run your connection over to a site (other than netflix, due to geoblocking) that normally runs slow for you. If your speed sees an increase, then yes they are throttling traffic to certain websites. If your speed is the same or less then no your connection to the outside world is just shitty. That is the #1 benefit to vpns for the consumer imho. Unless it is a specific service they are degrading they can't tell who your vpn connects to, and either they throttle all vpns, which commercial users would frown upon, or they throttle none of them beyond regular bandwidth limits and you can find out if that is where the problem lies.

Re: Alternatively:

[saloomy]

Not necessarily. The internet is comprised of a bunch of networks. If you are getting slow service to Netflix, it may be that you are traversing a saturated peer (maybe your ISP's peer, maybe their upstream provider's peer).

When you use a VPN, you are routing to a different site, which might have no saturated peers in the chain. Then from that site, you have a decent link to a Netflix node.

You are routing around using the VPN.

Also, 3mbps isn't great for streaming. When you say it's slow to Netflix but fast for sites, consider the volume of traffic. Netflix needs 5mbps for HD content, so you probably do t have a slow connection to one site, just in general.

Re:Alternatively:

[gravewax]

That doesn't prove it at all, by using a VPN you are going via different routes, possibly routes that are not as congested so you can get faster or slower speeds naturally through such a test. The reality is though services like Netflix et al do slow down considerably during peak viewing times as all ISP's have high congestion ratios, some worse than others.

Re:Alternatively:

[dwywit]

That's a big issue with the National Slowband Network in Oz. The resellers aren't buying enough capacity from NBN Co - so everyone experiences "buffering" from 5 - 10pm (or thereabouts).

I'm not going to sign an NBN contract that doesn't guarantee a minimum speed of at least 25MBit/s down - wish me luck.

There's a 5G network being rolled out on the Gold Coast to support the imminent Commonwealth Games - let's see what happens afterwards, shall we? After people have been been able to watch sports in 4K (and lots of people on the Gold Coast can afford 4K TVs), they won't be happy to go back to 4G or 3G performance.

Re:Alternatively:

[ColaMan]

It's not "buffering" or a "slowdown", it's advertised as "evening speeds".

As if it's a perfectly natural thing to experience congestion during the evening.

Re:Alternatively:

[gravewax]

if you don't want the slowdowns or at least for them to not be as bad then go with one of the providers that promises much lower contention ratios like Aussie Broadband. though the reality is the only way to absolutely guarantee you have 100% bandwidth is to buy a dedicated bandwidth, but then you better add at least another zero to your monthly bill.

Re:Alternatively:

[rally2xs]

Solution is:

Rent the CD with the movie you want to see from Netflix.

Use the program 123 Copy DVD Platinum to rip it to your disk.

Watch the movie from your disk. No buffering.

Simple.

Re:Alternatively:

[rally2xs]

Er... DVD, not CD... oops...

Re:Alternatively:

[Mr0bvious]

That's not true for all ISPs.

Avoid anyone selling "unlimited internet", that'll never work with the CVC costs and the leaches it attracts.

I'm on a $69 for 1TB a month on a 100Mbps plan and always get over 80Mbps.

Shop around and ignore the big guys, they're all shit.

Go to whirlpool and checkout Telecube and then thank me later.

Re:Alternatively:

[Mr0bvious]

But still... eeewww, DVD.

I'd hate to be your enemy.

Re:Alternatively:

[Anonymous Coward]

Or move to a civilized country. I live in Sweden and pay ~10€ for 100mbit unlimited fibre. I get more than 80MB through my vpn. .

Re: Alternatively:

[silas_moeckel]

If your ISP has a constantly saturated peer that is in effect throttling. We have seen ISP's use BGP traffic engineering to try and push all netflix traffic through a specific peer and then let it saturate the links as a bargaining tool. This should not be allowed. Realy any link that's saturated anywhere inside an ISP or with peers and transit providers should not be allowed in the long term.

Re:Alternatively:

[omnichad]

DVD is of a lot higher quality than a lot of my attempts to stream HD content from Netflix. And I don't even live in AU.

Re:Alternatively:

[omnichad]

That's nothing to do with civilized. It's about population density and the country not being on its own continent.

Re:Alternatively:

[Mr0bvious]

Sweden approx 450,000 km2, population approx 10m.
Australia approx 7,700,000 km2, population approx 24m.

Can you see why it may cost more to provide an infrastructure for Australians?

Re:Alternatively:

[Anonymous Coward]

I'm paying $50/m for a 150/150 dedicated fiber with dedicated bandwidth in the USA, at my home. I get 150/150 to nearly every datacenter in the world at every time of the day. They literally advertise ~"We're not like that other ISP. We sell *dedicated bandwidth. You will never experience any slow downs...*Dedicated through out network and to our transit provider".

When I first got services from them, I talked to the manager of network operations, asked them how they could afford dedicated bandwidth. He said the cost of bandwidth isn't a concern. He said bandwidth is so cheap, they're not even going to get a Netflix CDN, even though it makes up most of their peak bandwidth. Managing bandwidth is more expensive than just buying it. It's plain cheaper just to give dedicated bandwidth than to ration bandwidth. I asked him about the company's position on CDNs in general. He said they had no plans to add CDNs to the network because it's not worth the money saved, but if they did, it would be to make services better, because CDNs can provide lower latency even if bandwidth is not an issue.

To give a better idea of the cost of bandwidth, talking to their network admin, he said they have 6 transit links, each one is provisioned so much bandwidth, that if 5 of the 6 links went down, their 95th percentile would be around 50% during peak hours. I asked him about DDOS attacks. He said that most DDOS attacks just get soaked by their nearly 10x more bandwidth provisioned than needed, but even in the cases where attacks are larger, they just buy more bandwidth for the duration of the attack. Only takes about 15 minutes to get their bandwidth increased.

I can stream 149.9Mb/s of UDP over my 150 connection with zero packets dropped over a several hour window that spans peak hours. At 150, I start to see some loss and at 150.1 I see an overall statistical number that nearly perfectly matches being 0.1 greater than 150. One month I decided to ping a datacenter in Europe, ~140ms away. 10 samples per second, for 30 days strait, 25,920,000 total packets, fewer than 100 packets dropped, min ping was 0.1ms less than the average, and the std dev was less than 1ms with a max of ~160ms. Mind you I was seeding about 100GiB of the most popular Linux ISOs on BitTorrent during this time.

Hell, my state Uni has at least 1Tb/s of transit. At least 2 100Gb links to each of Level 3, Sprint, AT&T, Verizon, Hurricane Electric, possibly others that I'm not aware of. Not to mention another several 100Gb links spread to different regional IXs, and 100Gb peering links to local ISPs. This info is over 5 years old now. I'm sure they've increased their bandwidth since.

Bandwidth is not expensive. Sonic.Net gave an interview where they said infrastructure and transit only represents 1%-2% of the cost of being an ISP. Your connection and the bandwidth it provides is a freaking rounding error on your bill and less than the sales tax. And that 1%-2% also includes maintaining and upgrading. Short of living in Alaska, national park, an island nation, or starting a new ISP, there is little reason for non-dedicated bandwidth.

Re:Alternatively:

[Anonymous Coward]

Northern Sweden and Wyoming should be fairly equivalent.
Central Stockholm and whatever semi-large city of your choice should also be fairly equivalent.

Also, not being on its own continent has jack shit to do about it.
Population density matters a bit but it seems fairly obvious that cronyism and the aversion to forcing competition in the US have a bigger impact.

Re:Alternatively:

[pnutjam]

You can also try a speed test specific to the service you want to use.
Netflix has one:
https://ispspeedindex.netflix....

Re:Alternatively:

[mauriceh]

Most of the population of Oz is in an area that is much smaller.
Same applies for any country.
Your argument is weak

Re:Alternatively:

[rally2xs]

And 123 has an extra-cost option to rip blue-rays, too, if you really want HD.

Re: Alternatively:

[Anonymous Coward]

Surely some of the money that he's paying in is allocated for infrastructure upgrades. Ergo, isn't he already paying for it?

Re:Alternatively:

[dryeo]

How many countries have DVD rental from Netflix? I'm in Canada and don't seem to have the option to rent DVD's, though perhaps I just missed it.

Re:Alternatively:

[dryeo]

Not everyone lives in a small highly populated country with actual choices for an ISP.

Re:Alternatively:

[Anonymous Coward]

It's not "buffering" or a "slowdown", it's advertised as "evening speeds".

As if it's a perfectly natural thing to experience congestion during the evening.

Because the speed you are paying for is the "when nobody else is using it speed".

Re:Alternatively:

[rally2xs]

Yeah, sending thru mail back and forth to Canada may not be an option, don't know.

Re:Alternatively:

[Anonymous Coward]

What is expensive is running the cables which is why the telcos are so reluctant to do it.

@AC1: Running the cables is covered by the 1-2%, over time, but the up-front for a new ISP can seem large. Let put it this way. Running the cables costs LESS than sending a person to your house, even once. Back when I had Charter, they had to stop out at my place about 3-4 times a year. The average fiber install costs an ISP about $300 assuming bulk rates. The average truck roll cost is about $300. Over a 5 year period, Charter could have run fiber to ~15 houses and probably have zero issues due to crappy copper signalling.

At one point we were paying in the order of 50k a month for roughly 7gbits.

@AC2: Totally agree. Around here, AT&T used to charge $100,000/m for 1Gb/s, but then a non-profit with RoW access came in and started offering the same for $300/m. AT&T opened a law suit against the State, saying only Telcos and Cable have access to RoW, and this non-profit ISP is neither. The State ruled that the ISP was not allowed to sell to the public using RoW access, but they could sell to the State, education, or health care.

Maybe you think that AT&T sold a better connection, because it was $100,000 vs $300. Nope. AT&T's "dedicated" connection had constant down time, even though they had an SLA, and slow performance because AT&T only claimed to have dedicated bandwidth back to the CO, not to the backbone. The $300 ISP claimed full dedicated bandwidth and had nearly perfect uptime. That was about 15 years ago now. $300 from the non-profit gets you about 10Gb now, but they can't sell to the general public. But that's probably a good thing, because over 50% of most ISP's costs is support.

Re:Alternatively:

[Anonymous Coward]

Solution is:

Rent the CD with the movie you want to see from Netflix.

Use the program 123 Copy DVD Platinum to rip it to your disk.

Watch the movie from your disk. No buffering.

Simple.

Waiting for the DVD to arrive is the "buffering".

Re: Alternatively:

[skids]

Yup.

The proper tool to try to figure out where packets are being dropped or delayed is called "paratrace"

You kind of need to know what you are doing to use it properly... you have to find the connection that is being slowed and jump on it.

Also, and this goes for traceroute, too, if a single transit node has high loss or delay, but the nodes beyond it do not, then that node should not be blamed... returning packets due to TTL exhaustion may be CPU-bound or control-plane-policed on a transit node, which is normal on some equipment and won't matter for normal traffic.

Re: Alternatively:

[Kludge]

When you use a VPN, you are routing to a different site, which might have no saturated peers in the chain.

If your VPN can take a faster route than your Netflix, your ISP could choose, if they wanted to, to route your Netflix that way too. By choosing not to, they are in effect throttling your internet.

Re:Alternatively:

[lsllll]

Find a free vpn service, like vpngate, and run your connection over to a site (other than netflix, due to geoblocking) that normally runs slow for you. If your speed sees an increase, then yes they are throttling traffic to certain websites. If your speed is the same or less then no your connection to the outside world is just shitty. That is the #1 benefit to vpns for the consumer imho. Unless it is a specific service they are degrading they can't tell who your vpn connects to, and either they throttle all vpns, which commercial users would frown upon, or they throttle none of them beyond regular bandwidth limits and you can find out if that is where the problem lies.

Not necessarily. VPN traffic is very to determine based on size and frequency of packets. Take, for instance, using a BitTorrent client behind VPN. You see many short burst packets going in and out for a while and they can pretty much deduce that you're doing BitTorrent. Many equal length packets coming in and small packets going out? You're downloading a large file. How do you think you're get near advertised link speeds (or better) when going to speedtest.net, even through a VPN?

Re: Alternatively:

[saloomy]

No. BGP doesn't work that way. It is not a load balancing routing engine. It picks the shortest path first, and sends traffic that way. Also, different paths that he may be traversing might be significantly less congested but much smaller. Sending the traffic that way could overwhelm the other peer.

Re:Alternatively:

[Mr0bvious]

Your ISP will be purchasing bandwidth from an upstream provider, who will be doing the "provisioned rate', while it seems that your ISP purchases non over-subscribed bandwidth, they're just pushing the problem (limitations) further upstream.

It's still going to be there, just shared amongst and even larger pool of "customers".

fast.com

[Anonymous Coward]

https://fast.com

This site uses the same servers as the Netflix streaming service so it should be a clear indicator. You should note though that 3Mbps isnâ(TM)t very fast when it comes to playing streaming video. An HD stream from Netflix can easily hit 7 or 8 Mbps.

Re:fast.com

[Sarten-X]

It's disappointing that this is currently scored 0. This is the right answer for this scenario.

With strong Net Neutrality laws, there are limits to how sophisticated ISP throttling can be and still pretend to be legitimate. With that essentially eliminated now, the only meaningful test is to use actual traffic. Netflix has preempted this need for their own services by creating fast.com to look identical, going to Netflix servers over the same ports and protocols as normal Netflix traffic. It will be subject to the same throttling, and thus allows you to measure the speed you get when working with Netflix.

I'd love to see other services hosting similar tools, but for now Netflix is the only major company I know of offering their own user-accessible performance test.

Re:fast.com

[gravewax]

The problem with that is that by going over the same network and peering links all you will see is your actual current speed with no indication about whether you are being slowed down by contention/congestion or through throttling.

Re:fast.com

[Calydor]

Well, that can be gamed by shady customers setting up a ping script to that IP that fires every 20 minutes or so.

Re:fast.com

[Athanasius]

I was going to post fast.com if no-one else had.

There's also a list of other tests on https://www.eff.org/testyouris...

Re:fast.com

[omnichad]

Both are gamed by the ISP and have the same result. It doesn't really empower me one way or the other to know the difference

Re:fast.com

[omnichad]

It would fit the same profile and no - SNI requires the traffic to be decrypted. If you're able to distinguish that, then the encryption is utterly broken.

Re:fast.com

[Actually, I do RTFA]

All the major streaming services offer their own speedtest. There's an Apple App (yes, it was approved. It was only delayed while apple confirmed it wasn't snakeoil) that will check all the video services speedtest sites. I don't know if it's open source, but there was a slashdot article on it a while ago. Or you can build your own script.

Re:fast.com

[i.r.id10t]

Indeed. My ISP is pretty good (I'm the one throttling Netflix traffic here at my house... wife and 3 kids eat up a lot of bandwidth and I need some reserved for me to do homework) so it hasn't been a concern for me but I've wondered why there haven't been content provider supplied speed tests. Of course, I didn't know about fast.com either...

Re:fast.com

[WaffleMonster]

It would fit the same profile and no - SNI requires the traffic to be decrypted. If you're able to distinguish that, then the encryption is utterly broken.

SNI is transmitted in the clear for the simple reason servers absolutely need to know a hostname in order to determine what certificates it needs to load to
communicate with the client. Certificates often managed and owned by completely separate customers of the hosting provider.

There have been proposals to obscure SNI behind some kind of anonymous DH scheme yet even in TLS 1.3 as it is this is still done in the clear. Currently SNI on all production systems is performed in the clear.

Re:fast.com

[omnichad]

Oh, right. However, the UI is the only thing that needs to come from fast.com. Everything else can come from netflix-753.vo.llnwd.net or whatever they want. And Netflix can make all its clients load the fast.com page before streaming and pretend to be a speed test if it wants. Though it would be easier to implement a delayed throttle that lets the first few minutes in at full speed. On the other hand, video is requested in chunks and sometimes behind carrier-grade NAT or home LAN. It would be difficult to be selective.

Re:fast.com

[gravewax]

The question asked was "how can I prove my ISP Slows certain traffic", the site mentioned does not provide a solution the ability to discern that.

Re:fast.com

[uvajed_ekil]

It's disappointing that this is currently scored 0.

It's disappointing that you think 16 minutes (including the time it took for you to type your post) is too long for an anonymous post to get modded up. I'm rather glad that AC posts don't just start at Score:2 for no good reason. The post in question is proof that the moderation system is working correctly and effectively, even if not instantly.

Re:fast.com

[omnichad]

Both are the ISP slowing traffic. Intentional sabotage of peering link capacity is no different effectively than throttling.

Re:fast.com

[Sarten-X]

Eh... groupthink and vitriolic anti-corporate shitposts get bumped up in far less than 16 minutes, but I digress.

When a good comment that perfectly responds to the topic at hand (and with good information) sits at 0 because it happened to come from someone unregistered, what's really disappointing to me is that there is no good solution. Heuristics to guess an expected score can be gamed. Tracking users without accounts is a privacy minefield. Assuming an initial score and letting the mods do their thing leaves a time gap where good comments are scored unfairly... and that's precisely what we have now.

Slashdot's scoring and moderation system is the worst, except for everything else that's been tried.

Re:fast.com

[Scarred Intellect]

Except if I was an ISP and we throttled Netflix, I'd specifically whitelist fast.com (and other speed tests). I wouldn't use this as any sort of proof that throttling is or is not happening.

Re:fast.com

[suutar]

you mean you'd specifically throttle fast.com, right? Otherwise you're just making it really blatant that you're treating netflix traffic differently...

Re:fast.com

[Scarred Intellect]

No, I'd let those speedtests run at full speed.

The implication is that fast.com is owned by Netflix, and if that operates at full speed, Netflix obviously operates at full speed. If fast.com results are consistently lower than others, THAT would imply that Netflix traffic is being treated differently. And this way I, as an asshole ISP, could say "Fast.com is run by Netflix, if it's showing you good speed, and videos aren't playing well, then it must be an issue with Netflix's servers"

Five easy steps

[Anonymous Coward]

1. Get job working for ISP
2. Slow certain traffic
3. Film yourself
4. Post video to youtube
Profit

Prioritizing speed test servers a NN violation

[Anonymous Coward]

Many ISPs large and small explicitly add rules prioritizing speed test servers over normal traffic. To this day it baffles me why ISPs are getting away not only with blatant violation of NN by prioritizing specific destinations over others but with what is essentially intentionally misleading their customers with bogus results as a consequence of prioritizing speed tests. The trick to obtain meaningful results is in finding much less known speed test servers or just downloading large files from a sampling of random sites. This will win you a better baseline to reason about the speeds you are actually getting.

It's otherwise really hard to tell on an individual destination basis for sure WTF is going on ... where the bottleneck really lies. You can trace paths and grab routes from looking glasses and shit yet as an end user this shit is rarely worth doing or conclusive. Smaller ISPs ... especially WISPs have bandwidth management boxes some of the layer 7 variety that execute all manner of wizardry the end user has little to no hope of deciphering.

I'm on several industry mailing lists where ISPs often bitch about what is a very common luser complaint about how ISP is preventing them from access something or is responsible for slowing down their access to something. To be fair MOST of the time a user arriving at such conclusions is in fact both clueless and WRONG yet when ISPs pull shit like this it's really hard to sympathize with the ISP.

The marketing of speeds is especially is difficult because they don't guarantee any particular speed to any particular destination, they often don't disclose details about bandwidth management or the fact they are prioritizing speed test servers. Often times the concept that speeds are not guaranteed for every site is not communicated in any marketing materials and is hard for end users especially ones who have no technical knowledge of the workings of Internets.

Re:Prioritizing speed test servers a NN violation

[postbigbang]

Far be it from me to defend a carrier or telco, but you're not being realistic.

ISPs have routes to other peers. Inside their own realm, they may or may not have links or even hosted content distribution networks to ease the traffic on their core routers. Netflix, along with Akamai and plentiful others allow peering agreements to boost the QoS of their delivery to end nodes.

This in turn, is a bit different than how LTE carriers deliver their own feeds. Some people NEVER watch a movie on their phone, and others consume all of their video on their phones.

In a neighborhood with lots of WiFi users, many people misconfigure their access points, especially in densely populated metropolitan areas and apartment complexes, etc.

There is a HUGE problem called low-hanging-fruit where ISPs/carriers/telcos only want to invest in upgrades when it pays off for them by covering lots of density with small investments. Doing huge system-wide upgrades of their core isn't the problem, it's the last mile costs to all of those end points.

And I'm also sure they have shenanigans, but also the results of everyone going to a newly popular site that still doesn't have load-balancing, CloudFlare protections, and so forth. Some people also have devices that are completely clogged with anti-virus software, or who bitch when they have 150+ browser tabs going, each active content moving in them.

Summary: it's not easy to catch the shenanigans, and there are lots of rational explanations for odd slowdowns, including your own device CPU loads (due to anti-virus activity or poor resources). A simple speed test may or may not indicate something's wrong. It's just not that easy, sadly.

Re:Prioritizing speed test servers a NN violation

[pnutjam]

If you have access to a server elsewhere, you can use iperf. That's usually my gold standard for testing links.

you can...

[johnjones]

its not that hard

but you have to have at least two connections to compare the traffic

a study that was funded by a USA national science award does exactly this :

simply download a app and run it

http://bit.ly/2IAdbmD

you can thank me on twitter if you like http://twitter.com/johnjonesname

Re:you can...

[ruir]

I have worked in a couple of ISPes and can for sure say you can pretty much inspect traffic even more than people think. Besides inspect layer 7, those boxes do heuristics, and nowadays with higher speed Internet, even match with "home" IP addresses and content signature to detect malware, to try to guess the nature of encrypted traffic.

Re:you can...

[cciechad]

They technically do peer but it peers mostly locally in the ISP's core so it doesn't go over what we normally call "peering links". Even my smaller ISP here has a netflix CDN cluster here that takes about 90% of the traffic and serves it locally right from the ISP's regional DC. The rub is the ISP gives these boxes free power/space and access to peer to them in exchange for cutting the netflix traffic flowing out the peering links. In this case the only traffic that actually goes out is the replication traffic for the cluster, some small # of titles that aren't on the cluster and DRM traffic for playback.

Re:you can...

[max99ted]

I don't have a Twitter machine so I am thanking you here instead :)

Re:Hard to prove as end user

[WaffleMonster]

Video is very susceptible to jitter

No it isn't.

Not necessarily your ISP at fault

[gweihir]

Streaming is a complex game and it can get disrupted in many ways. Packet loss, jitter, reordering, buffer-bloat, brief interruptions on traffic spikes, etc. all things that are no so bad with TCP can really, really mess up streaming. My guess would be that the streaming services all use a very similar set of parameters for their protocols that in general work reasonably, but with your connection does not work well at all. The solution to that would ordinarily be to just download the files and then play locally.

Re: Not necessarily your ISP at fault

[DaMattster]

My guess is that most streaming is using TCP

If most streaming used TCP, there would be all kinds of buffering and slowdowns. I think you're wrong.

Re: Not necessarily your ISP at fault

[WaffleMonster]

If most streaming used TCP, there would be all kinds of buffering and slowdowns. I think you're wrong.

ALL of them use TCP.

Confusion likely stems from mistaking realtime bidirectional communications such as voice and video calling which are inherently delay intolerant with bulk transfer of static content. Two completely different technologies with very little in common.

The way you keep from buffering and slowing down is by creating a large enough stream buffer to hedge against presence of any future transient conditions.

Re: Not necessarily your ISP at fault

[gweihir]

Streaming over TCP requires abundant spare bandwidth to work. A good rule of thumb is reliably (including peaks) less than 50% utilization in both directions for it to work fine. This situation is rarely given, so basically no streaming uses plain TCP. It may use some other protocol masking as TCP to get through firewalls and other things, but that is it.

Re: Not necessarily your ISP at fault

[_Sharp'r_]

All of the most common streaming protocols (HLS, HDS) use TCP connections. In fact, they mostly use HTTP over TCP/IP connections.

It seems like you're confusing higher level "protocols" with lower level protocols, of which you can pretty much use TCP or UDP on the Internet. Streaming connections virtually all use TCP, not UDP.

So yeah, you'd run HLS as a specialized application for streaming version of HTTP, but you'd do it with normal TCP connections and all that entails as the underlying session/transport layer and with IP as the network layer.

Netflix places servers with ISPs

[NichardRixon]

Netflix has servers in strategic positions within most ISPs, or in some cases just outside. Please see:

https://media.netflix.com/en/c...

In addition, Netflix automatically adjusts the video resolution depending upon the bandwidth of the connection.

Re:Hard to prove as end user

[gweihir]

Depends entirely on what is used for flow-control and how much buffering is done. Jitter can have minimal effect on that or it can shoot things to hell.

Use the Netflix app speed test...

[ambrose.carracho]

Many devices allow you to check your connection speed within the Netflix app. (I would presume HBONow offers a similar utility.) The Netflix Page explaining how to do this is here: https://help.netflix.com/en/no.... On the same Netflix page are their internet download speed recommendations for playing TV shows and movies: 0.5 Megabits per second - Required broadband connection speed 1.5 Megabits per second - Recommended broadband connection speed 3.0 Megabits per second - Recommended for SD quality 5.0 Megabits per second - Recommended for HD quality 25 Megabits per second - Recommended for Ultra HD quality Remember that these figures are per stream, so concurrent streaming to multiple devices is bandwidth additive.

Re:Use the Netflix app speed test...

[dargaud]

Also, do some speed tests. Then load up a VPN (one you pay for, not a free one), and do the test again. If the speed jumps up, something fucky's definitely afoot.

Remote in?

[Presence Eternal]

I'm not an expert on these things, but maybe do it from the outside? Set up an internet facing server on your network and then push and pull data using a known honest provider.

Send them one of these

[wolfheart111]

https://tech.slashdot.org/stor...

Honorifics on Slashdot

[jrumney]

Can one of the editors explain why a high 6 digit Slashdot reader qualifies for the honorific title "long term", but a low 4 digit Slashdot reader does not?

Has darkharlequin done something to displease the editors?

Re:Honorifics on Slashdot

[Anonymous Coward]

A 4 digit ID would come from around 1999, whereas a high 6 digit ID would be from around 2003.

So one guy has been on Slashdot for 15 years, the other for 19 years. They are both "long term".

Matt's TraceRoute

[jrumney]

A good way to find where a connection is being slowed is to use MTR, or on Windows WinMTR. It's a combination of ping and traceroute that can show where the network becomes slow, or error rates become high between you are the server you are using.

Speed testing Comcast

[meerling]

For many years I've tested my connection on comcast.
If it's a popular or well publicized test site, comcast gives back great numbers.
On the other hand, if you use any of the various ways to obfuscate the address, or just use one comcast doesn't have on it's script yet, then you'll see MUCH lower speeds.
Yes, there are ways to verify that the obfuscation isn't causing the slowdown.

Short version, comcast slows you down unless they know they're being tested, then they give you a higher bandwidth. I've tested them for close to 10 years now, and it's always the same.

ask your question in a place

[weedjams]

where the people live and breath this stuff 24/7/365. http://www.dslreports.com/forums/

Compare it to a VPN

[duguk]

This will sound odd, but try a VPN over the same connection.

I found my ISP was slowing down all traffic, apart from to speedtest.net and other speed testing sites.
However, they were not slowing down VPN traffic.

After running all my traffic for a month over a VPN, my speeds were 10x faster and not slowing down at peak times.
Then I received a call from my ISP kindly asking me to leave. I'm now with a decent ISP.

Re:Compare it to a VPN

[dargaud]

Names ! And shames !

Re:Compare it to a VPN

[duguk]

It was Eclipse Internet. They don't really seem to exist any more and are part of KCOM.

Before that, I had a similar problem with O2 Broadband, and they went the same way.

Try comparing regular vs VPN but...

[TheOuterLinux]

If you're a Windows user, sometimes the system will switch back to the default ISP's DNS for certain programs (Edge browser *cough cough*) and not just p2p/torrent stuff because, and like we Linux users keep trying to say, they are evil, assuming that all the ads and tracking doesn't give you away or if Micro$oft itself made a deal to throttle on behalf of the ad related conpitition. How else can Cortana give Micro$oft their untainted data fix if your lying to her or get you to use Bing instead of Google? Skype instead of literally anything else? Payed subscriptions instead of free and open source cross platform tools? I even have people telling me their Windows system is warning them that Firefox is an insecure web browser because they clearly want them to use IE or Edge. Classic web browser war bs, but they go much further than that now and you don't get a much of a choice these days. I guess I'm saying it could actually be your Windows 10 system being a dick instead of the ISP. Linux users will not wave it in your face like you think if you switched unless you picked pure Ubuntu (they're partners of sorts). Using WSL isn't going to help. Also, using ping isn't going to help either because it only rates the response time and not the amount of data transferred which is what the ISP is going to look at. It might not hurt to check anyway because maybe it's the just servers acting up.

Use Netflix's speed measuring tool

[The MAZZTer]

It tests using Netflix so you can compare to other speed testers.

https://fast.com/

Wehe

[pgn674]

If you have an Android or iOS device, then download the Wehe app and run it while you're connected to your home WiFi. It was developed by university researchers to detect slowdowns of certain streaming services.

Check to see if they are using a proxy cache

[wezelboy]

My ISPs dirty little secret is that it routes all http(s) traffic through a proxy cache server, so most popular websites run load great but everything else is often slow.

Wehe

[alexo]

You are welcome.

Re:3Mbps

[omnichad]

Your math is wrong. All you have to do to prove it is to run Wireshark or a switch with a promiscuous port and log how much data has passed through your connection. Or download the same title for offline viewing in full HD quality. Those bandwidth numbers match up with a number of HD torrents of high quality and carefully tuned encoding.

3Mbps is required for Netflix

[duckintheface]

He says he get a standard 3 Mpbs from his ISP. This is about the speed required for a non-buffering stream from Netflix. So any small dip in speed or the use of a second application at the same time as Netflix would result in buffering. Doesn't mean the ISP is not throttling but it could just be that the standard speed is not up to the job.

Re:3Mbps is required for Netflix

[JDeane]

I have 200Mbps down and 40Mbps up, and in spite of that at times Netflix gets a hiccup now and then. Not often mind you but it can still happen, usually around 5PM to 7PM If it's going to happen.

About the only things a person can do on their end to improve things is getting your own modem and router, a good quality home network will make sure your getting as much bandwidth as you can out of the wall. (I have a 320Mbps Motorola cable modem and a Linksys EA7500 not the best stuff but it's leaps and bounds better than the stuff my ISP would try to give me.)

Also make sure you don't have 18 devices on that 3Mbps connection... Updates and random checks for the weather and and and will suck up that 3Mbps pretty quick.

At that speed you might go into your Netflix settings and change it to 480P :(

SFTP to Godaddy from Rogers blocked

[FeelGood314]

I can't use SFTP to access my Godaddy account over Roger's infrastructure in Canada. I have ISP accounts with Carrytel and Tecksavvy both using Rogers cable infrastructure and my traffic is blocked. However when I use Bell everything works fine. Godaddy support was good, Tecksavvy is always amazing but we couldn't resolve it and rogers was completely unhelpful.

Has anyone else experienced this and been able to resolve it?

Re:an anecdote from a Comcast technician

[pnutjam]

The company lies about its internet speeds but all of my customers know that and still subscribe to them.

The company lies about its internet speeds but all of my customers know that and unfortunately have no choice but to subscribe anyway.
FTFY

netflix slow

[OFnow]

Last night netflix movies would simply get to 20% (of the way to starting the movie) and stop, meaning the movie never actually started. In every case it got to 15% or up to 23% and...no further. I tried several times on each of several movies.

Switched to amazon prime and no delay, fast response, worked fine right away.

I was asking myself how I could check on comcast (the only ISP available here).

It's a TV, so no obvious way to run a speed test on it.

compare the speed on different ports.

[Washuu2]

Please note, the OP asks for the ways that can be accepted as a legal means. I would set up some host on the internet with public IP, and set the pack of speedtests (for instance using iperf) using different TCP/UDP ports. If the tests will show some significant differences on the ports used by P2P, than for ports 80/443 for example, this could be accepted as a proof, I suppose.

Re:an anecdote from a Comcast technician

[vtcodger]

If Ayn Rand had had to deal with Comcast, she probably would have embraced Marxism.

Re:Hard to prove as end user

[WaffleMonster]

Depends entirely on what is used for flow-control and how much buffering is done. Jitter can have minimal effect on that or it can shoot things to hell.

No it doesn't. The streaming services use TCP. Jitter and even packet loss are often concealed by the receive window depending on where event occurs. It's a statistical game where to win you need only a buffer big enough to account for transient conditions and of course channel capacity larger than consumption rate.

Massive jitter or packet loss at tail can indeed stall a stream and even result in visually affecting output yet TCP protocol is reasonably well designed and capable of efficient bulk transmission which is all that is necessary for video streaming services.

Remember parents comment was "Video is very susceptible to jitter" which is simply false despite ones ability to concoct ridiculous scenario where jitter becomes service affecting.

Method

[jd]

1. Use pchar to establish the bandwidth and packet loss ICMP sees between endpoints.

2. Craft packets that contain magic numbers or magic strings, I'm pretty sure that's hping, and see if there are behaviours that only occur with given sequences regardless of endpoints.

3. Traverse the same segment of net using an encrypted tunnel, as encryption is slow. If this causes a massive acceleration, then it cannot be explained by a change in path, only by a change in visibility.

4. Use a proxy that is on the other side, so you traverse the same path but alter the visible destination endpoint. This will detect the most common form of hostile traffic shaping.

If you detect an obvious, reproducible, pattern then you have shown hostile traffic shaping and the form it has taken.

OONI Probe

[GameboyRMH]

https://ooni.torproject.org/

HTH

Routes, pipes and bit-twiddling.

[skeezix-the-cat]

I've read most of the threads; some good knowledge out there. As said dozens of times, very difficult to pinpoint troubles. I mean, consider the architectures: Last Mile terminates inside a carrier's pop. From there on out, its kind of a crap-shoot. Even peering links get saturated (drops occur), even go down, in which case re-routes come into play. The internet's advertised functionality is a survivable cloud, where re-routes fix saturated or downed links. In principle. Working for tw telecom for a few years in their noc as a repair tech, outages and the like impact traffic in crazy ways. Without direct knowledge of what links are running at capacity, or what outages are going on it is just so hard to really maintain path integrity. And that's for enterprise customers, who pay through the nose for SLA. Residential broadband is at the bottom of the priority list, virtually always Best Effort. Which means exactly what it says. Factor in the corollary effects of *cough* net neutrality --it is my hope Ajit Pai can never take a cup of coffee in public again without risk of a beating-- we are indeed in a brave new world of throughputs and L2/L3traffic classifications. It is easy to rail at the ISP, but unless as a user you contract for SLA, and in specific SLA to that which is important to you, the Interweb will ALWAYS be a crap-shoot. The miracle is that it works as well as it does.

I would pay money ...

[EngineeringStudent]

I would love to see a standalone application that runs through enough encryption that the ISP can't selectively speed-up or slow down, that uniformly randomly tracks speed to at least one main (not advertiser/spam) site from my computer through my ISP and makes the results public. I would love to see a non-profit make this happen.

Every single customer of every single ISP would want this software. It would allow hundreds of millions of people to keep someone who has a monopoly over them honest. It would "the next killer windows program".

I think there are hundreds of millions of customers in the US who would like to see this. I think it would put teeth and a few hundred million customers behind "net neutrality" support, and make it happen. It might lead to internet access as a public utility, instead of toll roads and monopolies on driveway - which is what we have now.

Netflix is Amazon, which is insanely huge architecture and throughput. I can't imagine Netflix being the slow one. I can't not imagine your ISP being the slow one. They do this crap all the time. It is crap like this that drives cord-cutting. If AT&T/Verizon/Sprint weren't so miserly in their approach to cell phones, they could stunningly vastly improve their income by making an equivalent service to compete in the land of monopolies. If half of all ISP's are effectively a monopoly, and if the telecom just went "same price for same alleged service" then majority of all ISP customers would move, because ISP's don't make their on-paper commits in real life and lots of people are angry about it. Telecoms can't find the will to do that. Truth, however, would be transformative. If the ISP was measured/shown to be the ass every time they cheat, and they can't not cheat, then you would see what happens.

Typical peering problem

[GuB-42]

Your ISP probably doesn't slow down anything. It's just that they didn't partner with the content delivery networks of some popular services. There are often disagreements on who is going to pay... As a result, all Netflix or whatever traffic goes through small pipes that are not designed to handle such heavy loads.

If you notice that speeds vary during the day, with prime time being the slowest, than that's must be what happens. The mitigation is to force the data to take a different path, one that is not saturated. The obvious way is to use a well connected VPN, but sometimes, just using DNS tricks in order to use a different server than the "closest" one can do the trick.

Re:Hard to prove as end user

[WaffleMonster]

Neither of these statements is particularly precise, but the first is general enough that your response starting with "no" is less correct. TCP is very sensitive to loss. These are my favorite links on this subject:

Jitter is not the same thing as packet loss (PL). My response is about Jitter not packet loss.

When I said "Jitter and even packet loss are often concealed by the receive window depending on where event occurs" this means TCP has machinery capable of dealing with out of order receipt and sequence gaps in most cases without stalling the stream for round trip or worse RTO. Mentioning PL was intended only to highlight this capability.

It sure as heck does not mean TCP congestion algorithms should ignore congestion indications in the form of PL. Even a small percent PL to borrow a phrase from Joe is "a big fucking deal" shit had better slow down dramatically in that case. I only mention PL at all to illustrate a point about designed capability of TCP.

Real time communications systems generally have a "jitter buffer" reserving some delay up front in order to allocate time to reorder packets if necessary. This simply doesn't exist in any practical way with streaming of static content.

They have been doing this FOR YEARS.

[mitchy]

My favorite is when my allegedly >100MB line is so slow loading web content (not streaming, just web content) that when I go to SpeedTest.net it immediately says "dude you're great at more than 100MB!" but the test results come back before the logo for the page downloads... Essentially I am getting canned results before a 64k graphic can download. Should be called SpeedFraud.net.

Every ISP I've used for the past twenty years have been doing this kind of game. Just like how the airlines oversell capacity. Totally unethical? Absolutely. Illegal? Not really.

Net neutrality is all we got to keep these companies honest, and even that isn't enough anymore.

Re:an anecdote from a Comcast technician

[bugs2squash]

of course I don't know what the mod was. But I don't think the rpi can go that fast. After all the ethernet connects to the cpu over USB2 and the wireless isn't that fast. Maybe they sandwiched a special interface card to the top, but if they can do that they can probably do without the pi.

As the original OP...

[GerryGilmore]

....let me say that I very much appreciate the feedback from the /. community. As usual, a varied mix of great input, doom-mongering and nonsense. What I love about /. !!

Re:As the original OP...

[YomuraOwen]

Who is your current ISP? There are quite a few new broadband providers building in North GA. Maybe competition will to your city too!

Re:As the original OP...

[johnjones]

did you use any of the methods and what was the result ?

It's most likely QoS

[scott_evil]

Having worked in the industry in Australia, I've got a fair idea what the cause is.

I know Australia and the US are a little different, but the general theory remains the same - what you're probably looking at is QoS. They're not slowing the netflix traffic per se, they're just prioritising other traffic such as the speed tests.

Most streaming suppliers will have a big juicy link from their datacentres to prominent peering points. You can pretty much guarantee that it's your ISP and their choice of backhaul or their choice of backhaul supplier who are at fault if you're seeing traffic prioritisation affect your streaming.

BS

[PortHaven]

Comcast deliberately and specifically used to slow down Netflix traffic. Prior to Netflix paying them for peering.

I had Comcast's 25mbps speed package, but couldn't stream Super or 3D content. Bandwith was too slow. Dropped my service down to 3mbps. Netflix and Comcast signed peering agreement. Suddenly, the very next day my 3mbps connection was fast enough to stream 3D content from Netflix.

Ya....don't give me the congestion BS. The telcos very knowingly throttle certain content.

Yup...

[PortHaven]

Greedy bastitches.

Re:3Mbps

[PortHaven]

Comcast deliberately and specifically used to slow down Netflix traffic. Prior to Netflix paying them for peering.

I had Comcast's 25mbps speed package, but couldn't stream Super or 3D content. Bandwith was too slow. Dropped my service down to 3mbps. Netflix and Comcast signed peering agreement. Suddenly, the very next day my 3mbps connection was fast enough to stream 3D content from Netflix.

The telcos very knowingly throttle certain content.

Have you tried The Measurement Lab?

[jpiratefish]

https://www.measurementlab.net... has some good tests. Sadly, some of old Net Neutrality tests have died there - but some of their stuff remains and is good. Back when this fight started, Google backed some groups working to make speed tests that detect if specific slowing was going on - all those tests are at this site - even though the public servers behind them are offline now. Cheers.

Unsolicited Advice

[some old guy]

Comcast has more lobbyists and lawyers than you. You'll never win. Ever.

Unless you just want to prove the point for argument's sake.

Then I bid you godspeed, sir.