Slashdot Mirror
Ask Slashdot: How To Diagnose Traffic Throttling and Work Around It?
Aguazul2 writes "I live in Peru and use OpenVPN to connect to my own Linux VPS in the UK for non-live TV. Recently the VPN connection has slowed to a crawl (5% previous rate). Further investigation shows that all connections to my VPS from Peru (even HTTP) are equally slow, whilst the rest of the 'net seems fine. My VPS host says they do no traffic shaping, and connections from Germany to the VPS are fast. This leaves the NSA and Telefonica (Movistar) as suspects. Could the NSA be slowing all VPNs to/from South America because of Snowden and Greenwald? A traceroute shows traffic going through domains with NYC in their name — are my packets being indefinitely detained in transit? Or maybe it is Telefonica and their Sandvine traffic management? Either way this certainly isn't network neutrality, especially on an 'unlimited' plan. Is there a way to tell for certain who is throttling me? If Telefonica have throttled traffic to/from that one IP address, what options do I have to work around it? It seems that separate connections are throttled independently, so can I multiplex over many UDP ports without having to hack OpenVPN myself? This is really frustrating, especially with two untrustworthy parties on the route. I wonder, is this kind of mess the future of the internet?"
Try breaking free of the binary straightjacket. I transmit all my data in ternary and it is untraceable and unstoppable. This gives me unlimitered bandwidsh to post my brilliant world-changing essays and thoughts on Slashdort, the Facebook of the Internet!
UNITE with the Campaign for a Free Internet because today, our future begins with tomorrow!
I've had a client I provide consulting for suggest that their poor connectivity is also in some way due to the NSA. People need to understand that it is paramount to the NSA that they are covert. They do not need to do real-time processing of the data: that is only necessary for filtering. It suffices for them to simply capture raw data for later analysis or decryption as necessary. Of course capturing data does not result in any slowdown or other noticeable effects. It does not make any sense whatsoever for the NSA to be slowing or otherwise blocking connectivity, as that is counterproductive to the acquisition of intelligence data.
It's just amusing to me to see NSA as the scapegoat of the day for any quirk anyone experiences related to computers or connectivity in general.
Better known as 318230.
You are seriously lacking basic data telecommunications experience. All government tapping is span port based. This means that it is passive, not active, so there is no latency involved.
Years ago I worked on a broadband remote access server and one requirement we got was to support lawful traffic interception. Basically all law enforcement wanted was a copy of all of the packets. Packets are not slowed down or stopped by this process.
In my case the hardware was just not capable of doing what was needed but there was plenty of off the shelf hardware that could be installed in the network to provide the filtering and packet mirroring needed.
It is possible that one of the VPN's upstream providers is running into congestion. One of the best ways I have found is to use traceroute. At one time I was getting unusable Internet connectivity through AT&T after they acquired my local cable modem network from @Home. It took them many months to discover that throttling all aggregate upstream traffic to 128Kbps is a bad idea. As much as people bitch and moan about Comcast, it is lightyears better than anything I got through AT&T. In this case, traceroute clearly showed where packets were getting delayed and dropped, which was one of the routers inside AT&T.
Unfortunately, for a VPN this is much more difficult since the Internet hops are hidden via the tunnel.
There are many different ways to tunnel traffic. If the tunnel is Microsoft's PPTP protocol then it's not very secure. If on the other hand it is using IPSec then it should be a lot more secure. There are also other tunneling protocols that do not specify any encryption, i.e. MPLS.
This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
- that the (NSA?) taps are one-way feeds, not redirects/bounces. We just put up two local time-lapse job site camera feeds, and the already routes show one-way feeds from San Francisco, straight to Virginia. The feeds originate in the North West...
My office Internet connection recently went from about 30Mbps down to 1.5Mbps, then back to 50Mbps a month later. No explanation, and speed tests to our ISP all came through at full speeds. We only saw problems on routes going outside our city and headed west. There were also a few inaccessible sites, but those were in very specific local areas. Ultimately, the best guess anyone could come up with is that a network to the west of our city had some routing problems.
We weren't the only customers to complain about a slowdown, but our ISP couldn't really do much about it. The Internet is made up of many networks working together, and sometimes shit happens. I wouldn't jump so quickly to assume it's non-neutral throttling or the NSA, when it could just be a careless guy with a badly-aimed backhoe. Give it some time, see if it improves, and if not, it may be time to move your VPS.
As an aside, you're likely going through New York because that's how you're reaching Europe to get to your UK-based VPS. Many transatlantic cables end in New York City, mostly because the stock market pays dearly for the few nanoseconds of lower latency.
You do not have a moral or legal right to do absolutely anything you want.
You're being throttled.
why would they care about your pirated or whatever TV?
a super secret US intelligence agency that employs some of the smartest mathmatecians in the world is going to care about people's pirated movies instead of tracking down our enemies so the military can kill them
If you are a US ISP, it is required that you have monitoring in place. If you don't want to hamper your entire infrastructure while doing so, you get a bunch of taps and install them all over your network. One very good provider for this is Gigamon. Taps do not add any latency in your traffic. They are completely invisible to all other network devices. Traffic shaping (throttling) is done by the source typically but can be done at the destination ISP. Basically, your connection is assigned a Package in the Shaper. The packages determine how fast each classification groups of traffic are allowed to go. Classifications are determined by whoever manages the shaper for that ISP. Shapers can also dynamically change the speed you are allowed to have for a classification group based on bandwidth used, time used, and volume of traffic.
If you are not throttled from Germany to your home but are from Peru to your home, chances are you are throttled from your ISP in Peru. It is typical for transits to cross borders, so your traffic going through NYC is normal. BGP (the routing protocol of the internet) determined that to be the best path. This is mostly managed, but is still fairly dynamically determined by the routing protocol.
Course of action: Switch ISPs, get a new IP address (if they are not very good at configuring a shaper this will work, otherwise not), try a proxy, stop using it for a day or more and it will go away (temporarily most likely). This is done dynamically in the shaper. There is not some dude with his finger on a 'throttle' button. Everything is automatic. Just figure out the how their throttling deterministic state diagram works and you can avoid throttling. Most likely they are throttling you because of your volume of use. It costs a lot for transit access and you are using more than most others by streaming through a vpn.
Seriously, get a grip. Your precious little VPN is something they do not give a single flying frak about.
IF they did, you would never know. Duping a packet to another port for the NSA costs you exactly 0 in latency. Its done in silicon, and its no different than a broadcast packet as far as the hardware is concerned, i.e. 0 performance penalty.
You're pointing fingers at people and you have no clue whats going on. I can say that safely from your post.
As they say, when in America ... when you sound of pounding hooves ... you don't look for Zebra's, you look for horses.
I suggest you look for a more sane reason, start by dropping your paranoia.
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
I work in the ISP industry, and here's my $0.02...
The NSA (or other spies), not likely. Everything I have ever seen about what they do is passive monitoring. What that means is that somewhere there is a pretty dumb device (like an optical splitter) that takes one signal and makes two copies, one goes to the NSA, one on to its destination. In this arrangement there is no way for the NSA to inject data at all, including slowing it down. I am highly skeptical any government spying is the direct cause. It may be indirect, I'll come back to that in a minute.
Rate shaping is entirely possible, and would be most likely in your immediate provider. It's entirely common for residential consumer ISP's to employ products like Sandvine, or even more crude QOS controls to rate limit particular types of traffic (e.g. VPN or VOIP). Most won't admit to what they are doing as well.
Rate shaping is less likely, but possible at the country level. This is seen mostly in countries with strong government controls on technology (think Iran, China, North Korea). Egypt was doing it at one point in time. I'm not an expert on Peru, but I would not expect this problem in Peru.
Lastly, is plain old congestion. Likely your ISP has multiple paths to reach Europe, riding undersea cables. These are the most expensive assets an ISP owns, and often get congested before they get upgraded. It's entirely possible for instance there is one cable they use from South American to Western Europe that is congested, while another goes from South America to the US and is fine. You can probably map these routes out by traceroute, and may find that particular routes always show poor performance. This also happens, but to a lesser degree, where two ISP's meet. There can be peering disputes, or one customer may not order enough capacity from their vendor. Either way the result is full ports that degrade service for everyone passing through them.
Now, here's where the spies come back in. If a particular spy agency decrees "all new connections must have our spy apparatus on them" they can in fact be the delay to a new connection getting set up. It's not that they are delaying any packet traffic once it is up, but rather they are delaying the installation by not having their equipment ready on time for a new connection. I don't think this happens often, but I'm sure it does happen in some places.
So sadly, this is probably some plain old incompetence/bad luck. Someone either could not afford a timely upgrade, or didn't correctly order an upgrade early enough to get it installed before there was a problem, and there's now congestion somewhere. If it's not bad luck it's probably your provider deciding your particular type of traffic is "bad", and should be rate limited down.
Some more info would be appreciated. So, here's the basics of a few things you can do to make sure it really is the network*. First use iperf on the client and server. Test it on both the tunnel interface and the WAN interface. Second, use top via a separate ssh session. Make sure OpenVPN isn't eating all your CPU or memory. Lastly, what provider are you using? Lately the default Debian build that Edis.at gave me needs an ifconfig up/down every other day.
I've had a similar problem when using my own VPS as an HTTP proxy via OpenVPN. It turned out, the proxy application was crap. Allowing the machine to route packets and using it as a default gateway for all traffic fixed the problem, or at least worked around it.
Now. If it really is blocking, there are a couple of ways around it. The more complicated ones involve using some other VPN application. When dealing with more than one client, that rapidly becomes annoying. A simple one is using an SSH connection as a SOCKS proxy for your browser. It's not elegant, but it works. Another way is to mask your OpenVPN connection by encapsulating the UDP or TCP packets. Once again, SSH port forwarding works, but that's a TCP solution. socat was designed to do things like that, so it seems like a good choice. Finally, there's Ping Tunnel. It embeds traffic in ICMP packets.
Whoever is throttling you might detect one or more of these, but they're probably using some sort of signature based detection. Just about anything that requires a command line should get through.
Remember, since you are technically savvy enough to roll your own, you are the one percent. Good luck, and please let us know how it goes.
*I know you're probably familiar with all of these things. Just assume that I put this section here for those who aren't.
So lets pretend that we've just completed writing this code, as opposed to having just completed sabotaging it -Altera
sometimes when I wake up, there's white goo all over my penis. It wasn't there when I went to sleep! Do you think the NSA is breaking into my house and doing something to me?
You might be able to tell which hop is slow using something like pchar: http://stromberg.dnsalias.org/~strombrg/network-performance.html
Who gave a slashdot account to that computer trained to tell jokes?
That may have been their theory, or it may have been they wondered if US gov was intentionally slowing VPN connections from that part of the world.
If the theory was that capturing data would slow it down, the answer is "no". For that, you'd use port mirroring. Where a switch or router would normally take data in on one line and output it on another, you set it to accept data on the one line and output it on TWO others simultaneously. The data still flows at the same speed. It just flows to two locations separately - the intended recipient and the government.
| It suffices for them to simply capture raw data
... and you woun;dn't let this terminology pass with "piracy" because that involves depriving someone of their property ....
Ok, so the same people that say it can't be piracy because no one was deprived of their DVD give a free pass to "The NSA is capturing the data"??
They didn't capture the data, because if they did then when did they release it? It wasn't like they were tagging an antelope and then let it go at some later time. Why do you give a stamp of approval that the "NSA captures data" as if they held it hostage at Gitmo and wouldn't let the datas go unimpeded.
It isn't like they detained the data without a warrant and won't release it --- they let it go freely. You guys are acting like they are backing up your data stream like some fat dude that is clogging the toilet
Priest: "Universe from nothing, no laws of physics, sped up time"+ huge discrepancies. Creationism? No. Big Bang Theory
the ISPs will buy off Congress, meanwhile even suggesting we regulate the ISPs to enforce net neutrality is met with jeers about bureaucracy. Way I see it we're damned if we don't in that scenario, but I'm in the minority :(.
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
Many ISP's perform what is known as ICMP rate limiting. Traceroute and Ping both use this ICMP protocol *i'm not going to get into semantics* where as you start traversing the internet past your internet service provider your pings and such to any point along the path have a high chance of being dropped due to this. The only way to see your actual latency is using a host-to-host ping. From your source destination to your final destination. Traceroute acts as sending a ping to each and every hop in between the source and final destination (assuming the TTL doesn't expire or somebody's carrier firewall just doesn't' start letting replies come back through, ie, multiple * * * responses but still able to reach your end destination), they are in no way obligated to reply properly and or in a timely fashion to your Ping request. During the early days of the internet we didn't have many of the problems that we have today and these tools worked flawlessly during this time and really could tell you where your latency is (these tools still function normally in a local lan if you are not doing any "crazy" firewalling tactics). This is no longer the case with ping an traceroute.
IN EXTREME CASES it may be possible to route around other carriers using private tunnels, It's not something your average joe will not likely be able to accomplish without multiple services across the country or paying for some sort of service to do so. AKA you are a business with $$$$. There are instances where it can be done, but are few and very far in between.
If your ISP only has 1 way out to reach specific destinations which are having problems. Provide them traceroutes showing them good responses AND bad responses from when and where you are seeing the problem. The only thing a carrier is going to care about is your "average" response time in milliseconds, not your "maximum" response time.
Paranoid much? They only make copies of the data to process off-line, they don't insert themselves into the data stream to do it in real time.
---- Booth was a patriot ----
Use OpenVPN in TCP mode (rather than it's default UDP mode).
Then set up local ssh port forwards through a bounce host you know works well.
Instead of going from Peru --> UK instead go from Peru --> Localhost --> SSH bounce host in Germany --> UK.
Or try an onion network like Tor.
Do it for da shorties
Martin Bishop: Sorry to waste your time, gentlemen. I don't work for the government.
Agent Wallace: We know. (flashes a badge) National Security Agency.
Martin Bishop: Oh. You're the guys I hear breathing on the other end of my phone.
Agent Wallace: No, that's the FBI. We're not chartered for domestic surveillance.
Martin Bishop: Oh I see. You just overthrow governments. Set up friendly dictators.
Agent Wallace: No, that's the CIA. We protect our government's communications. We try to break the other fella's codes. We're the good guys, Marty.
Martin Bishop: Gee, I can't tell you what a relief that is, Dick.
Courtesy of Sneakers (1992) (video clip of the above here)
You're misunderstanding what PRISM supposedly does. (And you're not the only one.) PRISM does not cause any delays whatsoever - it's not a man-in-the-middle attack. It's simply a copy of all traffic on a fiber. Also an old fashioned "tap" on your Internet connection (usually port mirror at the ISP or Internet exchange) does not cause any delays.
Switch to a different VPS provider.
Such a thing would be ridiculous and childish - however things like the diversion of an aircraft that didn't even have Snowdon on it show that the NSA is being ridiculous and childish. Instead of toy soldiers and a way to funnel money out to friends in the private sector the task should be either handed over to military professionals with a focus on things that matter or abandoned entirely. Collecting more data than can be sorted let alone interpreted is a waste of time that just provides a false sense of security.
If I was a law enforcement agency, I would certainly consider slowing down VPN's just to discourage people from using them. So much the easier for me to snoop.
Try this service and see how it compares to yours:
https://www.vortexvpn.com/
See if you get the same behaviour. You get 1GB of free data, if you email support I can give you more. I could also open port 443 if they seem to be shaping non-Http(s) traffic. I have had it running for a few days. There is a server in Dublin you could use.
http://www.independent.co.uk/news/uk/home-news/time-for-a-change-as-mod-staff-run-up-40000-speaking-clock-bill-8782535.html
Ministry of Defence (UK) employees spend £40000 on illicit use of the speaking clock.
Down the hall, GCHQ is listening for free.
This perpetual motion machine Lisa made is a joke, it just keeps getting faster and faster. - Homer
Didn't you know? Slashdot is a large Turing Test system. Most of the participants are AIs.
Interestingly, the most promising test results are with the "First Post" trolls. Apparently nobody can imagine that an AI could be that stupid.
The Tao of math: The numbers you can count are not the real numbers.
I had a similar problem with O2 Telefonica, over 3G, in Czech Republic. Their FUP is quite bad. After you reach the imposed limit, they will throttle *all* connections individually to something like 4-5KB/s. Using OpenVPN, or even just HTTPS was impossible.
However, I noticed that HTTP connections were allowed a throughput 4-5 times higher. It's still very low, but usable. My guess is that they separate HTTP connections from everything else. Note that using OpenVPN over TCP port 80 did not help. So, I've started using OpenVPN over httptunnel. While it has some problems, it did offer me an overall better throughput. The downside is that you need it server-side too.
Bottom line, try httptunnel
Did you know that in most cases, you only need to bypass whatever method is used for checking your location. The server that does this, is usually not the one you stream your video from. It means that after passing the location check, you can actually connect directly to the video server for watching the video itself (and suffer much less from connectivity issues, if at all).
Look at this trick for example.
"Basically we are interested in proxying content only for certain domains. The actual streaming media sits on CDN networks and is usually not geo-locked. The amount of proxying we'll end up doing will be relatively insignificant compared to a VPN-based setup."
In case you want to try it out, there is a free service that does it. I'm a customer of a paid one which combines both VPN ("ibVPN") and DNS ("ibDNS") based services. On the paid front there are many other services that offer similar functionality. Most offer several hours of free trial, so you could see which ones works best for you.
Having said that, did you try contacting your ISP for support? Perhaps they change something in their routing tables which happens to work very bad for you? Maybe they can help.
Maybe the project I've been working on could be usefull to you.
MLVPN can do what you want by creating multiple connections and aggregating them together.
You can find it on https://github.com/zehome/MLVPN Let us know if it's usefull to you or not!
Somebody mod this cunt down into oblivion, please.
I've calculated my velocity with such exquisite precision that I have no idea where I am.
The OP mentions Sandvine: the EFF has a tool called Switzerland.
Is your ISP interfering with your BitTorrent connections? Cutting off your VOIP calls? Undermining the principles of network neutrality? In order to answer those questions, concerned Internet users need tools to test their Internet connections and gather evidence about ISP interference practices. After all, if it weren't for the testing efforts of Rob Topolski, the Associated Press, and EFF, Comcast would still be stone-walling about their now-infamous BitTorrent blocking efforts.
Developed by the Electronic Frontier Foundation, Switzerland is an open source software tool for testing the integrity of data communications over networks, ISPs and firewalls. It will spot IP packets which are forged or modified between clients, inform you, and give you copies of the modified packets.
Switzerland is designed to detect the modification or injection of packets of data traveling over IP networks, including those introduced by anti-P2P tools from Sandvine (widely believed to be used by Comcast to interfere with BitTorrent uploads) and AudibleMagic, advertising injection systems like FairEagle, censorship systems like the Great Firewall of China, and other systems that we don't know about yet.
Wasn't the whole reason one of the NSAs main schemes was called PRISM because it described the process they used to capture data. They would have optical fibre cables run through a junction box which would "split" the signal towards both the intended destination, and NSA hardware, therefore acting like a "prism". This therefore would both not affect latency, and not lower throughput.
Thanks -- I'll try port 443 if/when they unthrottle me. Yes, I also get packet loss when there are storms. I guess they interfere with the microwave links over the mountains. Anyway, that is irrelevant as I get good bandwidth on all connections apart from that one IP so the weather doesn't explain it.
I rather doubt that the NSA is the cause for the loss in throughput but if it were I can only see on reason why. While many others have pointed out that replication of your data for "intelligence" purposes would be unlikely to cause a lower throughput because replication in and of itself would be pretty much instantaneous. They are likely to have that kind of equipment and storage at a limited number of locations (thankfully, for now at least). Your traffic (along with many others) could be getting artificially routed to one of these locations for replication. This being government work they probably spent hundreds of millions on the facility, but were cheap with the fiber going to it, creating a bottleneck.
It seems that I was wrong about the multiple connections getting more bandwidth, so unfortunately MLVPN won't help me -- but thanks all the same. I was looking at multi-path in the past when we were considering moving to a distant village which only had slow 512kbps connections, to tie several of them together. This definitely has its use cases! I've made a note.
First, my piece of advice. Hire someone who know what is doing to debug this situation. Now for my suspicion. I wouldnt be much surprised if the Peru provider has some data/monetary limitation and just optimise the most common traffic. This often is done with deep packet inspection at the layer 7, so i doubt it would be easy to try to work around it, besides changing providers.
Their function is to *look* like they are tracking down actual enemies to national security while they really track down ordinary criminals, political opposition, and economic competitors in so-called allied nations.
Creating just a little bit of doubt in the public, without actually compromising their theoretical secrecy, accomplishes that.
A decent SOHO hardware switch does port mirroring. I just paid $99.99 for a Netgear switch which will mirror at full speed.
To do network mirroring like that in software you'd pretty much need to be flat broke or incompetent. As in totally, government style incompetent. Oh, yeah I suppose you have a point then. :)
For what it's worth, Telefónica is notorious for doing LOTS of testing different ways of throttling, caching, blocking, accelerating etc... (and not being that "great" at it)
The general INTENT is not "omg, they want to block me from doing things!!!", but rather they are trying to save/optimize on bandwidth. As far as I know, they have at times been known to block SOME traffic they consider "voice" as it comes in conflict with their main business line, but mostly this has been tried and then stopped as it generated more headaches than cash.
With all the caching/accelerating, etc, a LOT of times they mess up with ICMP packs which handle testing MTU, in conjunction with changing the actual MTU of the links, the result of this is that your kit sends larger blocks than the links can handle and then they get mashed/munged during fragmentation/reassembly. And the consequence of that is that a lot of "real" packets don't get through (often the ones on or around the *perceived* MTU limit), so your data then behaves as if it was working on a VERY lossy link (imagine around 40%+ packet loss...). You won't see problems with a regular ping, but you may if you check with ping sizes around the MTU limit.
At other times, as Telefónica is trying to optimize using DPI (deep packet inspection) to check what protocol is being used, they may not correctly recognize your traffic and thus munge it in someway. Effectively acting as a throttle, but not because they actually "want" to throttle.
What can you do about it? Not much, because all of this is handled by the inner sanctum of the tech-priests and they don't communicate with mere mortals such as tech support of commercial reps, there's no way to get through...
With a regular residential grade link, the attitude from Telefonica is "take it or leave it as-is, we don't care, this is what we give". In general they are valid for the purpose, but if you want business grade quality AND the possibility of complaining (and being heard), you need to get a business grade link (ie ditch Speedy and get info-internet at 5x the price with 1/4 the speed).
It's nice to WANT to notch it up to "they are throttling me" or "NSA is spying on me" and any other conspiracy theory, but once you mention Telefonica it's more a case of "Do not attribute to malice that which can be explained by incompetence".
(and yes, NSA is probably spying on you ANYWAY...).
The above come from a lot of experience with different telco's, a lot of contact with people inside telefonica and seeing how telefonica operates in quite a few countries (including Spain and Peru). Just take it as face value, I'm not trying to prove a point. If it helps great... if not, well, good luck with other venues...
My $0.02
I know you guys hate the French for reminding you that they gave you a country every time the US tries to put pressure on them but you are going a bit beyond freedom fries here. Look up extraordinary rendition and you'll see that US agencies had enough clout to get away with abducting people all over Europe while European governments pretended to look the other way.