Slashdot Mirror
Ask Slashdot: VPN Setup To Improve Latency Over Multiple Connections?
blogologue writes I've been playing Battlefield for some time now, and having a good ping there is important for a good gaming experience. Now I'm in the situation where I have mobile internet access from two telecom companies, and neither of those connections are stable enough to play games on, the odd ping in hundreds of milliseconds throws everything off. How can I setup a Windows client (my PC) and a Linux server (in a datacenter, connected to the internet) so that the same TCP and UDP traffic goes over both links, and the fastest packet on either link 'wins' and the other is discarded?
(Have your own question for the teeming masses? Ask away — be sure to include appropriate detail and context — via the Slashdot submission form.)
What makes you think the servers can deal with multiple copies of data sanely?
I do not fail; I succeed at finding out what does not work.
A VPN or any kind of encapsulated network traffic will only add to the latency.
Well, for one thing the servers have to cope with that, because internet service in general doesn't guarantee that packets don't get duplicated. But in what OP is suggesting, the servers won't see duplicate packets - the idea is that every packet gets sent out of both internet connections to one, private, hosted VPN server - and that server runs a service that forwards whichever copy of the packet arrives first to the "real" destination (and discards the losing packet) - so the game server will only get one copy of each packet in the stream.
Need to type accents and special characters in Windows? Use FrKeys
This is why local LAN play with your buddies beats some unknown remote server. Plus, then you can keep playing after the central server is taken offline.
What's that? Your favorite game doesn't support LAN play? Well, better support the ones that do, and not support the ones that don't, if you want this option to remain viable into the future.
You should look into multipath networking, IEEE 802.1aq etc. There actually is a company called multipath neworks that sells a hardware solution, but you should be able to find software solutions as well.
That's true, but it seems that the real problem the OP is trying to solve is huge variance in the latency (i.e. jitter) - that is, the idea is to trade a very small amount of extra latency for the latency being much more consistent (without the massive spikes currently being seen). I'm not sure how well it would work in practice (e.g. if some of the spikes are due to local RF interference, it's possible they will affect both connections at the same time), but there's potential at least for a much smoother gaming experience.
Need to type accents and special characters in Windows? Use FrKeys
Your latency and unreliability comes from your mobile links. Get better providers or find a different lower-latency game to play.
Bandwidth != latency unless you are going to send crafted packets to exploit the game
How can I...
Simple. Just write a custom driver on both the Windows and Linux boxes to handle both ends as described (you'll want the traffic duplicated both ways, I'd imagine, since you're not just dealing with one-way communication here).
I doubt there's anything off the shelf that will handle what you want. Sounds like a fun project... but don't undertake this unless you think the project will be as fun to work on as actually playing your game. And be prepared to drop a hundred hours into it (depending on your coding abilities and familiarity with the associated APIs).
Any sufficiently simple magic can be passed off as mere advanced technology.
If you want a hands-off fix, buy a D-LINK DGL-5500, or a ZyXEL NBG6716. They will help keeping your latency low, via Qualcomm's "StreamBoost". Don't use the WiFi for gaming.
For a more Slashdot'ish answer, buy a router that supports OpenWRT and roll your own solution with fq_codel and htb. StreamBoost uses that, and some hand picked traffic shaping rules that Qualcomm will send if you use known games. Most of the profit comes from the fq_codel though.
When you're connected via two providers, you have two different public IP adresses.
You want to send each data packet over both links to some server on the internet, which would relay the first incoming copy of each packet to the game server (or another host). Likewise, the game server sends its data to the intermediate server which would need to send each packet to both your public IPs.
On IP level, this is nearly impossible to do, because target and source IPs would need to be rewritten and the intermediate server would need to be told on a different way what the game/other server's IP is.
But on VPN level, I think this can be done. Start with an open source VPN software and when you're good, maybe 3-6 months of software development will do.
A couple of milliseconds extra for all traffic doesn't matter. The point is to avoid those short spikes in network latency that get me kicked from online servers..
The blogologue
Sounds like an interesting problem; I check the comments to see what solutions to the specific problem laid out might exist. Instead, the comments show varying levels of misunderstanding the question and/or the proposed solution.
The proposed solution is simple:
1. Client duplicates packets over two mobile links to an intermediate, user-controlled server.
2. This server sorts things out and discards the losing packet, and forwards the winner on to the real gaming server.
Both client/intermediate server are under the control of the user, with two possible links. The communication protocol between these two nodes can be user-defined to anything. The question was how to configure this.
"How will the server deal with duplicate copies?" Duh, the gaming server won't. That's what the intermediate server is for. Did you read the OP?
"A VPN will add to the latency." Yes, but that wasn't the problem. The problem is random jitter on one of the links.
"Local LAN play is better!" Well yes, but that's not remotely related to the problem. Maybe he doesn't have anybody local to play with regularly?
"Use this exotic hardware solution." Why, if the problem can be solved for free with software?
"Your latency comes from your mobile links." Duh, but he already measured the main problem to be random jitter. Why not comment on the proposed solution?
The only concern I read that is accurate was that an RF disturbance could interrupt both links.
You may be able to check if your latency varies due to mobile protocol switches to lower power due to less activity.
As a first test, try:
ping www.google -t -l 1000
vs
ping www.google -t
If the 1000 byte ping has more consistent times than the standard pings - that's the issue and you may be able to find a minimum payload size which makes the connection more consistent without switching.
The answer is: no, this can not be done with current protocols.
In theory with new protocols that your game doesn't support, sure. But only the end-to-end machines understand latency and jitter (your problem is jitter) so a middlebox won't help you.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
You mad bro?
http://www.multipath-tcp.org/
of course, this requires the other end to support it, which it probably doesn't.
I've also considered selling "multipath vpn" service... the idea being that people with two DSL providers (and one dsl and one cable) provider would setup their gateway (or use a linux box that I sell them and manage) to send all packets out VPNs on both connections, to my own vpn endpoint in a datacenter. The idea being that then my server on the other end of that connection would take the first packet and send it on to it's destination. Assuming that my datacenter has a good connection, you would suffer less packet loss, and less latency.
My solution here would solve the problem if the problem is latency/loss on your last mile connection. It would not help at all if the problem was further along the connection, while multipath-tcp would
Hi, I'd like to hear a TCP joke
Hello, would you like to hear a TCP joke?
Yes, I'd like to hear a TCP joke
Okay, I'll tell you a TCP joke
Okay, I'm ready to hear a TCP joke
Okay, I'm about to send a TCP joke, that'll last for 10 seconds. It has two characters, it does not have a setting, it'll end with a punchline.
Okay, I'll get your TCP joke, that'll last for 10 seconds. It has two characters, it does not have a setting, it'll end with a punchline.
I'm sorry, your connection has timed out
On the other hand, I could successfully tell you an entire UDP joke, but you might not get it.
Ask me about repetitive DNA
So, just to clarify I believe what the poster wants to do is this:
|||| Gaming Client PC ||||
|||| Local Windows Box ||||
|||| Internet 1 |||| Internet 2 ||||
|||| Hosted Linux Server ||||
|||| Gaming Server ||||
Local Windows Box acts as a router and duplicates all inbound traffic sending it out box Internet 1 and Internet 2. Hosted Linux Box receives traffic, picks whatever packet arrives first and forwards it and throws away the slower duplicate when it comes it.
It is an interesting idea. As far as I am aware routing protocols only do best route and fail over but I am not aware of any that always sends both routes.
I find it hard to imagine that it would (at least routinely) be faster than using his current wireless setup to route his traffic from his desktop ... through his cable modem ... through his ISP ... through a remote datacenter (somewhere) ... to the Battlefield servers
I have no problem at all believing that. The OP says he is using two MOBILE access devices from two (wireless) carriers and is (if I read him right) experiencing substantial intermittent (but separate) delay and/or drop events in both of them. If he throws each packet down both of them and the first one to arrive at the data center gets to the game server, the packets that are lost or delayed on BOTH paths will be very much rarer and his gaming experience will be substantially improved.
Yes, he'll get a little extra latency on the fast packets - which is most of them. But server farms generally have fat and blazingly fast backbone connections, so it shouldn't be a lot added. A small price to pay to make almost ALL packets arrive reasonably quickly and almost NONE experience big delays or loss.
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
Bingo. I had a very similar problem on my train journey tonight, but related to packet loss instead of latency. Both the WiFi on the train and my 3G cut out at various points, often diverging in coverage as the journey progressed. If such a solution, which sounds a lot like channel bonding (as mentioned by another poster) is available, it would seem to cover this scenario as well. Imagine the implications for mesh networking as well - or is this something already dealt with in MN?
This isn't possible, nor should it be.
Actually, it's almost trivial.
I don't know if there's something already available and free, so here's how I'd build it.
It'd startt with OpenVPN. (Mosly because it's the only hackable VPN I'm familiar with that's currently supported.)
I'd first take advantage of the fact that servers and clients are SUPPOSED to be able to handle duplicated and reordered packets and do it the simple way: Just hack it to throw each packet down both pipes, and at the receiving end just forward both copies.
Then, if the server and/or the client DON'T handle the duplication gracefully, I'd add sequence numbering to the VPN wrapping (if it isn't there already) and a mechanism at the receiving end to keep a small amount of history of what has gone out and drop the slower copy (if it arrives before the history of its other copy has timed out).
Data structure might be a small tree or heap of ranges-of-packets-that-have-left (pruned to drop older stuff when getting too nodefull or near sequence-number-wrap-around due to good packet weather), or maybe a rotation of three hash tables - "now", "recent", and "being cleared / cleared and waiting for rotation" - with the first two in use.
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
Bonding was my first thought too - not sure VPNs are necessary, though. I guess this would allow you to isolate point-to-point traffic though, rather than bonding the entire Internet.
You're not asking for a VPN. You're asking for a new routing protocol.
Once you have written a new routing protocol just submit it to the IEEE. Then you have to convince the large router manufacturers to use your standard.
You might want to get a job at CISCO. You would have better luck there.
Load balancing & bonding over multiple NICs. Isn't this what LACP was made for?
No idea if there's anything available for Windows, but chuck a Linux VM on it to act as a virtual router & presto!
How well it works would depend on the LACP stack's ability to handle the issues presented by wireless modems. It works great in a server environment.
And stop stealing your community's bandwidth.
Of course it is doable. It is probably even fairly easy with tun/tap. However, it requires programming; I doubt there are any pre-built solutions for doing this.
Finally! A year of moderation! Ready for 2019?
Take the money you were going to spend on hosting a Linux server in a datacenter and instead use it to buy a decent internet connection, rather than relying on two mobile data plans.
Or give up one of the mobile data plans, and use that savings to buy a decent internet connection.
#DeleteChrome
Seems like reading and comprehending the question is not doable for most of the folks in the comments of this story today. If you go back and read what the original poster asked, I think you'll realize that it's completely doable, with some (perhaps significant) effort. Certainly there aren't any out -of-box solutions that I know of. Basically a combination of mTCP and VPN is what he's looking for. The multipath connection is not between him and the gaming server. He wants it between him and a VPS running linux. The gaming server part is the final goal, but nothing to do with his problem or question. He certainly could invent his own tunneling protocol using, say UDP. As an example, if we consider the tcp/ip protocol, each packet has a unique sequence number. So if we take a TCP/IP packet, wrap it in a UDP packet and send one to the server through each interface, the server could unpack the UDP packet, note the sequence number, and if it already saw it recently, discard it. Otherwise, make a note of it and drop it onto the internet. On the return trip, acknowledgements would have to be handled on the client side. IE if one ack comes, we can safely ignore any others for the same sequence number. If no acks come from either pathway, then it's a standard timeout. This is TCP/IP only. I'm sure UDP could be encapsulated in a similar way, ICMP also probably.
As I type this, I wonder if this could be done by hacking OpenVPN. OpenVPN already has udp encapsulation of UDP, ICMP, and TCP/IP. It would just be a matter of hacking in some support to send the same packet out multiple interfaces at once, and then logic to track and discard duplicates. Not sure how long either hand would have to track things for, or how much would have to be tracked.
Here's a real product that seems to almost do what the original poster is wanting, but not quite. But the it's a similar solution to what I described, but instead of discarding packets to allow the fasted packet to win, it aggregates bandwidth. Different problem, but similar solution.
http://www.pcpro.co.uk/news/br...
http://speedify.com/features/
This kinda sounds like what you're looking for.
Chas - The one, the only.
THANK GOD!!!
What you have described is Connectify's Speedify, it's a VPN that combines multiple internet connections together. As of the latest release it handles both jitter and loss, please check it out: http://speedify.com/blog/speed...
That's interesting. Maybe combined with this: http://www.cs.columbia.edu/~le... - it could provide what's needed. What I need in my case is good handling of TCP and UDP, anything else isn't relevant. Maybe I should try creating some sort of virtual network card that handles TCP and UDP, and hands the rest over to a real network card?
The blogologue
Hey, thanks for the mention. This Alex from Connectify. We've launched a new VPN service called Speedify that combines multiple network connections. It's very smart about jitter and retransmitting lost packets. I think it's exactly what the OP is looking for: http://speedify.com/blog/speed...
Tried it, looked sucky. ;)
The blogologue
Soft ether
Get a real connection, you remind of all the people on WiFi wondering why their gaming experience sucks.
"If any question why we died, Tell them because our fathers lied."
I have run into the same issue with my cable ISP. I run a voip setup using voip.ms as my provider and have my ATA connect to their servers. I have been plagued with random audio dropouts, talk-off and the occasional robot voice problem. After much research, troubleshooting I determined that the issue with jitter my ISP. Most pings to a know good server like Google DNS (8.8.8.8) averages say 40 ms but occasionally (say every 30 pings) the time jumps up to 800 ms. This happens regardless of the server I ping and also occurs when I ping my ISP's gateway address. This tells me that the problem is internal to my ISP and not an external routing problem.
The reason why is what is called Node Congestion. Most North American cable ISP's use DOCSIS with hybrid-fiber nodes located through the geographic area. Nodes may start off with 100 active users on it meaning all 100 users are sharing that piece of the pipe. As time progresses, traffic changes, people ditch their cable tv for Netflix. All of this has a huge impact on congestion and bingo as a result ping times suffer. The average person will never notice but with any time sensitive service like voip and some gaming, you will notice it.
There is not much you can do other than a) complain to your ISP (good luck) or b) find another that's not just a reseller of your existing cable's infrastructure. I'm not sure if DSL suffers the same issue as the shared cable plant.
OK, CEO emailed me with a free trial and link to a new beta, giving it a go.
The blogologue
I don't have an answer to accomplish what you are talking about however....
You can typically get a T1 line installed anywhere.. The prices vary, last one I had was $400 a month but that was about 30 miles from the closest city 10 miles down a paved country road and another mile or so down a dirt road and almost 200 miles from the closest city with an internet backbone. You might find out you can get a T1 or something along those lines for less than I was at that time.. I split my connection with my neighbor..
s/©//g
This isn't possible, nor should it be.
Actually, I know of some people who have built their own network appliances to perform this task. It's feasible and can work but requires encapsulate and decapsulation on each end. You can MSS clamp for TCP and timestamp/reassemble the UDP frames. Not impossible, but certainly requires effort. The people I know who did this was for redundancy between DSL + Business DOCSIS services so they would get the fastest performance of each direction from their links with redundancy should one fail.
I run a dual VPN link over two telcos (Comcast and U-Verse in my case), between my home and a colo. I don't try to repeat the traffic on both links, however, because they have different bandwidth capabilities and it just doesn't work well if the line becomes saturated. Instead I use PF and FAIRQ in both directions to remove packet backlogs at border routers in both directions, and to ensure that priority traffic gets priority. Either an aggregation-with-failover or a straight failover configuration works the best (the TCP connection isn't lost since it's a VPN'd IP). That way if you lose one link, the other will take over within a few seconds.
The most important feature of using a VPN to a nearby colo is being able to prioritize and control the bandwidth in BOTH directions. Typically you want to reserve at least 10% for pure TCP acks in the reverse direction, and explicitly limit the bandwidth to just below the telco's capabilities to avoid backlogging packets on either your outgoing cablemodem/u-verse/etc router or on the telco's incoming router (which you have no control over without a VPN). Then use fair queueing or some other mechanism to ensure that bulk connections (such as streaming movies) do not interfere with the latency for other connections.
In anycase, what you want to do just won't work in real life when you are talking about two different telco links. I've tried it with TCP (just dup'ing the traffic). It doesn't improve anything. The reason is that one of the two is going to have far superior latency over the other. If you are talking Comcast cable vs U-Verse, for example (which, the comcast cable will almost certainly have half the latency of the U-Verse. If you are talking about Comcast vs Verizon FIOS, then it is a toss-up. But one will win, and not just some of the time... 95% of the time. So you might as well route your game traffic over the one that wins.
-Matt
Peplink works like magic - failing over very gracefully. The same can be achieved on Linux through network interface bonding, or on pfSense through Link Aggregation. You would need an intermediate server on the internet that supports the same. VPS servers are cheap and suitable for this purpose.
However, all this will not help reduce latency - which is what the original question is about.
For that, we need the solution proposed else where on this thread:
1. Client duplicates packets over two mobile links to an intermediate, user-controlled server.
2. This server sorts things out and discards the losing packet, and forwards the winner on to the real gaming server.
For mobile internet connections... for dual mobile internet connections. I haven't done that but I have used VPNs over mobile hotspots extensively. There is just no way to get low latency even over multiple mobile links. The main problem is that the bandwidth capabilities of the links are fluctuating all of the time, and if you try to dup the packets you will end up overloading one or the other link randomly as time progresses because the TCP protocol will get acks from the other link and thus not backoff as much as it should. An overloaded mobile link will drop out, POOF. Dead for a while.
For VPN over mobile links, the key is to NOT run the VPN on the mobile devices themselves. Instead, run it on a computer (laptop etc) that is connected to the mobile devices. Then use a standard link aggregation protocol with a ~1 second ping and a ~10 second timeout. You will not necessarily get better latency but it should solve the dropout problem... it will glitch for a few seconds when it fails over but the tcp connections will not be lost.
-Matt
1. Set up OpenVPN on the datacenter Linux server to act as your Battlefield endpoint (single IP + port).
2. Set up 2 VPN connections, one from each phone, to that box.
3. Set up a Linux box to act as a router at home. Use bluetooth or whatever to connect it to your mobile connections.
4. Follow the directions here http://www.lartc.org/autoloadb... on how to set up iptables rules as needed on both Linux boxes.
5. Modify the iptables rules as needed to your specific requirements.
6. Keep on modifying iptables. It will take days to work out all the kinks.
7. Verify that your latency problems still exist.
Old people fall. Young people spring. Rich people summer and winter.
I have used Talari, it rocks. Multiple Internet path aggregation between Talari appliances and instant failover when links have problems. I haven't really seen anyone else do this, but I've heard that Fatpipe and possibly Riverbed are playing in this space now.
The easiest would be a script like this: 1. Connect to one of the VPNs 2. Run a generic performance test (ping, dropped packets, jitter, t/x rates, etc), preferably against the Battlefield server (or the same neighborhood as the server) 3. Store values of the performance test 4. Repeat using the other VPN 5. Compare values, use the VPN that has the best values from the performance test I doubt that using VPNs as a "duplex" is possible, or necessary, even. Just get a VPN that's as close as possible in terms of network distance to reduce the tension your network is receiving and you (probably) won't have to deal with any of this; get one with a high quality route and it *might* improve your ping. If the VPN doesn't solve your problem, you should definitely consider a ISP switch, as painful as it may be.
It can't be done. Advertisements for IP addresses will only let the information be sent to a single IP address.
Technically if a server supported a specific configuration it might be sort of possible, like the battlefield server you were connecting to, and I don't mean the actual server but the software, but the reality of it is what many people have said, which packet arrives first, which gets priority, packet loss could occur, all sorts of weird things.
Instead of thinking complex solutions, you could also think of simpler solutions. Why don't you focus on improving your mobile connection.
Like: make extension cord to tether your phone, and place the phone near or even outside the window.
Or, buy a 'real' (seperate) G2/3/4 modem with a big (and seperate) antenna for $150.
Or. See if you have local interference. Or, see if another type or brand of phone has a better connection.
And of course you already stripped all apps from your tethering phone and disabled wifi, as your phones processor isn't that fast and may easily be stale to other tasks for a few hundred ms.
Also, you could / should check which provider has the strongest signal at your place, may well be a 3rd provider.
I'd seek solution in optimizing one mobile connection. My personal experiences with tethering are that in general it is actually more reliable than wifi, with less latency and less packet loss, but obviously this may vary depending on your location. However, i'd go for all 'low tech' solutions first, starting by putting you cellphone antenna at the most optimal location, like the roof of the building (...).
A glitch a day keeps the bugs away.
Long story short: it won't work. First you would have to convince the device with the two wireless interfaces (the windows PC I suppose) to send the packet on both path. Good luck with that: your typical routing software, including the one in the windows kernel, will choose whatever route it thinks is faster and stick to it. At most you can get some sort of load balancing but it's not what you are describing. If you somewhat manage to duplicate the traffic, TCP should handle that without problems. The impact of dropping half the incoming packets on network performance remains to be seen, however. UDP on the other hand has no way of knowing the absolute sequence of a packet (it's the whole point of UDP after all) and it's left to the in game protocol to deal with duplicate data. I suspect it will fail at some point, defeating the purpose.
... and reminds you that this is exactly the wrong use for port channels.
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
Ignorant troll remarks notwithstanding, this idea is sensible and doable, but requires deep understanding of TCP: e.g. yes it is supposed to be robust to duplicated or misordered packets, but no it will not perform acceptably with even low incidence rates of either. Symmetric PEPs w/explicit multipath packet processing will be needed: the Windows client is not a good candidate for one end of that pair; a lightweight Linux based transparent gateway, using e.g. tun/tap and netfilter, with some coding, can do it. We have been doing similar things for disadvantaged mobile wireless platforms (e.g. aircraft in flight) for years. Beware: transport mode IPSEC, if running between the ultimate client & server, makes it much more difficult (requiring guesstimates of what is happening with opaque transport layer headers). With an IP tunnel over a Type II Hybrid ARQ/FEC transport over packet by packet concurrent multipath routing, you can accomplish the OP's goal & more, but overhead is more than 100% unless you have 3 or more paths.
http://ombakrindu.heck.in/ and http://medium-service.heck.in/
You should probably have your proxy choose just one path for the initial connection setup and then after some configurable number of packets start the flow cloning process to the secondary route. You want to make sure that the server has a chance to get whatever house keeping it does at connection setup time completed before you start relying on the magic of TCP to keep the server from going insane. If you mess with the connection too early you are likely going to mess something up in game's connection setup process. If you send the very first SYN packet twice there is a good chance the server will reset the connection. Then you are going to have to start adding TCP protocol logic to your proxy which is going to make it way more complex. You will need to peak at the TCP sequence numbers when deciding what to pass back to your client from the server.
UDP is a simpler protocol and therefore more complicated for you to handle. You won't have a sequence number and you will need to hash the contents of every packet coming from the server and only pass packets back to your client that you haven't seen before. And of course you will need some sort of expiration on the hashes.
Without fully implementing the TCP protocol in your proxy you can expect issues from time to time, particularly when you pause play and a reset might slip in at the TCP protocol level. But you should be able to create something that works most of the time pretty easily.
Good question!
Great tools looks here
I can't see that they're optimizing latency on single placket level. Just the regular link aggregation and failover stuff.
It seems to me that this is real!
Great here android apps
I'm no expert, but I think this would work better for what the op is trying to accomplish. I've seen it work under linux. It's called PEPsal. This will improve performance. http://sourceforge.net/project...