Traffic Shaping on DSL?

jackla asks: "I'm now looking for software to do traffic-control on my Windows XP box. I am connected with DSL and my upstream is capped at 96kbit/s (down is 1.5Mbit/s) - this means that high(>70kbit/s) upstream utilisation KILLS my downstream: it just drops down to about 400kbit/s and stays there unless there's more upstream space. That said, I read alot about the Linux shaping solution (wondershaper or something) which sounds exactly right, except I need something that works for Windows. What I want to do is prioritize upstream ACKs (for example) so that my downstream isn't affected by upstream use. If anyone heard of a peace of software that can do this, I would love to hear about it." It would be nice if something like this existed cheaply for Windows. I am unaware of such, but maybe a few of you have ideas. Could such a traffic shaper be built using low powered computers? If so, how would you build and configure it so it would maintain compatibility for the single Windows machine, behind it? (Think: homebuilt traffic-shapping appliance)

Comments

What are you running?

[secondsun]

Are you a server, business or home user? I understand that you want as much bandwidth as possible, but if you are just a home user, 400 kbit down stream is not bad at all.

To answer your question directly, my solution would be to buy a cheap box (like say, the Mandrake boxes from Wal-Mart) and use it as your traffic shaper. Linux products for this are much cheaper than any (useable) solution you can find for windows.

Re:What are you running?

[DrVxD]

> 400 kbit down stream is not bad at all.
Unless you're paying for 1.5Mbit (which it sounds like he is), in which case it sucks. 400k isn't bad, but it's only about a quarter of 1.5M. That's a BIG difference if you're transferring a large file (e.g. a movie), and is even more apparent if you're trying to (e.g.) stream video.

Build a Linux box, use QoS and cbq.init

[fialar]

My recommendation, keep that Windows box behind the firewall box (let the fw box NAT for it). Run Linux or BSD on the firewall box.

If you're using Linux on the firewall machine, make sure you enable QoS and ALL the modules in it. Then grab cbq.init and set up the traffic shaping rules. The script file is well documented.

-F-

Re:Build a Linux box, use QoS and cbq.init

[Wesley Felter]

Linux Advanced Routing & Traffic Control

The Wonder Shaper

DSL Reports...

[Jeffv323]

... has a Tweak Test that tests some connection settings such as your RWIN. I had the exact same problem as you and it turned out that my RWIN was set wrong and once I fixed it, the problem pretty much went away. Try it and I bet it helps.

first thing to find out is..

[trybywrench]

what is going upstream from your pc. You can download an eval. version of Iris ( a sniffer ) and see the traffic originating from your pc destined to outside your network.

You might find something crazy going on because a non-serving pc should be pretty quite. You will see broadcasts and ACK's but thats normal. If your computer is spewing traffic and you can't find the source your NIC could be off in the weeds or you may have been hacked (not uncommon with windows and DSL/Cable). I have 38 IP's in hosts.deny because of detected port scanning on my DSL.

Dummynet

[seanadams.com]

Could such a traffic shaper be built using low powered computers?

Fire up that 90MHz Pentium, install FreeBSD, and build a kernel with bridging and dummynet enabled. Dummynet is an awesome network simulator. Just set up a couple of ipfw rules for the types of traffic you want to limit, and then set the bandwidth parameters in dummynet. It's very easy to do basic stuff like you're describing, but you can do all kinds of other things with dummynet... latecy, loss, queue limits, simulating multiple hops and multipath links with different latencies. There are no tools of this caliber (let alone free!) for Windows.

Next question?

Re:Packet Scheduler

[bogie]

Unfortunatly this won't help him. The QOS packet service is designed primarily for audio/video applications and would require that every router he connected through strictly obeyed the 802.1p protocol. That is just not going ot happen.It also requires a heck of a lot more then just clicking that little check box on the XP client. You can just consider that an on/off switch, the real work is done at the server and endpoint router levels.

Have a look at this link for some futher explaination. http://infocenter.cramsession.com/techlibrary/geth tml.asp?ID=1674

Nope.

[mindstrm]

Unless I'm mistaken, that won't hlep.

What the QoS scheduler does in windows is permit applications who specifically request a certain qos to keep it. Í don't think it's a robus, configurable queuing system.

Bandwidth Limiting HOWTO

[dimator]

Bandwidth Limiting HOWTO might be of some assitance? Or not... either way. It just caught my eye on linuxdoc.

Lots of people complain about this....

[billstewart]

There are a couple of problems that can lead to this symptom, and asymmetric connections are more likely to suffer from it. Basically, if your upstream bandwidth is swamped with file transfers, it's hard for the ACKs for packets you receive on your downstream to get through, so the downstream TCP sessions get throttled. Many DSL providers oversubscribe their links (though the problem is usually worse downstream than upstream)(cable can have this problem also), so that 128kbps upstream you thought you had might only be 38kbps if lots of people are uploading at the same time - so your upstream packets may be queued at the DSLAM, and your little ACK packets have to wait for a windowful of big file transfer packets to clear out, or they may be queued in _your_ DSL router. An 8KB TCP window is 80k bits, so even if you're getting your whole 128kbps upstream, that's over half a second of data that could be sitting ahead of your ACK, dogging your downstream flow rate. It's especially bad if your upstream traffic has a big window size, while your downstream is using slowstart and has never gotten a very big window.

So yes, traffic shaping can be your friend. Unfortunately, it may be hard to know what to tune it too, depending on where the bottleneck is.

Re:Lots of people complain about this....

[WhiteKnight07]

Give Cable Nut a try. It lets you play with all that stuff and well, a butload more stuff having to do with how the TCP/IP stack behaves. It's for windows only but it maybe just what this guy needs. Although its not true traffic shaping it might help solve the problem.

BBIAgent

[t0qer]

I've been running BBIagent
http://www.bbiagent.net
for about the last year or so. I share my DSL with a few neighbors via some cat5 we strung up over the last year, and my counterstrikin was startin to hurt real bad whenever one of the teenager kids was downloading from kazza

Before I was using a linksys router. My current BBIagent setup is a packard bell p60 with 32 megs of ram, and 2 old 16 bit isa 3com nics. The performance increase is stunning.

You can specify what kind of traffic to create priority for, forward, block ports. There are a few options for hack detection blocking. The nicest features of bbiagent is.

1. Single floppy distro which is configured on the fly via the bbiagent website
2. Java app for admining the sucker, very well laid out.
3. Realtime control over your network connection without losing it. On my linkstink whenever I would change a rule or a forward the whole thing would reboot itself, bbiagent doesnt do that.
4. Open source!
5. Linux Based!

I can go on and on about it, but if you're really looking to control what and how packets are handled, I would really recomend giving bbiagent a try.

--toq

You aren't going to be able to fix this locally

[tlambert]

The reason your downstream rate gets limited by your upstream rate limit is because of your inability to send acknowledgements upstream for the downstream packets you are receiving because of the congestion on the upstream link.

Once the advertised receive window is filled with packets, the remote side is not going to send any more data until it gets an ack from your computer.

At the point that the rate is being limited, the way to fix this is to give preference to ACK packets -- i.e. packets without payload.

The problem is that this has to be done at the point the rate is being limited, which is at your providers router, not at your router, and not at your Windows machine.

Installing the Microsoft packet scheduler, or some UNIX box with a real netowrking stack that you can exercise fine control over packet priority, as some people have suggested, isn't really going to fix your problem, except by maybe 10% (the exact amount depends on your average outbound payload size relative to an "empty" ACK-only packet).

This is because the problem is the queued data in the transmit buffer in the rate limiting device not containing your ACK's for inbound data.

Really, the TCP/IP protocol wasn't built for asymmetric reates, without equally asymmetric data transfers.

Effectively, you need to be able to control the transmission of data packets from your end based on knowledge of how full the input buffor on the outbound leg for the rate limiting device gets, so that you can throttle your data payload packets accordingly, to keep that buffer as empty as possible.

The only way you can really do this is to put code in your stack to monitor the advertised window from the other side, and the locally classify outbound packets as to whether or not they contain data payload, or are merely acks. You basically have to avoid filling the outbound queue on the rate limiter above 50%.

In an ideal world, the machine doing the rate limiting would do this for you. Some rate limiting machines for asynchronous connection do this, and it's not a problem (you can see the posts from the people who are rate limited, and don't understand your problem). But those machines are more expensive, and it's just as well for the provider if you feel pain as feedback for uploading, since it serves their purpose in providing you asymmetric service in the first place.

The problem's a lot easier when you are trying to avoid filling the inbound receive buffer for a router with a speed differential on one side (e.g. the inbound receive buffer on a router connected to a dialup modem bank, with a customer on a modem wanting to do QoS based on protocol type, so their SSH didn't lock up when an FTP or large HTTP transfer started up). *That's* where things like "AltQ" and the Microsoft packet scheduling engine become useful... not here.

-- Terry

Re:You aren't going to be able to fix this locally

[Malor]

This really isn't true. This can be fixed locally.

Consider: his problem is that his upstream provider is deciding what packets to throw away, and is making bad decisions.

If he installs a local rate limiter, such that the upstream bandwidth is never oversaturated, the provider will no longer be making decisions about packet drops... he will be making them locally, and can prioritize them however he likes. Since he decides what gets dropped, he can pass the ACK packets first.

That said... doing this with Windows isn't easy. I'm not that familiar with it, but Routing and Remote Access Services *might* have something along this line. I'd suggest perusing the Microsoft web site to see if there's anything like that. (or someone else may post this info here on /.)

At work, we're doing bandwidth management on a 40 megabit connection, using software from www.etinc.com. It runs on Linux or FreeBSD. I'm mentioning it here not so much for this question, but as a general-purpose recommendation for people trying to do bandwidth management on large connections. This software costs about $700 and is closed-source, but it works really well. The earlier version we have totally choked and died, but the 3.21 version does exactly what it says it will.

I'm still not sure, offhand, how you'd use that software to solve this specific problem. You can prioritize and limit based on ports and source/destination, but I'm not sure offhand that you could use bwmgr to optimize based on packet payloads.

Basically, what it REALLY sounds like is that the connection is TOO asymmetric and that he might consider an alternate provider, if available.

Re:You aren't going to be able to fix this locally

[joe user jr]

Nope. It may well work to apply a fix at the local end. It all depends on what actual traffic is causing the problem

The asymmetry of ADSL is a business not a technical "feature" - and it's based not on some wild conspiracy to stamp out freedom and file sharing (as in the response far above this one) but on the sound knowledge that most traffic just is asymmetric - typically you have 1500 bytes of webpage arriving downstream for every 40 byte acknowledgement you send upstream, at least in the use-model that ADSL was constructed on. So it would just be a plain old waste of money to provision for symmetric network use.

Where I am, we have a network of about 80 users sharing a 2M down, 256k up, ADSL connection. This worked fine until some of our users discovered gnutella.

Gnutella is a very bandwidth-hungry protocol, and tends to saturate the upstream bandwidth on ADSL. This resulted in dramatic loss of performance for our web users, for reasons explained above (namely that since acks get delayed, the nagel algorithm on the other end kicks in, reading this as general congestion, and slows the sending rate down dramatically.

I fixed this by installing iptables and the tc ("traffic control") application on the linux box we use for a router. This works using "class based queueing" - you divide up your traffic into several classes, depending what their source and destination IP and ports are (or if they're related to other traffic, with particular ports and IPs). And then you give each class a bandwidth limit (hard or soft).

The way we do this is to use the iptables (successor of ipchains) functionality to insert a "marker" into each relevant packet, and then have tc put them into the appropriate class based on that mark - this gives you all the selectivity (and clarity!) of picking packets that iptables offers.

In our simple setup, reserving 64k or so of bandwidth purely for acks going back to web servers (ie going to port 80) and a few other types of traffic, and a bit of fine tuning, is enough to keep the connection very usable, and let people use gnutella on it as well (at a rate that's reduced a little.)

In the face of constant gnutella traffic, this improved our web connectivity by about 900% rather than 10%. Since you only send a 40 byte ack for each 1500 bytes you receive, a ratio of about 37:1, reserving 64k for acks is enough to cover for 37*64 = (over 2M) downstream traffic.

If you run, say, a local ftp server, you could isolate the traffic from that very easily by marking packets which originate from port 20 and 21 on its IP address (assuming the ftp server is well-behaved and sends its data-connection packets from port 20) and limiting them so that you save 30k or whatever upstream for your other traffic.

None of this needs to be done at or beyond the provider's equipment. Because we limit the rate at which we send traffic to the provider, their equipment doesn't get its queue filled, ever! (unless they're not fulfilling their committed data rate, which we can't really control).

So a local solution may be entirely possible - this will depend on just what traffic is clogging up the upstream.

As for specific software recommendations, I don't know of anything that does this on windows, personally. I suspect it's likely to be payware, and will cost more than an old PC that you can run linux on. We're using a P133 with 32M and I have a feeling that it's slightly overspecified (at least on RAM - I think 16 or likely 8 M would do fine).

Just change the TCP maximum receive window

[Animats]

OK, let's look at what's going on here. The downlink is being underutilized when the uplink is saturated. That just looks to TCP like a link with a long round-trip time, which TCP can handle. But the Windows default configuration isn't set up for it.

Try changing your local TCP buffering parameters instead to allow a bigger receive window. Set DefaultRcvWindow to something like 32768; the default is 8192, which is low for a DSL line, especially one that asymmetrical. With a bigger setting, you can have more data in flight, which makes the TCP connection more tolerant of delays in ACK return.

Given the numbers you're reporting, your ACK delay isn't that severe. You're only losing 2/3 of your downstream bandwidth. So quadrupling the amount of data allowed in flight should overcome that problem. Prioritizing your uplink traffic should be unnecessary at this time.

A real question to ask is "why are you trying to run a server on a 96Kb/s line". Buy hosting from somebody; it's cheap and they'll have far more bandwidth.

Setting up Win2K/XP for broadband

[rabtech]

You can play with these settings to get the best performance, but in general these should help out some.

Note that, at most, simply disable then re-enable the network adapter in question. No rebooting should be required to make any of this take effect.

Keys: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Servic es\Tcpip\Parameters

GlobalMaxTCPWindowSize:DWORD = 256960
This should be a multiple of MSS, which is generally MTU - 40. Best results - pick a multiple of MSS lower than 16-bit (65535) times a scale factor that's a power of two. In other words, pick any multiple of MSS as long as it's under 65535, them multiply that by any power of two to arrive at the TCP Window size (RWIN).

Tcp1323Opts:DWORD = 1
This enables RFC 1323 options, which allows for a TCP RWIN greater than 64k. If you don't do this, most of the other settings are bunk as they will be limited by the 64k RWIN value.

EnablePMTUDiscovery:DWORD = 1
Enables automatic discovery of the MTU for your line, with the MSS set appropriately. You can set this to "0" to force your own value (see below).

TcpMaxDupAcks:DWORD = 2 (range from 1 to 3)
Number of duplicate ACKs recv'd for the same seq number of sent data before fast retrans is triggered.

NOW on to the MTU: it must be set on a per-interface basis. Find your TCP/IP interface associated with your NIC under Parameters\Interfaces\

MTU:DWORD = 1500 (probably... depends on your provider.)

On an unrelated note, you can force IE to hit up a web server with more connections than normal, which can help web pages load more concurrently.

it's under HKEY_CURRENT_USER\Software\Microsoft\Windows\Curre ntVersion\Internet Settings\

MaxConnectionsPerServer:DWORD = 20
MaxConnectionsPer1_0Server:DWORD = 20

First is HTTP1.1+ servers, the other is HTTP1.0 servers. Specifies the max # of connections IE will open to a single web server in the process of downloading a page.

Run the numbers.

[tlambert]

His upstream is 96Kbit, and his downstream is 1.5Mbit.

In round numbers, this means his upstream is slightly less than 1/16th his downstream.

Say his MTU is 1500 bytes. Let's say his ACK packets are 64 bytes (we were generous on the 1/16th, so 64 instead of 60 is not that far off).

That basically means that, assuming no packets are lost, and that all ACK packets are precisely equally spaced, then 2/3rds of his upstream bandwidth needs to be dedicated to ACK traffic for the downstream bandwidth.

Why do we have to assume equal spacing? Because he can't control anything other than his send order, because he can't control the rate limiting machine's discard.

Is it possible to do this with a traffic shaper on the client machine? Yes, with a very sophisticated traffic shaper, which maintains stateful information (e.g. like a PIX firewall maintains per connection state information). It's possible because now we know the numbers for his connection -- we don't know the general numbers, though, for *any* assymetric link, so this isn't something you could make into an installable package, without the user having to resort to math/tuning tools.

Even so... this only works if the traffic is connection oriented. So far, people have asked -- and he hasn't responded -- about the kind of traffic he's running.

If the download traffic is RTSP or UDP or any protocol based on packets other than TCP packets, then there's no way to make preference choices on packets sent out. And then he's back in exactly the same boat he was in before.

So it's not possible to say "install this", and it's not possible to say "install this, and set these tuning values based on your relative upstream and downstream speeds", unless you really limit the problem you are trying to solve.

Doing that will probably not be satisfying, since the primary reason for a (mostly) unidirectional pipe is to push content to users, and most content streaming protocols are not based on TCP or other things which can be stateful.

-- Terry

ADSL tweak guide...

[frleong]

For those interested in knowing how to tweak your ADSL, cable modem settings in Windows, the following link contains excellent and comprehensive information on how to achieve peak download speed: Navas Cable Modem/DSL Tuning GuideTM

So much disinformation

[Nicolas MONNET]

The latency is -- most likely -- caused by the huge buffers in the modem. It *is* possible to improve the situation locally. It's got nothing to do with asymetric lines or somesuch.

It's simple: what happens is that the upstream buffer in the DSL modem does'nt prioritize traffic at all, most likely it's just FIFO and big. So if the buffer is 128kB and you're serving a big file, your next Telnet packet is going to have to wait for these 128kB to go up before going itself.

The solution: have a router that artificially limit the outgoing bandwidth to slightly less that the DSL line permits to make sur the modem's buffer never fills up. Then it's the router's buffers that are filling up; but your router is smarter and you can have it order packet. IE if you have 128kB worth of warez0r waiting to go up, it can decide to let that lone Telnet packet go first.

Me I installed Wonder Shaper, works very well esp. when you've identified what causes the contention (just add the relevant ports to the junk traffic list), even if I completely saturate the link. There's one thing that doesn't work tho: I discovered that at times I had huge ping, again, even with wshaper. What happens (*I think*) is that my ISP is getting overloaded at times, and my actual bandwidth goes below what I set it to in Wshaper. I have to find a way to improve this.