Slashdot Mirror

large numbers of dropped packets (esp. packets over 4000-5000 bytes) in exchange for almost-no-dropped packets.

hmmm... try playing with MTU, packets over 4000-5000 bytes?

I have an MTU of 576 in similar setups. VOIP works flawlessly and I have 0 dropped packets. Using qdisc, I use htb, to do traffic shaping is recommended too. The important thing is to keep your cable-modem empty...

http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm

I don't have any complaints for ISPs that throttle Torrents and take other measures against "high usage" users, who are file sharing.

My only issue with throttling is that there are better ways to manage your network than arbitrarily lowering someones bandwidth. My office is located out in the sticks and we can only get a T-1. I have to share 1.5mbits with 60 employees, including time critical services such as VOIP and VPNs. I set up a priority list that looks like this:

1. VOIP
2. Priority packets (small SSH packets, NTP, DNS, pings)
3. TCP ACKs
4. VPN Packets
5. Core network services (e-mail mostly)
6. Unclassified packets from administrative workstations
7. Unclassified packets from other workstations
8. Idle Tier for large downloads that aren't time critical (I use this myself when I need to download large patches, Linux ISOs, etc.)

Each tier gets a promised amount of bandwidth. When they need more they borrow unused bandwidth from the other tiers in order of priority, VOIP gets first dibs, then the priority tier, then TCP acks, etc. etc. I don't arbitrarily limit the bandwidth available to a certain protocol or end user. If the line is free then even bittorrent is allowed to use 100%. If the line is pegged then traffic is allowed through according to the above priority list. Bittorrent doesn't care if it winds up in the queue -- VOIP does.

I did this with an old Linux box and the HTB packet scheduler. It cost nothing but time to setup. If I can manage to do this then doesn't it stand to reason that a company with the resources of Comcast could pull it off? Why do they need to impose bandwidth limits and the like when they could simply prioritize interactive traffic ahead of the torrent kiddies?

HTB is Hierarchical Token Bucket, a CBQ (Class Based Queueing) discipline for Linux. It lets you create a hierarchy of queues for a network link. The "Token Bucket" part means each leaf and node in the tree has a "bucket" that constantly, slowly fills with tokens. Sending a byte removes a token. So, on average, you're only guaranteed the fill rate, but if you haven't used it for a bit, you can send a burst until your bucket is empty. Extra tokens can be borrowed between nodes if they're not used by the others, up to the max rate. Thus you get minimum guarantees, max limits, and bursts, such as being able to quickly fetch a web page even if the link is full from others' usage, if you haven't used up your tokens.

For instance, you could have Customer A, Customer B, and Customer C at the top level, and then they each have a second level of HTTP, BitTorrent, and SSH. Customer A and B get a rate of 128k, and C gets 512k since he pays extra as a business customer. They all have a max rate of 6M, since that's the speed of their DSL lines, and a burst size of 1MB. Then, they have SSH (with a small rate and a small burst), HTTP (with a high rate and a large burst), and BitTorrent (with a 1k rate, and a small burst).

As long as Customer C isn't using any bandwidth, A and B can use it all. As soon as C wants to use some, he first gets his guaranteed 512k - no matter what - and then they all split any leftovers in proportion to their committed rates (So A gets a share, B gets a share, C gets four shares). If C only wants 512k, A and B each get to split all the leftovers evenly.

If A is using BT like mad, but then opens an HTTP connection, it'll be allowed most of his net connection (it has a high rate, but still lower than the full line speed). BT will automatically (and instantly) be throttled until HTTP is done. When he types on the SSH connection, it'll use little bits of its burst speed to refresh the window instantly, but its small rate won't let it consume the whole net if he accidentally cats /dev/urandom.

Sounds great, right? There are a few gotchas: You can only queue packets like this when *sending*. What're you going to do, receive a packet from the slow link and then delay it before sending it over the fast one that's not saturated? (Well, yes, you can, and it makes a limited amount of sense to fine tune TCP's flow control, in addition to selectively dropping packets to make it back off, and other tricks.) It's good, but it doesn't necessarily make optimal tradeoffs between latency and bandwidth - HFSC is an attempt to address this. Also, this is a moderately heavyweight way to do things. It has to spend some CPU classifying packets, and memory to track the buckets' state, so other queueing disciplines and schedulers exist that work on other methods (such as statistical, instead of discrete tracking), that are more appropriate for very large ISPs. Also, as a large ISP, you're going to be using Cisco, not Linux, for routing. :) But Cisco has sophisticated QOS as well.

Despite how complex this sounds, even using the simplest case on your home router will make a huge improvement in the weak side of your DSL line, the uplink. Several of the open source WIFI router firmwares support it out of the box for this reason. I have survived having my web site on my DSL linked to the front page of a popular site known to bring servers to their knees, without any lag in SSH or games, or interruption of mail or other services. We only noticed because our bulk transfers slowed to a crawl, as intended.

Learn more:

HTB: http://luxik.cdi.cz/~devik/qos/htb/ (the user guide has a good overview and pretty graphs)

HFSC: http://linux-ip.net/articles/hfsc.en/ (More pretty graphs and good explanation)

Linux Advanced Routing and Traffic Control list: http://lartc.org/ (The howto is out of date, but very enlightening)

• The ADSL Bandwidth Management HOWTO from TLDP is what got me started. It is quite outdated by now, though.
• The Advanced Routing HOWTO from TLDP. In particular, sections 9, 9.5 and 9.5.5. Note that it was outdated even when I read it, which was several years ago (you haven't had to patch your kernel to get HTB since 2.4.20 or so).
• The LARTC site and mailing list archives.
• The HTB home page.
• The manpages tc(8) and htb(8). (I wish they had existed when I learned about it.)

but you want to tell everything you know about "P2P traffic shaping for home use" and be useful to more people.

This won't be directly helpful to the submitter (he's working with a WRT54G), but this is how I do it in Linux. Set up the shaping rules with tc. Classify traffic with iptables. Examples follow:

(in /etc/ppp/ip-up -- would likely be rc.local or similar file for a cable modem user who doesn't use ppp)

# ADSL connection is 832,000 bits/s on upload.
#
# We rate limit to 632,320 bits/s (76%) to account for ATM/PPPoE/IP protocol overhead.
#
# This is broken up as follows:
#
# 72,000 bits/s for TCP Acks (keep our downloads fast even if upload is pegged)
# 35,000 bits/s for interactive packets (icmp echo/reply, tcp syns, network time protocol, small ssh packets -- only small ones so we don't prioritize scp transfers)
# 236,500 bits/s for priority traffic (traffic to my work VPN)
# 236,500 bits/s for normal traffic (this is the default)
# 35,000 bits/s for low priority traffic (udp trackers in bittorrent)
# 35,000 bits/s for idle priority traffic (bittorrent uploads)

/sbin/tc qdisc add dev $1 root handle 1: htb default 50
/sbin/tc class add dev $1 parent 1: classid 1:1 htb rate 632320bit ceil 632320bit
/sbin/tc class add dev $1 parent 1:1 classid 1:20 htb prio 1 rate 72000bit ceil 632320bit quantum 1454
/sbin/tc class add dev $1 parent 1:1 classid 1:30 htb prio 2 rate 35000bit ceil 632320bit quantum 1454
/sbin/tc class add dev $1 parent 1:1 classid 1:40 htb prio 3 rate 227660bit ceil 632320bit
/sbin/tc class add dev $1 parent 1:1 classid 1:50 htb prio 4 rate 227660bit ceil 632320bit
/sbin/tc class add dev $1 parent 1:1 classid 1:60 htb prio 5 rate 35000bit ceil 632320bit quantum 1454
/sbin/tc class add dev $1 parent 1:1 classid 1:70 htb prio 6 rate 35000bit ceil 632320bit quantum 1454

/sbin/tc qdisc add dev $1 parent 1:20 handle 20: sfq perturb 10
/sbin/tc qdisc add dev $1 parent 1:30 handle 30: sfq perturb 10
/sbin/tc qdisc add dev $1 parent 1:40 handle 40: sfq perturb 10
/sbin/tc qdisc add dev $1 parent 1:50 handle 50: sfq perturb 10
/sbin/tc qdisc add dev $1 parent 1:60 handle 60: sfq perturb 10
/sbin/tc qdisc add dev $1 parent 1:70 handle 70: sfq perturb 10

In order, those commands establish a htb scheduler with a celing of 632,320bit/s (you have to set this around 70-80% less than your actual upload to force the packets to queue at your box and not the dsl/cable modem), then establishs children underneth it for each class of traffic. The children will get AT LEAST the specified rate and when extra is available will borrow it according to their priority number. Prio 0 gets all extra bandwidth until satisifed or no more exists, then prio 1, prio 2, etc, etc.

The second set of commands attaches a fair queuing algorithm so individual connections within those classes will share the bandwidth (more) fairly.

From there it's just a matter of using iptables to classify the traffic. This example shoves all bittorrent traffic into the lowest queues. We assume that anything coming from 172.25.42.254 is bittorrent traffic because we add that as a second IP address on the client behind NAT and make Azureus bind to that IP (all other traffic goes out on the default IP).

iptables -t mangle -N LOW
iptables -t mangle -N IDLE
iptables -t mangle -A LOW -j CLASSIFY --set-class 1:60
iptables -t mangle -A LOW -j RETURN
iptables -t mangle -A IDLE -j CLASSIFY --set-class 1:70
iptables -t mangle -A IDLE -j RETURN
iptables -t mangle -A FORWARD -p udp -s 172.25.42.254 -j LOW
iptables -t mangle -A FORWARD -p udp -s 172.25.42.254 -j RETURN
iptables -t mangle -A FORWARD -p tcp -s 172.25.42.254 -j IDLE
iptables -t mangle -A FORWARD -p tcp -s 172.25.42.254 -j RETURN

Those commands

a packet scheduler built into the linux kernal.

http://luxik.cdi.cz/~devik/qos/htb/

Stupid question, but I've never had a chance to see a FiOS connection up close: Can you put the FiOS routers into a bridge mode and get the globally valid IP directly on your PC? The first thing I've always done with my DSL connections is put the router into bridge mode and run pppoe/pppd on my Linux box.

I'd much rather have the full power of iptables and the HTB packet scheduler at my disposal than use their router. I'd hate to think that I won't have this option when FiOS hits my area.

There are ways to isolate them and limit their bandwidth. Whether or not you'd call them "easy" is up to you.

Requires CBQ or HTB. I personally think it works better with HTB. See http://lartc.org/wondershaper/

Nope, you can use: HTB shaping and IPtables tagging to filter on just about anything you want. port, user, ACK or regular packet etc. At least for outgoing traffic, incoming is a bit harder :) Or use tbf and uc32 filters or cfq's or ...

Linux has rather impressive QoS facilities (so many options that it has an entire submenu in menuconfig). What you'll want is some floppy router distribution (mine's not quite ready yet, but I hear LRP has most of these options) and a decently powerful machine (100-200MHz pentium with 32-64MB should PLENTY for a DSL, but remember, you're not just routing/NATting).

There are tons of preconfigured things out there, but you might want to read up on tc (the traffic control manipulator, part of the iproute2 distribution), ip (from iproute2) and iptables (to help classify packets) before you dive in. The kernel ships with most of what you'll need (including the common CBQ scheduler), but there is a really cool scheduler known as HTB that is more accurate because it's resolution is traffic based, not time based. If you want to shape inbound traffic destined to teh router itself, you'll also need the IMQ patch.

Hope this helps. If you want more info, EFNet #iptables, look for KurD, the human router. He plays with this stuff all day at his job.

--MonMotha

Linux has rather impressive QoS facilities (so many options that it has an entire submenu in menuconfig). What you'll want is some floppy router distribution (mine's not quite ready yet, but I hear LRP has most of these options) and a decently powerful machine (100-200MHz pentium with 32-64MB should PLENTY for a DSL, but remember, you're not just routing/NATting).

There are tons of preconfigured things out there, but you might want to read up on tc (the traffic control manipulator, part of the iproute2 distribution), ip (from iproute2) and iptables (to help classify packets) before you dive in. The kernel ships with most of what you'll need (including the common CBQ scheduler), but there is a really cool scheduler known as HTB that is more accurate because it's resolution is traffic based, not time based. If you want to shape inbound traffic destined to teh router itself, you'll also need the IMQ patch.

Hope this helps. If you want more info, EFNet #iptables, look for KurD, the human router. He plays with this stuff all day at his job.

--MonMotha