Slashdot Mirror

Your acknowledgement packets probably aren't getting through.

http://www.benzedrine.cx/ackpri.html

Blacklists can be great if they are effectively managed. A dumb text file that you drop IPs into and forget about is lame. There are easy to use tools like relaydb in any *BSD ports tree. Given the system resources required to scan mail for spam and viruses, a dynamic blacklist can keep things rolling along at a nice clip. Use after helo checks and before your RBLs. Good blacklists don't keep non-offending IPs around for more than a day or two.

This is clever: filtering spam by exploiting properties of spam pumps in general, vs. straight content analysis. The competition of ever-more-sophisticated content scanning techniques on one side, and spammers' escalating workarounds and huge botnets on the other side, is an arms race that shows no sign of abating.

Of course, this approach does still depend on something—probably content analysis—to determine which messages are spam and which are not, so that receivers' spam statistics can be computed.

The smartest (and reportedly most effective) anti-spam technique I know is spamd, which completely sidesteps content analysis. In a nutshell, it's an SMTP proxy that issues a temporary error code to unknown senders; legitimate MTAs retry delivery (at which point spamd lets the message through), while spam pumps don't bother. Voilà—spam gets stopped before it's ever received. A friend of mine reports that spam volume has dropped to zero since he set up spamd for his department.

If I understand the "receiver reputation" approach correctly, it could use spamd (rather than content analysis) to identify spam; similarly, content analysis can supplement spamd. The two are potentially complementary.

Ha ha, silly admin. My money's on greylisting.

We use both SpamAssassin and OpenBSD's spamd, to great effect. spamd does most of the work, though. Daniel Hartmeier (site down ATM, unfortunately) has an example of how to tie SA scores back into spamd for blacklisting, which is just awesome. I'd implement it here, but our current setup is effective enough as to not make it worth my time.

This was posted here previously, but it's a great idea. Annoying spammers with pf and spamd

I don't like getting spam. The problem is not detecting it automatically, that works very well with tools like SpamAssassin and bmf. Even though I can automatically delete spam without reading it, the spammers still successfully deliver their mails and get paid by volume. I want to hurt them. They should not be able to deliver their mails, and waste as much of their resources as possible attempting to do so.

...

Now I'm building my own blacklist, based on the evidence I've seen myself, classified by my own spam detector configuration. The only politics involved in someone getting blacklisted are my own, I don't have to trust a third party to make fair decisions. And I use this blacklist to redirect hosts to the tarpit, using pf and some cronjobs:

The biggest problem is that it requires some OpenBSD knowledge. It'd be great if we could get a nice idiot-proof install ISO for a drop-in box.

From the link...

--snip log example--
This spammer got stuck for 47 minutes. Current spamd sets its socket receive buffer size to one character, forcing the sender to send one TCP packet for each byte of data, even if its a non-compliant "dump and disconnect" mailer. Of course, the spammer nearly immediately tries to retransmit the spam. Repeatedly.

It seems that you have picked up a few cisco buzzwords, but don't really understand where the problem lies. You need to learn more about how TCP truly works before spouting off.

rules to give TCP ACKs

"Hogging" the line isn't the problem. Even with prioritization, fair queueing and ingress policing being performed by my router, the problem isn't that the packets don't get through, or bandwidth isn't shared fairly, it's that latency gets large.

You are missing the point. TCP ACKs are the problem. The clearest explanation is Daniel Hartmeier's page. Pay attention to the graphs, where the "latency" seems to be getting too large, it is because the empty TCP ACK packets are getting delayed or dropped, limiting the download speeds. There are other things you can do on a loaded asymmetric line, such as enlarging the TCP window on your end equipment. WRED on your cisco is not going to help you at all, if it isn't putting ACKs in the higher priority queue. If you know so much about cisco (or are at least learning), then learn to make some class based rules for TCP ACKs, or some advanced ACLs to put ACKs into custom priority queues. Googling doesn't easily turn up the answer, and I don't really know why.

There rarely is any saturation internal to an ISP's network, it tends to happen at the transit peering connections (which cost money, and thus lag in development). Once customer traffic clears that first asymmetric hop, all the internal connections are symmetric. Your problem is that the asymmetry is accentuating TCP congestion problems, so concentrate your solution there.

Your ISP does not perform queueing. This is unlikely

Realise that no ADSL provider is going to do IP based prioritization, (although we'll tweak individual VCI channels at the ATM level for voice and video). ICMP packets have always been lowest priority, and are the first type of packets dropped on heavily loaded routers and switches. There isn't enough marketing return from whiny gamerz to prioritize ICMP.

I believe there is a very lucrative market out there for the first ADSL modem box manufacturer to produce a system which prioritizes return traffic and target the gamerz community. Come up with some snazzy marketing slogans "guaranteed best ping times" "cut your lag by 20%", and they could charge quite a premium to the gamerz crowd.

the AC

http://www.benzedrine.cx/ackpri.html

That should do it. I've been using ack prioritization since a couple months after the artitcle was released. I've noticed recently that I still get good pings when torrenting and playing RTCW:ET.

Have fun.

Beware TPB

So what does the Slashdot crowd use when they need to secure their Linux and Windows servers? Does it cost less than US$100?"

As long as iptables functions as it should (which it does), there's always a way to deal with the syntax issue.

You don't get the point. The OpenBSD pf firewall and its configuration syntax are superior to anything out there including hardware boxes. Check out the man page or this benchmark if you don't believe me.

A very nice read:

http://www.benzedrine.cx/relaydb.html

Here's the plan:
1. Set up High Availability router with pfsync. (using computers rescued from the trash)
2. Set up a HA Network RAID system using DRBD or something similar. (using more computers rescued from the trash)
3. Build a Kerrighed or OpenSSI Single System Image cluster. (using the latest and greatest computers one can rescue from the trash)
4. ???
5. Profit! (and thus, have enough money to actually buy equipment)

I've already set aside Tuesday evening to upgrade my bandwidth throttling OpenBSD router. I set it up the day before 3.6 came out, so I didn't feel like upgrading until now. I'm tired of the typical hardware failures you tend to get out of computers people throw out (maybe that's why they threw them out in the first place) but mostly I'm looking forward to getting a learning experience hundreds of times more valuable (personally) than getting my MCSE 2003.

\/\/\/

Linux with wondershaper

Have fun!

Learn more about queueing at the pf FAQ.

See my previous post here for a pf.conf recipe to implement traffic shaping based on packet type. You could also prioritize via IP, require a ssh session to gain higher priorities for a specific IP (authpf), based on time of day, or any other number of factors.

THE guide to pf (packet filter) can be found here. pf will run on FreeBSD as well as, I believe, a few other open OSs. I think it's really the best. Almost any reader here could surely benefit from at least a partial working knowledge of packet filters ("firewalls") in general.

=======
EXTRA CREDIT
=======
Got a few connections you'd like to tie together into one? Read more about Address Pools and Load Balancing with pf.

Another Bandwidth management HOWTO for Linux systems (last revised in '03 - may be better for concepts than router config recipes)

bittorrent traffic shaping

A nice K5 article about packet filtering with OpenBSD firewalls

Prioritizing empty TCP ACKs with pf and ALTQ

Making the most out of a busy connection

Turn that old P5 and two network cards into an OpenBSD firewall and learn to setup your own router. You will learn a TON about TCP/IP, how to protect your internal network, and BSDs in general (they're pretty neat in the way that they don't have as much "cruft" as usually found in your typical - yeah, that works :) - Linux distro. The simplicity, if you've never experienced it before, can feel both constraining and liberating at the same time. Give it a try if you've got a spare box. It's hard to experiment without learning SOMETHING - and if you're here I'm sure you're into learning, right? So give it a whirl. If you're not sure what BSD to try, give this a read. If you just want to buy a router, learn from the recent Ask Slashdot - Home Routers w/ Decent QoS Performance?. Best of luck!

If you're going to use OpenBSD (which I'd recommend for a firewall/NAT box), be sure to support the OS which strives for portability, standardization, correctness, proactive security and integrated cryptography by ordering a CD, T-shirt, book, or hacker bunker enhancing poster. OpenBSD supports binary emulation of most programs from SVR4 (Solaris), FreeBSD, Linux, BSD/OS, SunOS and HP-UX. Development is active and it won't let you down as a gatekeeper or internal server.

Puffy says "Stay off my computer!" and means it. I sleep well at night knowing "puffy" (the name of my box) is standing guard just behind my cable modem and in front of the 5+ computers my roommates and I are running inside. Has never let me down and doesn't get in my way. Keeps Freenet and torrents from introducing lag into my ssh sessions as well..... Good luck finding a solution to keeping your pipes clean :)

PF hasn't been ported to Linux. It has been ported to all the BSD operating systems. OpenSSH is userland software, not kernel, which is why it can run on so many platforms.

Daniel Hartmeier of PF fame has a nice little howto on how to tarpit spammers on http://www.benzedrine.cx/relaydb.html I just love that idea, grab them in the act, delay them and send a NDR in the end.

And of course, if you ever meet anybody who bought anything form a spammer, hit them on the head with a big plank.

Sporri

dude, you should seriously consider changing the OS of your gateway box. If you have the skills or time, learn to use something like OpenBSD or FreeBSD. Both currently include the amazing PF packet filter, which supports NAT and amazing packet filtering capabilities. Everything is well documented on OpenBSD FAQ web page.

Due to your slow link, consider getting regular CVS patches via CTM, one way of keeping your CVS tree updated via email. Patches are usually small and can be quickly downloaded via FTP.

Good luck.

http://www.benzedrine.cx/pf2k4/DSC01359-medium.jpg

..it was released on 29th of October already, as you can read here. When download, please don't forget to use the torrent!

From RFC 3449: "performance often degrades significantly because of imperfection and variability in the ACK feedback from the receiver to the sender."

prioritizing ACK packets with PF on OpenBSD

This was posted on Slashdot a while back. I personally use this for my home network with pretty much the same results posted there (as far as I can tell without doing formal testing). It's pretty much the only way I can keep my connection usable when my web server or torrents are having a good day.

PF is now available on OpenBSD, FreeBSD, NetBSD and DragonFlyBSD. I think some of them may still be lacking ALTQ support (needed to do the stuff in that link), but I'm not sure, I don't keep track of all of them.

I'm vaguely aware of NetFilter on Linux having similar capabilities, but AFAIK you need to tell it how to detect ACK and URG packets, it doesn't "just know" like PF does. I could be wrong about that, as it's been a long time since I looked into it. IPF and IPFW may or may not be able to do this, I have no knowledge of their capabilities in this area.

The official release has just happened. Here are the official announcement, the undeadly.org thread and a torrent for the i386 binaries (149MB, matching MD5 which might beat some of the mirrors). Cheers ;)

Then install OpenBSD and run kissd. :-)

I use Sendmail at work, and realizing how things have changed on the spam front I updated my /etc/mail/access file so it now starts like this:

# Reject cable and DSL users who are now Damned Zombie Spam Bastards - keep adding to this
cable.mindspring.com ERROR:"550 Blocked"
cq.shawcable.net ERROR:"550 Blocked"
cg.shawcable.net ERROR:"550 Blocked"
...
rd.shawcable.net ERROR:"550 Blocked"
va.shawcable.net ERROR:"550 Blocked"
...

I had tried the same /etc/mail/access approach for a while on my home email server. My experience was that it works well for some ISPs, but not well for others. (Ex: it deals poorly with ISPs using customer hostnames like "cust-1-2-3-4.some-isp.com" and legitimate mailservers named "mx01.some-isp.com" -- if you block "some-isp.com" to deal with the spambots on customer machines, you end up blocking the legitimate mailservers for that domain as well.)

It's also somewhat frustrating to have to type 51-odd nearly identical entries for, say, all the charter[state].com domains, or the various [province/city].shawcable.net subdomains.

You might want to take a look at Sendmail's "milter" API, and use it with something like milter-regex to implement your site blocks. For those who haven't heard of it before, the milter API allows sendmail to hook into external programs to evaluate/classify incoming email connections, and these external programs can provide more flexible and/or more advanced tests than sendmail itself can easily implement.

By using regular expressions, you can easily weed out "cust-1-2-3-4.some-isp.com" hostnames (just "^cust-.*\.some-isp\.com$") without running any risk of blocking legitimate servers for that domain.

You can also avoid a lot of the cut'n'pasting for ISPs using state/province/etc abbreviations in their customer hostnames, and have a single ".*\.(cq|cg|ed|...vs)\.shawcable\.net" rule. Shaw Cable's sort of a bad example for this, since there's a pair of much simpler/shorter expressions that can block all their home-user machines without having to figure out the abbreviation for every province/city in Canada -- from what I've seen, Shaw customer hostnames all start with either "h" followed by their IP address with dashes between the octets, or "S" followed by a string of hex digits (probably based on the customer's MAC).

My experience with milter-regex is that the vast majority of ISPs with spambot-infested customer machines can be handled with a single rule per ISP - most have an obvious, standardized pattern for customer hostnames. There's a couple of exceptions, like Charter and RoadRunner, but even those guys have less than a dozen different customer hostname schemes.

The other feature I've liked about milter-regex is that it makes it reasonably easy to deal with mail coming from machines that lack verifiable hostnames (either there's no reverse-DNS info, or their DNS server is temporarily offline for some reason). I've adopted the approach of returning a temporary failure message for these cases. Spam software doesn't usually bother resending a message if it gets a tempfail message back from the target server. OTOH, most serious/legitimate mail servers will do the right thing, and retry the message several more times, usually for several days. I've only had a couple of cases where legitimate email was trying to come in from a server lacking reverse DNS info (one from a friend whose corporate IT department didn't know how to set up reverse DNS, and another from a product newsletter that was farmed out to a [totally legitimate] bulk email service). I handle these cases by monitoring the sendmail logs and watching for entries from servers that are retrying messages at regular intervals (usually once per hour for the first day), temporarily whitelisting the server

What about this here Spamd stuff I've been reading about with OpenBSD? Does it make spammer's lives a little more difficult? Tying up their resources while never sending any of their junk out?

http://www.benzedrine.cx/mirror/w2k.zip.torrent (note: do not fear the christmas islands, you racists!)

I've checked it, it works!

There is this an old slashdot story that recommende this article by an OpenBSD hacker about priotizing tcp ack packets. So the solution is to have enough bandwidth to send the ACKs back, and you will only have that if scheduler handles the bandwidth (Look at his graphs they show it crystal clear from peaks at 130kbps to an even 83kpbs upstream).

I don't know about wondershaper but with tc you can match ACK and give them priority..


tc filter add WhereEver
match u8 0x10 0xff at nexthdr+13

This sounds a lot like OpenBSD's spamd program. Check out a possible use of it like spamhole.

Well, here's what I've done and it hasn't gotten me on any black lists for running an open relay because I don't.

First, my mailserver runs OpenBSD, this allows me to use pf for my port filtering software. Then each user on the server has a copy of CRM114 installed. This is a very powerful and extremely accurate bayesian classifier. I've gotten 1 piece of spam in the last three months, 0 false positives and it blocks about 150 pieces of spam a day (for my account alone).

For each piece of mail that I receive, the relays involved are entered into relaydb. This wonderful little program logs each mail relay listed in the message. When a relay has 3 times as many bad messages as good messages it is added to the black list. Because I'm using pf, this blacklist is updated in real time to the mail server's pf configuration, which causes spamming hosts to be sent to the tar pits.

I'd estimate the total accuracy rate (defined as non-Type I and non-Type II errors) to be somewhere around 99.95%. User interaction is zero for most of the time, I've got a nice corpus that I train the accounts with. On the off hand that there is an error the user mails the message to themselves and it gets fixed.

So, to summarize:
This idea won't work, you'll get your host marked as an open relay.
This is what I did to kill spam and it does work.

Really? Got any links to back this claim up?

more info

I can't rememebr where, but somebody on the FreeBSD also mentioned poring it for 5.3 or so.

I've read through the comments at score 3, and I'm surprised that no one has mentioned using the spamd utility that comes with OpenBSD 3.3 and above. Basically, it hurts the spamming mail servers, while although normally some poor machine caught in the crossfire, does make me happy to use.
Basically, for every email I get, it gets put through a baysien filter (bmf in this case) and handled through procmail correctly. However, the email is then passed through another program which constructs and adds to a whitelist and blacklist of IP's. Basically its a 3 to 1 ratio. If a mail server sends me a legit message, followed by a spam, it is still considered a legit server until I recieve 3 more spams from it and no more legit email.
Once an email server is considered to be on my blacklist, my firewall redirects the next connection from that server to my teergrube, which emulates an deaf man which you have to keep repeating yourself to rather loudly and slowly. The use of my resources is minimal.
For a more detailed explanation, take a look at Annoying Spammers with pf and spamd.

This could be because the TCP stack on your OS doesn't implement ACK prioritization, which can be pretty useful on links with asymmetric bandwidth limits, like DSL. Essentially, when you max out your upload bandwidth, you also increase the amount of contention/latency experienced when sending the TCP ACKs which need to be sent in response to the TCP packets you download (to let the sender know you've received them, since otherwise the server has no means of knowing if they actually got through to you, rather than having been dropped along the way). This in turn could lead to retransmits/rate throttling by the server sending you the matrix (or whatever), thus causing slower downloads. Of course, part of this may also be due to some other factors, such as a broken algorithm in bittorrens. Here's a good link to learn more about this: ALTQ

If you've got a decent router you can use QoS to prioritize ACKs to mitigate this problem. Check out this. The article targets OpenBSD's pf, but if you're using something else it shouldn't be too difficult to adapt the idea to your system.

Daneil Hartmeier (the guy who started OpenBSD's pf firewall) has an explanation of how this can be used in conjuction with filters such as spamassassin. Using this method, each time you get a spam, the spammer gets blacklisted to be directed to spamd the next time. It's documented at http://www.benzedrine.cx/relaydb.html -

Until, after several attempts, wasting both his queue space and socket handles for several days, he gives up. The resources I have to waste to do this are minimal.

If the sender is badly configured, an uncooperative recipient might actually delay his entire queue handling for several minutes each time he connects to the tarpit. And many spammers use badly configured open relays.

Combining this method on the web server with something like this on the mail server could be fun.

bemis
forgot his password

Uploading at full speed throttles my whole connection

I use an OpenBSD 3.3 box as my DSL firewall. The new version of pf integrates ALTQ, meaning that you can assign packet traffic to queues based on their ToS priorities. See this writeup for details.

Before I upgraded to 3.3, Internet response was abysmal whenever I was running a concurrent BitTorrent uploader. Now it is much more bearable. (If only I could get PPPoE to stay up for more than 20 minutes at a time...)

Before we cheer legal solutions (which will have their fair share of downsides) maybe more people should take technological measures.

Also have a look here: Annoying spammers with OpenBSD's pf
Slides explaining how Bayesian email filtering is successful

PS: I know people might say, but what about the economic cost of spam, blah blah blah. Read the slides. If no one ever gets spam, people will stop sending it, and the economic cost goes away.

Good luck!

I'm going to take up the challenge here of explaining why this is interesting. Since November of 2002, OpenBSD's pf has had support for load balancing. RedHat's $2499 Premium Edition of their Enterprise distro features Piranha load balancing which was derived from the Linux High Availability project.

So what the OpenBSD pf project is giving you is enterprise-class high availability and load-balance clustering for a tiny fraction of the price. With a handful of cheap dotcom-throw-away x86 servers, a small company or mildly well-capitalized individual can personally build a multi-datacenter-fault-tolerant clustering setup that will rival Fortune 500 uptime ratings.

In other words, the pf project's list of accomplishments is starting to read like a ToDo list for RedHat's Enterprise Linux development team.

Okay, in case you're not joking: PF = packet filter

Packet Filter, the best way to do whatever you want with your packets.

http://www.benzedrine.cx/ackpri-norm.jpg
http://www.benzedrine.cx/ackpri-priq.jpg

benzedrine.cx - Prioritizing empty TCP ACKs with pf and ALTQ Prioritizing empty TCP ACKs with pf and ALTQ

Introduction ALTQ is a framework to manage queueing disciplines on network interfaces. It manipulates output queues to enforce bandwidth limits and priorize traffic based on classification.

While ALTQ was part of OpenBSD and has been enabled by default since several releases, the next release will merge the ALTQ and pf configuration into a single file and let pf assign packets to queues. This both simplifies the configuration and greatly reduces the cost of queue assignment.

This article presents a simple yet effective example of what the pf/ALTQ combination can be used for. It's meant to illustrate the new configuration syntax and queue assignment. The code used in this example is already available in the -current OpenBSD source branch.

Problem I'm using an asymmetric DSL with 512 kbps downstream and 128 kbps upstream capacity (minus PPPoE overhead). When I download, I get transfer rates of about 50 kB/s. But as soon as I start a concurrent upload, the download rate drops significantly, to about 7 kB/s.

Explanation Even when a TCP connection is used to send data only in one direction (like when downloading a file through ftp), TCP acknowledgements (ACKs) must be sent in the opposite direction, or the peer will assume that its packets got lost and retransmit them. To keep the peer sending data at the maximum rate, it's important to promptly send the ACKs back.

When the uplink is saturated by other connections (like a concurrent upload), all outgoing packets get delayed equally by default. Hence, a concurrent upload saturating the uplink causes the outgoing ACKs for the download to get delayed, which causes the drop in the download throughput.

Solution The outgoing ACKs related to the download are small, as they don't contain any data payload. Even a fast download saturating the 512 kbps downstream does not require more than a fraction of upstream bandwidth for the related outgoing ACKS.

Hence, the idea is to priorize TCP ACKs that have no payload. The following pf.conf fragment illustrates how to set up the queue definitions and assign packets to the defined queues:

ext_if="kue0"

altq on $ext_if priq bandwidth 100Kb queue { q_pri, q_def }
queue q_pri priority 7
queue q_def priority 1 priq(default)

pass out on $ext_if proto tcp from $ext_if to any flags S/SA \
keep state queue (q_def, q_pri)

pass in on $ext_if proto tcp from any to $ext_if flags S/SA \
keep state queue (q_def, q_pri)
First, a macro is defined for the external interface. This makes it easier to adjust the ruleset when the interface changes.

Next, altq is enabled on the interface using the priq scheduler, and the upstream bandwidth is specified.
I'm using 100 kbps instead of 128 kbps as this is the real maximum I can reach (due to PPPoE encapsulation overhead). Some experimentation might be needed to find the best value. If it's set too high, the priority queue is not effective, and if it's set too low, the available bandwidth is not fully used.
Then, two queues are defined with (arbitrary) names q_pri and q_def. The queue with the lower priority is made the default.

Finally, the rules passing the relevant connections (statefully) are extended to specify what queues to assign the matching packets to. The first queue specified in the parentheses is used for all packets by default, while the second (and optional) queue is used for packets with ToS (type of service) 'lowdelay' (for instance interactive ssh sessions) and TCP ACKs without payload.

Both incoming and outgoing TCP connections will pass by those two rules, create state, and all packets related to the connections will be assigned to either the q_def or q_pri queues. Packets assigned to the q_pri queue will have priority and will get sent before any pending packets in the q_def queue.

Result The following test was performed first without and then with the ALTQ rules explained above:

• -10 to -8 minutes: idle
  
• -8 to -6 minutes: download only
  
• -6 to -4 minutes: concurrent download and upload
  
• -4 to -2 minutes: upload only
  
• -2 to 0 minutes: idle

The first graphs shows the results of the test without ALTQ, and the second one with ALTQ:

Image 1, ACK PRI Normal

Image 2, ACK PRI PRIq

The improvement is quite significant, the saturated uplink no longer delays the outgoing empty ACKs, and the download rate doesn't drop anymore.

This effect is not limited to asymmetric links, it occurs whenever one direction of the link is saturated. With an asymmetric link this occurs more often, obviously.

Related links

• The OpenBSD project
  
• The OpenBSD packet filter (pf)
  
• pf mailing list archive
  
• pf.conf(5) man page
  

http://www.benzedrine.cx/ackpri-norm.jpg
http://www.benzedrine.cx/ackpri-priq.jpg

benzedrine.cx - Prioritizing empty TCP ACKs with pf and ALTQ Prioritizing empty TCP ACKs with pf and ALTQ

Introduction ALTQ is a framework to manage queueing disciplines on network interfaces. It manipulates output queues to enforce bandwidth limits and priorize traffic based on classification.

While ALTQ was part of OpenBSD and has been enabled by default since several releases, the next release will merge the ALTQ and pf configuration into a single file and let pf assign packets to queues. This both simplifies the configuration and greatly reduces the cost of queue assignment.

This article presents a simple yet effective example of what the pf/ALTQ combination can be used for. It's meant to illustrate the new configuration syntax and queue assignment. The code used in this example is already available in the -current OpenBSD source branch.

Problem I'm using an asymmetric DSL with 512 kbps downstream and 128 kbps upstream capacity (minus PPPoE overhead). When I download, I get transfer rates of about 50 kB/s. But as soon as I start a concurrent upload, the download rate drops significantly, to about 7 kB/s.

Explanation Even when a TCP connection is used to send data only in one direction (like when downloading a file through ftp), TCP acknowledgements (ACKs) must be sent in the opposite direction, or the peer will assume that its packets got lost and retransmit them. To keep the peer sending data at the maximum rate, it's important to promptly send the ACKs back.

When the uplink is saturated by other connections (like a concurrent upload), all outgoing packets get delayed equally by default. Hence, a concurrent upload saturating the uplink causes the outgoing ACKs for the download to get delayed, which causes the drop in the download throughput.

Solution The outgoing ACKs related to the download are small, as they don't contain any data payload. Even a fast download saturating the 512 kbps downstream does not require more than a fraction of upstream bandwidth for the related outgoing ACKS.

Hence, the idea is to priorize TCP ACKs that have no payload. The following pf.conf fragment illustrates how to set up the queue definitions and assign packets to the defined queues:

ext_if="kue0"

altq on $ext_if priq bandwidth 100Kb queue { q_pri, q_def }
queue q_pri priority 7
queue q_def priority 1 priq(default)

pass out on $ext_if proto tcp from $ext_if to any flags S/SA \
keep state queue (q_def, q_pri)

pass in on $ext_if proto tcp from any to $ext_if flags S/SA \
keep state queue (q_def, q_pri)
First, a macro is defined for the external interface. This makes it easier to adjust the ruleset when the interface changes.

Next, altq is enabled on the interface using the priq scheduler, and the upstream bandwidth is specified.
I'm using 100 kbps instead of 128 kbps as this is the real maximum I can reach (due to PPPoE encapsulation overhead). Some experimentation might be needed to find the best value. If it's set too high, the priority queue is not effective, and if it's set too low, the available bandwidth is not fully used.
Then, two queues are defined with (arbitrary) names q_pri and q_def. The queue with the lower priority is made the default.

Finally, the rules passing the relevant connections (statefully) are extended to specify what queues to assign the matching packets to. The first queue specified in the parentheses is used for all packets by default, while the second (and optional) queue is used for packets with ToS (type of service) 'lowdelay' (for instance interactive ssh sessions) and TCP ACKs without payload.

Both incoming and outgoing TCP connections will pass by those two rules, create state, and all packets related to the connections will be assigned to either the q_def or q_pri queues. Packets assigned to the q_pri queue will have priority and will get sent before any pending packets in the q_def queue.

Result The following test was performed first without and then with the ALTQ rules explained above:

• -10 to -8 minutes: idle
  
• -8 to -6 minutes: download only
  
• -6 to -4 minutes: concurrent download and upload
  
• -4 to -2 minutes: upload only
  
• -2 to 0 minutes: idle

The first graphs shows the results of the test without ALTQ, and the second one with ALTQ:

Image 1, ACK PRI Normal

Image 2, ACK PRI PRIq

The improvement is quite significant, the saturated uplink no longer delays the outgoing empty ACKs, and the download rate doesn't drop anymore.

This effect is not limited to asymmetric links, it occurs whenever one direction of the link is saturated. With an asymmetric link this occurs more often, obviously.

Related links

• The OpenBSD project
  
• The OpenBSD packet filter (pf)
  
• pf mailing list archive
  
• pf.conf(5) man page
  

http://www.benzedrine.cx/ackpri-norm.jpg
http://www.benzedrine.cx/ackpri-priq.jpg

benzedrine.cx - Prioritizing empty TCP ACKs with pf and ALTQ Prioritizing empty TCP ACKs with pf and ALTQ

Introduction ALTQ is a framework to manage queueing disciplines on network interfaces. It manipulates output queues to enforce bandwidth limits and priorize traffic based on classification.

While ALTQ was part of OpenBSD and has been enabled by default since several releases, the next release will merge the ALTQ and pf configuration into a single file and let pf assign packets to queues. This both simplifies the configuration and greatly reduces the cost of queue assignment.

This article presents a simple yet effective example of what the pf/ALTQ combination can be used for. It's meant to illustrate the new configuration syntax and queue assignment. The code used in this example is already available in the -current OpenBSD source branch.

Problem I'm using an asymmetric DSL with 512 kbps downstream and 128 kbps upstream capacity (minus PPPoE overhead). When I download, I get transfer rates of about 50 kB/s. But as soon as I start a concurrent upload, the download rate drops significantly, to about 7 kB/s.

Explanation Even when a TCP connection is used to send data only in one direction (like when downloading a file through ftp), TCP acknowledgements (ACKs) must be sent in the opposite direction, or the peer will assume that its packets got lost and retransmit them. To keep the peer sending data at the maximum rate, it's important to promptly send the ACKs back.

When the uplink is saturated by other connections (like a concurrent upload), all outgoing packets get delayed equally by default. Hence, a concurrent upload saturating the uplink causes the outgoing ACKs for the download to get delayed, which causes the drop in the download throughput.

Solution The outgoing ACKs related to the download are small, as they don't contain any data payload. Even a fast download saturating the 512 kbps downstream does not require more than a fraction of upstream bandwidth for the related outgoing ACKS.

Hence, the idea is to priorize TCP ACKs that have no payload. The following pf.conf fragment illustrates how to set up the queue definitions and assign packets to the defined queues:

ext_if="kue0"

altq on $ext_if priq bandwidth 100Kb queue { q_pri, q_def }
queue q_pri priority 7
queue q_def priority 1 priq(default)

pass out on $ext_if proto tcp from $ext_if to any flags S/SA \
keep state queue (q_def, q_pri)

pass in on $ext_if proto tcp from any to $ext_if flags S/SA \
keep state queue (q_def, q_pri)
First, a macro is defined for the external interface. This makes it easier to adjust the ruleset when the interface changes.

Next, altq is enabled on the interface using the priq scheduler, and the upstream bandwidth is specified.
I'm using 100 kbps instead of 128 kbps as this is the real maximum I can reach (due to PPPoE encapsulation overhead). Some experimentation might be needed to find the best value. If it's set too high, the priority queue is not effective, and if it's set too low, the available bandwidth is not fully used.
Then, two queues are defined with (arbitrary) names q_pri and q_def. The queue with the lower priority is made the default.

Finally, the rules passing the relevant connections (statefully) are extended to specify what queues to assign the matching packets to. The first queue specified in the parentheses is used for all packets by default, while the second (and optional) queue is used for packets with ToS (type of service) 'lowdelay' (for instance interactive ssh sessions) and TCP ACKs without payload.

Both incoming and outgoing TCP connections will pass by those two rules, create state, and all packets related to the connections will be assigned to either the q_def or q_pri queues. Packets assigned to the q_pri queue will have priority and will get sent before any pending packets in the q_def queue.

Result The following test was performed first without and then with the ALTQ rules explained above:

• -10 to -8 minutes: idle
  
• -8 to -6 minutes: download only
  
• -6 to -4 minutes: concurrent download and upload
  
• -4 to -2 minutes: upload only
  
• -2 to 0 minutes: idle

The first graphs shows the results of the test without ALTQ, and the second one with ALTQ:

Image 1, ACK PRI Normal

Image 2, ACK PRI PRIq

The improvement is quite significant, the saturated uplink no longer delays the outgoing empty ACKs, and the download rate doesn't drop anymore.

This effect is not limited to asymmetric links, it occurs whenever one direction of the link is saturated. With an asymmetric link this occurs more often, obviously.

Related links

• The OpenBSD project
  
• The OpenBSD packet filter (pf)
  
• pf mailing list archive
  
• pf.conf(5) man page
  

http://www.benzedrine.cx/ackpri-norm.jpg
http://www.benzedrine.cx/ackpri-priq.jpg

benzedrine.cx - Prioritizing empty TCP ACKs with pf and ALTQ Prioritizing empty TCP ACKs with pf and ALTQ

Introduction ALTQ is a framework to manage queueing disciplines on network interfaces. It manipulates output queues to enforce bandwidth limits and priorize traffic based on classification.

While ALTQ was part of OpenBSD and has been enabled by default since several releases, the next release will merge the ALTQ and pf configuration into a single file and let pf assign packets to queues. This both simplifies the configuration and greatly reduces the cost of queue assignment.

This article presents a simple yet effective example of what the pf/ALTQ combination can be used for. It's meant to illustrate the new configuration syntax and queue assignment. The code used in this example is already available in the -current OpenBSD source branch.

Problem I'm using an asymmetric DSL with 512 kbps downstream and 128 kbps upstream capacity (minus PPPoE overhead). When I download, I get transfer rates of about 50 kB/s. But as soon as I start a concurrent upload, the download rate drops significantly, to about 7 kB/s.

Explanation Even when a TCP connection is used to send data only in one direction (like when downloading a file through ftp), TCP acknowledgements (ACKs) must be sent in the opposite direction, or the peer will assume that its packets got lost and retransmit them. To keep the peer sending data at the maximum rate, it's important to promptly send the ACKs back.

When the uplink is saturated by other connections (like a concurrent upload), all outgoing packets get delayed equally by default. Hence, a concurrent upload saturating the uplink causes the outgoing ACKs for the download to get delayed, which causes the drop in the download throughput.

Solution The outgoing ACKs related to the download are small, as they don't contain any data payload. Even a fast download saturating the 512 kbps downstream does not require more than a fraction of upstream bandwidth for the related outgoing ACKS.

Hence, the idea is to priorize TCP ACKs that have no payload. The following pf.conf fragment illustrates how to set up the queue definitions and assign packets to the defined queues:

ext_if="kue0"

altq on $ext_if priq bandwidth 100Kb queue { q_pri, q_def }
queue q_pri priority 7
queue q_def priority 1 priq(default)

pass out on $ext_if proto tcp from $ext_if to any flags S/SA \
keep state queue (q_def, q_pri)

pass in on $ext_if proto tcp from any to $ext_if flags S/SA \
keep state queue (q_def, q_pri)
First, a macro is defined for the external interface. This makes it easier to adjust the ruleset when the interface changes.

Next, altq is enabled on the interface using the priq scheduler, and the upstream bandwidth is specified.
I'm using 100 kbps instead of 128 kbps as this is the real maximum I can reach (due to PPPoE encapsulation overhead). Some experimentation might be needed to find the best value. If it's set too high, the priority queue is not effective, and if it's set too low, the available bandwidth is not fully used.
Then, two queues are defined with (arbitrary) names q_pri and q_def. The queue with the lower priority is made the default.

Finally, the rules passing the relevant connections (statefully) are extended to specify what queues to assign the matching packets to. The first queue specified in the parentheses is used for all packets by default, while the second (and optional) queue is used for packets with ToS (type of service) 'lowdelay' (for instance interactive ssh sessions) and TCP ACKs without payload.

Both incoming and outgoing TCP connections will pass by those two rules, create state, and all packets related to the connections will be assigned to either the q_def or q_pri queues. Packets assigned to the q_pri queue will have priority and will get sent before any pending packets in the q_def queue.

Result The following test was performed first without and then with the ALTQ rules explained above:

• -10 to -8 minutes: idle
  
• -8 to -6 minutes: download only
  
• -6 to -4 minutes: concurrent download and upload
  
• -4 to -2 minutes: upload only
  
• -2 to 0 minutes: idle

The first graphs shows the results of the test without ALTQ, and the second one with ALTQ:

Image 1, ACK PRI Normal

Image 2, ACK PRI PRIq

The improvement is quite significant, the saturated uplink no longer delays the outgoing empty ACKs, and the download rate doesn't drop anymore.

This effect is not limited to asymmetric links, it occurs whenever one direction of the link is saturated. With an asymmetric link this occurs more often, obviously.

Related links

• The OpenBSD project
  
• The OpenBSD packet filter (pf)
  
• pf mailing list archive
  
• pf.conf(5) man page
  

NFS under Linux isn't there yet. It's unreliable under heavy load (eg. when saturating Gigabit ethernet). Linux NFS also has a very funny idea about RFCs, ie. it ignores them when it doesn't like what they say (NFS fragments packets in some cases when the RFC explicitly says not to).

NFS over TCP would also be nice. Linux has NFS over UDP only (in production code). Connection-based NFS gives a warm fussy feeling wrt. actually *knowing* about lost packets.

Multiple network cards are a pain, largely due to how Linux handles ARP. Routing table and ARP cache can contradict each other in certain situations (involving multiple network cards connected to the same switch, even on different subnets, caused by how Linux handles ARP requests).

The fact that the Linux packet filter doesn't track TCP sequence numbers is disconcerting. (This "feature" is used to allow TCP window scaling over NATted connections.)

In general, it seems that Linux networking tends to be rather sloppy about rules that concern security (TCP seq num) and interoperability (NFS frags) whenever it fits them, and don't care much about how other OSes cope with it. This mindset is almost Windowsian.

Daniel Hartmeier uses OpenBSD and packet filter to waste spammers time.

Daniel Hartmeier uses OpenBSD and packet filter to waste spammers time.

Daniel Hartmeier uses OpenBSD and packet filter to waste spammers time.

This is the same thing as OpenBSD's spamd, which Theo de Raadt wrote specifically to cause spam relays pain. spamd uses some new features of pf and blacklists from Spews to create a tarpit for incoming messages from known spam relays. It was even discussed on Slashdot in this article. Also, Daniel Hartmeier, pf developer extraordinaire and all around good guy, wrote a little piece about annoying spammers using pf, spamd, and bmf.

Daniel has a mirror of the interview at his site.

Daniel has a mirror of the interview at his site.

Somebody mod the parent to this up!

This is reason enough for Linux and other OSS projects to be wary of wedding themselves to proprietary software whose licensing can change on a whim. Of course, such an event would hamper kernel development, for a time anyhow, but it would also light a fire under developers' arses to improve CVS (or write a new OSS product) out of necessity, after which I imagine we'd hear "never again!" shouted from open source rooftops.

This kinda reminds me of when de Raadt removed IPFilter from OpenBSD's ports tree over restrictive licensing and the OpenBSD crew took it upon themselves to develop pf from the ground up.