Apache Bandwidth Limiting?

IOOOOOI asks: "I work at a high traffic web hosting company and we're trying to find a simple effective way to limit bandwidth hogs, some of whom we've clocked pulling over 4Gb/hr off our servers. We've tried mod_throttle and have looked into QoS/fair queuing as well as a couple of custom solutions in-house. None of these quite did the trick. Has anyone found an effective way to do this, one that can handle individual connection streams?"

Comments

Maybe I'm missing something...

[BitGeek]

But the solution seems easy to me. Simply charge your customers for their bandwidth.

This rectifies the disparity between flat rate pricing and incremental bandwidth costs.

When I went to find a solution for my web appliction, I chose to put in a DSL line and host it myself (Because of the complexity of the app, this is cheaper than colocating my computers there).. but I chose a DSL provider that doesn't give "all you can eat" - but instead charges for bandwidth.

The reason I chose this is the theory that the bandwidth hogs would go elsewhere and the latency at this ISP would be much lower. so far this has proven tru, and I've yet to exceed the basic "free" bandwidth level.

If, on the other hand, you're talking bout people who are downloading your customers content at huge rates, then maybe you should charge your customers based on the service they are providing. If they're hosting lots of large files, they should probably be paying more...

Dunno if that's a viable solution-- but smart customers will prefer someone who charges "by the byte"... because the bytes are better quality.

Re:Maybe I'm missing something...

[Apreche]

you are missing something. My friend runs a semi-large website and has had multiple hosts. On his website are very many images (not porn), mostly gif, jpg, etc. The rest is html and cgi. What happens is most visitors to the site come in and look at a few pictures, download a couple and leave. Very recently someone wrote a spider program that went through his entire website and downloaded every single image. He stopped it before they finished, but he had a very hard time finding a way to prevent it from happening again.

What apache needs is something where you can say
someone who is visiting the site only gets x amoutn of bandwith. And if someone tries to use up too much by downloading too much, stop sending things to them.

Re:Maybe I'm missing something...

[Electrum]

The issue is not about our customer's bandwidth consumption and how much they can/can't use. It's about being able to provide services to all of their users without experiencing slowdowns because of the occasional hog.

Are your slowdowns bandwidth or CPU based? If you are serving lots static content (like porn), then Apache is going to kill you, due to its process-per-connection model, which the developers refuse (read: are too lazy) to fix. Zeus doesn't have this problem. Neither do the open source boa or thttpd (but they unfortunately lack many important features that may stop them from being used for commercial web hosting). Zeus will allow you to max out your network card (100mbit) on a modest machine (P3/500 w/ 1gb RAM).

Re:Maybe I'm missing something...

[Electrum]

Common, have you tried to tune Apache? Process-per-connection not a problem - you just have to keep process pool big enough.. There are some other tricks, but you could saturate 100Mbit network with p3/500 and Apache as well.

I seriously doubt it, not in real world conditions. When you include things like mod_php and mod_perl, those Apache processes get big. Our hosting servers (running Zeus) get 15-20 thousand hits a minute. That's ~333 hits per second. Say each client is downloading 50k images at 2k per second. That means you have 300+ new connections opening per second, that stay open for 25 seconds. So you need to be handling 7500+ concurrent connections.

Keep alives and such will help with this, but a high traffic HTTP server needs to handle at least 1000-2000 connections concurrently. Show me a p3/500 that is running 2000 Apache processes, and processing scripts, etc., and isn't dying. It just won't happen. The process switching overhead alone will kill you. Read this page, then tell me that Apache's I/O model doesn't suck.

altqd

[schmaltz]

try altqd. i've only used it on openbsd, but with it you can selectively throttle bandwidth.

Packeteer

[shave]

Not an Apache based solution, but check out Packeteer Packetshapers..specifically the ISP models.. lets you set SLA's by protocol, IP, etc, perform rate limiting, and all other kinds of really cool stuff. Not exactly cheap but extremely effective, and simple to manage.

One solution if you have mod_perl

[merlyn]

The solution that the (defunct) etoys.com adopted for their site was based on code from one of my Perl columns. My code is based on CPU throttling, but you can quickly change it to bytes sent using the same technology.

mod_bandwidth

[Gormless]

I use mod_bandwidth at work to simulate 56k connections to the web server.

It works quite well and will throttle per-connection or per-virtualhost.

my experiences

[LinuxGeek8]

I'm not verey experienced with bandwidth limiting.
I did play with mod_throttle, and all it did was actually allow all traffic until the limit was reached, and then deny the next new connections. Hmm, not too great actually.

I'm planning to try out mod_bandwidth, but I dunno if it works different.
Bad link (sorry, I don't feel for html now):
http://www.cohprog.com/v3/bandwidth/doc-en. html

I tried playing with QoS on linux 2.4.
According to the documentation it's actually quite hard to have that functional, because if you have a 10 Mbit connection, it will shape the traffic elative to that. But 10 Mbit is not always the same. If you have lots of lost packets it will behave different then with a perfect connection.
In my experience I couldn't reliably limit the traffic on a 10 Mbit connection down to 80 kbit (almost 1% of the 10 Mbit). My cable connection of 16 kbyte still could get choked.
Maybe I should just get a card of 1 Mbit and try again, the numbers might be better then.
Or hey, a card of 100 Kbit :-) should be doing perfect.

Re:my experiences

[LinuxGeek8]

The sad part is it took you longer to type this than to type .

No, not really, I would have to type all
tags, which is quite annoying imo.
And for 10 or 12 lines of text I need to format it by hand.

Squid as accelerator

[d-rock]

You could look at using a combination of content acceleration and bandwidth pools in squid. I've used these features before and it actually works pretty well for static content. You can tune the caching params to allow for large files, etc.

Derek

You're missing the point

[gibmichaels]

I am having the same problem, and I think you guys are missing the point. He said 4GB an hour, which means he probably has an OC-3, OC-12, or Gigabit Ethernet connection.

"Blocking" network appliances such as Packeteer can't handle these high rates, and even if they had gigabit interfaces, they would only be able to do 600-800mbps on them.

None of the kernel QoS/queueing options I've seen allow for anything other than classifying traffic or "fair" queueing. None of this seems to help someone that wants to limit all webserver connections to 2mbps - everything here is expecting an IP range, ports, or something to distinguish by. What if I don't want to?

Apache needs real per connection, per user, and per IP rate limiting. mod_throttle and everything else I've seen has to starve connections after they perform too well. How about something that hard limits connections to 2mbps/sec. I will pay for anything that can do that for Apache today...

Forgive me if I have overlooked the obvious...

Re:You're missing the point

[satch89450]

Apache needs real per connection, per user, and per IP rate limiting. mod_throttle and everything else I've seen has to starve connections after they perform too well. How about something that hard limits connections to 2mbps/sec. I will pay for anything that can do that for Apache today...

Then head for eBay, because a moderate-cost solution to your particular problem (limiting all web traffic to 2 megabits/s) is available for two bids and some cable work: buy two Ascend Pipeline 130s and run them back-to-back with a T1 cross-over cable. Another advantage of this solution is that your web server can be located near the webmaster, up to 5000 feet (without repeaters) from your network access point. Indeed, if you partition all of your services (mail, news, web server, ftp server) then no one service can completely swamp your connection.

Don't like using T1 routers? Then get a moderately powerful Intel computer, install enough Ethernet interfaces to satisfy your needs, load up a modern Linux distribution with 2.4.18 kernal and IPTABLES, and set up rules that will traffic-limit to the interface to which you connect your Web server. If you are like a lot of people who run multiple servers on the same box, the rules can "customize" the throttling by service. Not only that, but you can throttle by direction as well: incoming HTTP could be limited to 30 kilobits/s while outbound HTTP could be limited to 3 megabits/s -- that takes care of some of the problems with DoS attempts on HTTP. The same can be done for other services, such as FTP, mail, and IRC. The amount of control that IPTABLES provides is, well, interesting.

(Yes, I know that the *BSD people have something similar, but I know the IPTABLES stuff better and have seen it work.)

C'mon, people, this isn't all that hard to do if you think and are willing to put a little money where your wishes are.

Re:You're missing the point

[Electrum]

I am having the same problem, and I think you guys are missing the point. He said 4GB an hour, which means he probably has an OC-3, OC-12, or Gigabit Ethernet connection.

That's only 9.1 mbps. T1 = 1.544, T3 = 44.736, OC1 = 51.84. OCx = OC1 * x.

Use Zeus

[Electrum]

High traffic and Apache is almost an oxymoron. If you are running a high traffic web hosting company, then you need to stop playing games and use Zeus. Apache has its strong points, like being free and open source, but that's about it. If Zeus was free, then it wouldn't just be the best web server for UNIX platforms, it would also be the most popular.

You want Zeus because it is high performance (it doesn't use the toy process-per-connection model). It comes with an easy to use, powerful web based GUI. The GUI doesn't just hold your hand. It lets you set everything, and then will show you the exact lines that are changing in the config files.

It doesn't use extremely complex format for config files that Apache uses. A good comparison is BIND and djbdns. Do you want to try and deal with the incredibly complex BIND zone files, or the simple, one record per line data files that djbdns uses? Zeus config files are one record per line of the form "modules!throttle!enabled yes". It also comes with tools that let you do everything from scripts. But only if you want to. Otherwise, use the GUI.

And speaking of throttling, Zeus does it correctly, unlike any other web server (at least any of the freely available UNIX ones, as that is all I am familar with). It will let you set a limit on the number of users, or set a max number of bytes per second on a virtual server or subserver level. It doesn't serve some people at max speed and then start dropping connections (mod_throttle) or set the throttle speed at the beginning of the request, then start dropping connections (thttpd).

Virtual servers in Zeus actually make sense. There is no master server configuration like in Apache. Instead, you create one or more virtual servers. As such, each virtual server has its own separate configuration. Virtual servers can serve a single website, or any number of websites, via subservers. Subservers all share the configuration of the virtual server (kind of like Apache's mass virtual hosting only much better). No more restarting the server to add a site. Simply create the directory, and it starts serving the site.

There are plenty of other reasons why Zeus is superior to Apache, but the ones I listed should be enough to start considering it. No, I don't work for Zeus or own stock (don't think they have any) or anything like that. I'm just a satisfied customer.

For some things, Apache works just fine. But for anything high traffic, requires throttling or needs a flexible or scripted configuration, Zeus beats Apache hands down. It's worth every penny. Check it out. I doubt you'll be dissapointed.

(subconscious message to Apache developers: stop being lazy and make Apache more like Zeus!)

heh...

[keepper]

how much are you going to pay me then?

Using ipfw and dummynet on freebsd is the way
I have gone in a VERY high traffic hosting
and colo company.

you can not only simultae a link of a certain speed, but can also limit any ip that hits a certain destination to a max speed...

:-P