Bandwidth Limiting Policies for Web Hosting?

Silas asks: "I run a small website development and hosting company. We're trying to develop creative, fair, but standard policies in limiting the bandwidth of our individual hosting accounts. I seek the opinions of Slashdot readers who have experience as hosting providers or hosting users. More details below. We're running Apache, and have pretty much decided on using mod_throttle as our bandwidth limiting technology. I know it's not everyone's favorite, but it looks great for us. We have less than 200 domains being hosted, all with varying degrees of bandwidth requirements. As you might suspect, we've got our own ideas and have done our own research about the answers to these, but now I'm interested in yours."

"The basic question is 'what's fair and standard' in these areas:

• Our two hosting packages offer 5 GIG/month and 10GIG/month respectively, with the option to upgrade in $5 per 1 GIG/month increments. Other hosting providers seem to be all over the board - what's the average hosting account want/need?
• The policy that seems common is 'allow a certain amount of data to go through in a certain time period, and then start rejecting requests until the end of the time period'. Is that fair? What policies do other hosts use? When is it appropriate to delay the response to a request instead of rejecting it?
• What should the user be able to do automatically in terms of upgrading/controlling their bandwidth usage? If a user is fine with 5 GIG/month but then gets slashdotted, what should their options be (right away, within 24 hours, etc.)?

We know we're not a powerhouse hosting company, and we know we can't compete with the big players in this industry, so please avoid comparisons/critiques based solely on price points, as we'll likely never be able to match the bulk hosters in that area. We're focused on excellent customer service and convenience to our design clients."

Comments

Re:I'd LIKE a cap on my accounts.

[LordNimon]

Yes, this is very important. You should give the user the opportunity to have the site shut down if it hits is bandwith cap. For personal web pages, it doesn't really matter if the site is up or down, but the overcharges can be very painful. I once had an $80 bill one month because I happened to put up a large web page that was very popular. I could have moved it to another site, but I didn't realize what was going on until I was over my usage. Bastards didn't even bother to email me!

Put it to the users

[__aafkqj3628]

I would set it up so the users have the option of "after 5gig, ignore requests" or to start paying for extra bandwidth at the normal rate (to a certain point/unlimited).

Throttling simply does not work

[Electrum]

We gave up on throttling. It just doesn't work. And not from a technical standpoint, either. If you are going to do throttling, then you need a web server that does real throttling. The only one I know of is Zeus. It does real throttling, letting you limit the total bandwidth for all of a user's sites to a bytes/sec value. No Apache modules do this. Even thttpd doesn't seem to get this right.

I assume that you want to limit bandwidth that your customers use because the bandwidth to your server(s) is limited (i.e. you don't have a 100mbit connect to the internet). In this case, Apache modules will not do what you want. Someone puts up a 10mb file. That file gets downloaded and uses up all of your outgoing bandwidth. While it is being downloaded, the Apache throttle module refuses other requests. This is obviously not what you want.

So suppose you end up using Zeus, or find some other way to do real throttling. Now what do you set the throttle speed to? 5gb over a month averages a little less than 2k/sec. Say you set it higher, like 20k/sec, and there are ten connections downloading that user's files (which can easily happen with certain browsers). What does the average clueless user or webmaster think? They don't understand throttling. They just think that the website is slow and that your service sucks.

We would throttle down user's sites when their bandwidth ran out. Customers did not understand that they had run out of bandwidth, even though they were notified via email. They just thought their sites were slow. We received a lot complaints about that.

We found that the best thing to do is to not throttle and to presell bandwidth cheap. Our different packages come with different amounts of bandwidth, ranging from 5gb to 180gb. After that, customers can purchase extra bandwidth (for $0.50/gb). Customers receive a notification via email when their bandwidth is running low and again when it is completely gone. When their bandwidth is gone, we redirect their sites to a page stating that they used up all their bandwidth.

This solution is simple and it works. Customers always know how much bandwidth they have left and can buy more at anytime. We never have to worry about users running up a huge bill and not paying it, since everything is prepaid.

Re:Please tell us HOW you did this!

[Electrum]

How exactly did you go about setting this up? Can you go into details?

It's really not that difficult, if you have a good hosting setup. Our hosting system is designed to be 100% automated, all controlled from our custom written control panel.

We use Zeus' virtual server feature. Every customer has their own virtual server. In Zeus, everything is a virtual server. There is no concept of a main server like in Apache. Thus each virtual server can have its own configuration and be independently controlled.

Each virtual server can have subservers. Subservers is similar to Apache's mass virtual hosting (but better). A virtual server has a subserver directory, which is a directory full of symlinks with the name of the site hostname, pointing to the site content directory. When a customer runs out of bandwidth, we simply place a .htaccess file (Zeus mostly supports these) with a redirect line in the root of the subserver directory. Due to the directory layout, customers do not have access this directory.

If I had to set this up under Apache (ick), I would use the mass virtual hosting module. The virtual host directory would be full of symlinks with the name of the site hostname, pointing to the site content, as with Zeus. The difference with Apache is that all sites for all customers would be in one directory. Apache configuration is much more messy and not easily scriptable. To redirect the disabled sites, the symlinks would be changed to point to a different directory.

Accumulating bandwidth statistics for this can be done in different ways. We use a custom ISAPI module which sends logging info through a socket to a daemon which accumulates statistics and periodically saves them to a MySQL database. Doing a SQL query for every web request would not work.

For Apache, a similar module could probably be written. Or server logs could be processed. Logs could be written to a pipe and the process on the other end could accumulate bandwidth and other statistics (or write out logs for something like Analog to process).

A script runs from cron that processes the bandwidth statistics in the database. It updates the bandwidth counters in the database for the user, making it easy to see how much bandwidth is left from the control panel. When a customer is low or runs out, the script sends a notification email. When the customer is out of bandwidth, the script updates a table which tells the backend hosting daemon to actually disable the sites. After the customer buys more bandwidth, the script updates the table telling the daemon to enable them again.

(Hint: SQL databases are a very nice way to let daemons on different servers communicate with each other.)