Slashdot Mirror
Morality of Throttling a Local ISP?
An anonymous reader writes "I work for a small (400 customers) local cable ISP. For the company, the ISP is only a small side business, so my whole line of expertise lies in other areas, but since I know the most about Linux and networking I've been stuck into the role of part-time sysadmin. In examining our backbone and customer base I've found out that we are oversubscribed around 70:1 between our customers' bandwidth and our pipe. I've gone to the boss and showed him the bandwidth graphs of us sitting up against the limit for the better part of the day, and instead of purchasing more bandwidth, he has asked me to start implementing traffic shaping and packet inspection against P2P users and other types of large downloaders. Because this is in a certain limited market, the customers really only have the choice between my ISP and dial-up. I'm struggling with the desire to give the customers I'm administering the best experience, and the desire to do what my boss wants. In my situation, what would you do?"
This is not a hard problem. You can not maintain a reasonable oversell ratio unless you have low average usage. Yes, one way to get that is throttling, but it's difficult to do that in an effective way that won't piss off your customers.
What you should do is tell them they get 40G/mo or whatever, plus a usage fee above that, and let the customers throttle themselves if they want to. If you want to be a nice guy about it, you could give them the option of being auto-throttled or suspended if they approach the limit, so they don't get an unexpected bill. Of course whatever you do, you'll need to revise your terms of service.
Voila, you maintain low pricing and good performance for everyone, because the p2p guys will police themselves now. If you have customers that routinely transmit hundreds of GB because they're a professional video editor or something, then they won't mind paying for the bandwidth.
Here's the thing - you have no choice. Do the shaping.
That said - form a compelling argument for doing the right thing, and present that to your boss. Don't defy him, but give him a reason to reconsider. In the meantime, do as you're told. You can always undo shaping. Don't screw your employment in the interim.
Karma: Chameleon (mostly due to the fact that you come and go).
I had a situation once where my bandwidth was metering during regular hours but free from midnight - 7am. Any smart heavy user will set up their downloads to happen during the free period and take the load off the network during peak hours. I've never understood why more ISPs don't do that.
If you just tell people they have a 40G cap then they'll feel entitled to use it whenever they want, and you really can't argue with that.
Morality is a tool for the herd to feel more important than their leaders. Instead, get pragmatic: how can you make this business work for most people?
You probably want heavy downloaders to use another service, anyway. You might even consider setting up two plans, one for ueber-users and one for normal users.
However, I would prioritize traffic. Email, web, SSH, et al come first; after that, all p2p protocols in order of usefulness.
You need to define your business audience. If it's people who are going to check the mail and web surf, and 5% of your customers are p2p users, cut out the p2p users and focus on the people you want to serve.
Futurist Traditionalism
Im wondering what you have for backbone that you are 70:1 oversubscribed. If you deploy 768/256 connections with 400 customers sounds like a whopping 3 T1 lines (~4.5Mb/s). if you do a more standard 1.5MB thats 6 T1 lines(~9Mb/s).
Maybe you should look at your upstream provider and see if you can get a fractional T3 to replace the T1s if my math is anywhere near correct. You will likely have a longer contract to sign but you may be able to pull in 10Mb/s for less than you currently pay. Then you could try to match the current expense.
There are other ways to trim back your backbone usage. Consider a cluster of transparent proxy servers. You can get pretty aggressive with the cacheing mechanise in squid and you can easily balance the cluster with DNS and not have to worry about session awareness as clients also cache DNS temorarily so each client will use the same proxy for their browsing session.
Certainly some sort of QoS will work for you and lessen the need to directly throttle.
If you just throw some proxying in there and give http and https higher priority and do some packet inspection to sniff out the P2P traffic and drop it down a level you will put off the inevitable need to grow your bandwidth for a while.
if my math is correct on 1.5Mb/s cable, you look like you have a per users upstream cost of just $7.50 each. That is pretty low. Too low.
If they signed for and are paying for unlimited internet access then where exactly does the abuse part come into it?
As they don't know what the P2P traffic is, you can't say it's illegal. Statistically, it probably is violating a copyright, but that isn't sufficient justification for singling out the P2P traffic alone. That would be like sending everyone in your city with a drivers license a traffic ticket, because you just know that virtually all of them will speed, roll through a stop sign, or commit some other traffic violation this year.
Besides, he didn't even mention what kind of traffic was going on during peak hours, just that the company is (my interpretation) screwing customers by oversubscribing them 70:1 (his statement).
It's possible that their biggest traffic spike is youtubers. Until someone does an analysis, you just won't know.
In my opinion, the best solution is to strongly throttle large bandwidth usages (P2P, FTP and NNTP streams, etc) during the periods of near-capacity, and automatically relax the filtering during off hours.
That's one way... Here's another:
Instead of trying to choose which protocols are heaviest usage, traffic shape people based on what the actual criteria that you care about is: Too much overall usage over long periods.
In Linux terms, set up a HTB with a queue for every customer. Set the base rate to whatever your backbone speed is (1/70th of the customer's line rate), the ceil rate to their line rate, and give them a nice big bucket - say, 120 seconds times their line rate.
Then, people who are normal users - web surfing, downloading an occasional email attachment, etc - will go full bore, any time they want it. People who are bittorrenting will go full speed for a couple minutes, and then decrease down to whatever bandwidth is available. At night, if there's a lot of backbone free, it'll go fast. At 7 PM, they get best effort on whatever is available.
This is a very simplified example. You could additionally shape them so that their web and email will take priority over bittorrent when they're at the bottom of their token bucket, or other fine tuning...
The basic message I'd like to get across is: you don't have to shape based on protocol, because you care about the usage, not the protocol. Just shape based on usage, and let them work out which protocols they want to use.
Your details are a bit vague, but let's pretend "your pipe" is a single DS3 (45 megabits) out in the boonies somewhere and you are offering a mix of plans that average out to 7.8 megabits per customer (400 * 7.8 / 70 = 44.5).
Assuming you are in the US, 45 megabits of transit is unlikely to cost you more than ~$2k/month ($50/megabit transit is easy to come by, you can do way better if you shop and have access to many carriers), but due to the amazing power of phone company pricing, the DS3 to carry it could easily run $10k-40k/month depending on how far out of a major city you are. (Within a major city, DS3s are closer to $3k/month.) Let's use the low end of that range and call it $10000/mo for the DS3 and $2000/mo for the bandwidth, or $12000/mo total for 45 megabits or your total cost of ~$267/megabit.
If your customers were to demand no oversubscription (as most Slashdotters seem to), delivering a 10 meg cable connection would therefore cost you $2670/month to deliver to your customers. At standard retail markup (including maintaining the cable lines, buying routers, paying rent, paying salaries, etc) of ~2x, let's call it $5k/month per customer. This poses a problem, since no residential customer will pay $5k/month.
If you work it from the other angle, starting from what your customers will pay, let's pretend they are comfortable paying $80/month for their 10 meg cable connection. (This is high if they were in a city, but if this is their only option vs dialup, they'll buy it anyway.) Assuming you have some overhead and only half that can pay for bandwidth, you have $40/month for 10 megabits or $4/megabit.
How do you reconcile that your customers will only pay $4/megabit when your costs are $267/megabit? The magic of oversubscription.
These customers need to be willing to live with the idea that they are expected, on average, to use only 143Kbit/sec on their 10 meg pipe. If on average they want more than that, they have to be willing to pay for it, otherwise the ISP is just going to fold, and they can go back to dialup.
For some reason, Slashdotters see this as evil. Is it? How else can you make the numbers work? (Most of these numbers are ballpark since the posters details were so vague, but they real-ish.)
Aaron