Morality of Throttling a Local ISP?

An anonymous reader writes "I work for a small (400 customers) local cable ISP. For the company, the ISP is only a small side business, so my whole line of expertise lies in other areas, but since I know the most about Linux and networking I've been stuck into the role of part-time sysadmin. In examining our backbone and customer base I've found out that we are oversubscribed around 70:1 between our customers' bandwidth and our pipe. I've gone to the boss and showed him the bandwidth graphs of us sitting up against the limit for the better part of the day, and instead of purchasing more bandwidth, he has asked me to start implementing traffic shaping and packet inspection against P2P users and other types of large downloaders. Because this is in a certain limited market, the customers really only have the choice between my ISP and dial-up. I'm struggling with the desire to give the customers I'm administering the best experience, and the desire to do what my boss wants. In my situation, what would you do?"

Comments

bill, don't throttle

[seanadams.com]

This is not a hard problem. You can not maintain a reasonable oversell ratio unless you have low average usage. Yes, one way to get that is throttling, but it's difficult to do that in an effective way that won't piss off your customers.

What you should do is tell them they get 40G/mo or whatever, plus a usage fee above that, and let the customers throttle themselves if they want to. If you want to be a nice guy about it, you could give them the option of being auto-throttled or suspended if they approach the limit, so they don't get an unexpected bill. Of course whatever you do, you'll need to revise your terms of service.

Voila, you maintain low pricing and good performance for everyone, because the p2p guys will police themselves now. If you have customers that routinely transmit hundreds of GB because they're a professional video editor or something, then they won't mind paying for the bandwidth.

Re:bill, don't throttle

[geekboy642]

From the tone of the article, it doesn't sound at all like subby has the freedom to change the ToS or implement hard caps.

In my opinion, the best solution is to strongly throttle large bandwidth usages (P2P, FTP and NNTP streams, etc) during the periods of near-capacity, and automatically relax the filtering during off hours. A simple email or letter to your subscribers to announce the change, and everybody will be happy. As a bonus, the notification of the changes will help to encourage your subscribers not to attempt to circumvent your filters, especially given that it's so easy for any modern downloading client to schedule for off-peak hours.

Re:bill, don't throttle

[volsung]

Amen, but to add to this: If you are going to institute some kind of usage billing, it is *absolutely* critical you give people the tools to monitor their usage. At a minimum, there should be a web page that customers can view their current usage (no more than 24 hours old) relative to the quota. For bonus points, give people the ability to get email updates when they pass predefined levels, or if their one-day usage exceeds some value.

Re:bill, don't throttle

[bigcmoney]

I run a similar sized WISP. All I do is use NFSEN to see who is using the bandwidth, and then give them a call. Almost all the time the customer's kids are doing the downloading, or they have a virus. This level of service really makes the customer appreciate doing business with you.

Re:bill, don't throttle

[PhoenixAtlantios]

P2P throttling? Not here.

Exetel do, and we know of this only because they've been vocal about it; other ISPs may do it with more subtlety.

Forbidding servers on residential connections? Not here.

The Whirlpool broadband survey 2008 disagrees (search for "not allowed to run server", optus certainly restricts it).

So while the majority of ISPs don't do it, you shouldn't make out that it's all sunshine and roses in bandwidth cap land; some of the larger ISPs (Telstra and Optus) measure both uploads as well as downloads when considering your monthly bandwidth cap too (which seems to be an effective way to reduce p2p since you'll hit your cap that much faster by "giving back").

I agree that shaping connections rather than billing for excess usage makes more sense for ADSL/Cable connections though; it's much less daunting to get throttled as opposed to being charged extra. Internode have implemented a "Data Block" system that allows you to purchase chunks of bandwidth to extend your monthly cap in a pinch if you're about to get throttled (i.e. it isn't cost effective to do regularly) which could be worth looking into later on.

One more thing, if you do implement caps you'd want to look into some sort of monthly usage meter that's easily accessible to your customers. Net Usage Item is an example of a Firefox addon that tracks usage from various ISPs that helps people avoid overrunning their caps.

Re:bill, don't throttle

[mysidia]

There's really very little moral question here, you are selling a service. The quality of the bandwidth you use, and whether the same amount of bandwidth is available in bulk heavy usage, for bulk file transfers, as for normal, expected usage patterns, is your call as an ISP.

And for the most part ISPs don't buy a bit of internet bandwidth, for every bit of subscriber bandwidth. This practice is not oversubscription (per se), you should calculate the expected usage patterns for your average subscriber, and multiply by your total number of subscribers, and add 'safety' factors for flash crowds; as for P2P applications and "bulk data transfers", you should do the math there as well, and determine, what proportions of your traffic are P2p transfers.

Keeping usage of heavy users under reasonable control just as much about providing everyone a quality service, as it is about 'saving on bandwidth bills' -- because, even if you add more bandwidth, downloaders will manage to eat it, if you don't put something in place.

And ISPs all over the country are taking measures to limit P2P's usage, so a few users don't get to hog all the network resources, or to overutilize.

This is not so much a justification based on the theory "everyone is doing it", but more a justification based on "your consumers probably expect you to do this" (do your best to block, prevent, or control, excessive usages from other subscribers that would degrade their services)

What you should do is tell them they get 40G/mo or whatever, plus a usage fee above that, and let the customers throttle themselves if they want to....

He only has 400 customers. There's not enough play here to provision capacity on demand, if a few users want to heavily use the service, he may need to get commitments for this to be affordable.

They can stay below those monthly limits and still cause major problems, if they happen to all be on at the same time fully utilizing their pipe fairly continuously.

Also, consumers will rightly be concerned about the possibility of malware or unwanted DoS attacks artificially inflating their bandwidth bill.

There are a lot of good things to be said for using technologies like NBAR and policing to reduce the flow of unwanted traffic.

Actual general shaping is not recommended, as it will very possibly degrade proper operation of the service, for non-bandwidth-hungry users.

Re:bill, don't throttle

[MrEricSir]

If my ISP called, that's what I'd tell them too.

"Yeah, my 'kids' must be 'downloading' a lot of stuff. Don't worry I'll go spank them until they stop."

Re:bill, don't throttle

[causality]

Do you really have reason to believe that dissenting is "a down-moddable 'offense'," or are you just letting off steam because AC's always start at -1 and seldom get up-modded unless they're presenting a view that is relevant to the discussion? I can't speak for anyone else but I use about 95-98% of any mod points I'm given to reward those who say something meaningful, informative and/or funny in relation to the subject raised in the original summary. So, in answer to your implied question, I'd be glad to have the same moderation standards applied to my submissions.

FYI, Anonymous Cowards don't start at -1 by default. By default, they start at 0. You can modify your personal preferences so that they are displayed as -1 (or +5 if you really wanted...) but of course that is unique to your own account.

Incidentally, you really don't sound like the sort of moderator that the GP was talking about and I mean for that to be a compliment. That you realize promoting good posts is a better use of your points than demoting bad ones is strong evidence that you're one of the better moderators. As someone who values constructive criticism (the real thing, not personal attacks veiled as constructive criticism), who often takes relatively controversial positions and enjoys challenging people to think in new ways, I can tell you from my own personal experience that there is a lot of poor-quality moderation going on. You won't see that very much for posts that just repeat a "party line" (almost anyone's party line) but you do see this targeting some of the more freethinking posters. It became much more noticable after the old metamoderation system was "upgraded". I am not at all surprised that I more and more frequently see a backlash against it to be honest with you. It's not that it's so terrible so much as it is that this is heading in the wrong direction so the bad examples are slowly becoming more common.

To me the situation is quite easy to understand: weak or insecure people think that even the most civil disagreement or the most constructive criticism is an attack against them and they look for ways to retaliate. When those people are moderators, they retaliate by enforcing drastically distorted standards of "flamebait" or "troll" or "offtopic".

Re:bill, don't throttle

[Mistlefoot]

Metering almost positively requires Docsis compatible cable modems. For a business that runs cable as a sideline, as per the submission, I would guess there would be a reasonable chance they don't have the most up to date equipment. With 400 subscribers it's also difficult to implement many high cost options. Will setting up filtering actually cost more than providing more bandwidth? How cost effective is it (it's easy with docsis) different speed options (ie 512 down - for basic email/chatting etc., 5120 for the average user and maybe 10240 for high users, priced incrementally) 400 subscribers in a limited area - where the user base likely won't increase a large amount, isn't going to allow for many cost effective options in my opinion.

Re:bill, don't throttle

[Jurily]

"Yeah, my 'kids' must be 'downloading' a lot of stuff. Don't worry I'll go spank them until they stop."

But if it turns out to be a virus, you get a) a happy customer, b) reduced bandwidth usage, and c) the world will be a slightly better place. All for a phone call.

Re:bill, don't throttle

[Anonymous Coward]

You made an interesting and valid initial point about the quality of the Ask Slashdot section. Regardless of the status quo, this is the type of comment I generally mod up, and in fact, I happen to personally agree with your assessment. But you then spent the next 3 paragraphs, the majority of your post, bitching about how valid conversation points are suppressed and insulting the moderators.

Some mods do take their modpoints and the moderation system in general as something worthy of actual effort because it benefits the community in some minor way and because we wish to reciprocate the benefits that others have provided us, so it took me a moment to consider your post. In the end, I believe that you overshadow your relevant and valid argument with irrelevant assertions about its validity and complaint bordering on flamebait or ad hominem, so I cannot mod this up. I hope that you've vented enough steam to make your next post more succinct. I also believe that there should be a 'meta' tag; something to indicate that the post is valid but about slashdot itself, as there is no proper forum for this kind of conversation.

Re:bill, don't throttle

[c0p0n]

You seem to live in a world in which every person is a geek and knows what you're talking about.

Re:bill, don't throttle

[drsmithy]

Um, if you're paying for X bandwidth, but getting less than that, wouldn't you consider that a "limit"?

You don't pay for "X bandwidth" on home internet connections, you pay for a particular type of connection with a theoretical maximum that the provider makes quite clear a) you may never actually reach (eg: due to unavoidable technical limitations like distance from the exchange) and b) they are under no obligation to deliver at all, let alone constantly.

This is in the fine print of pretty much every ISP contract you'll ever see. It certainly has been in every one that I've ever read. A consumer-grade internet connection is a "best effort", not a contractual SLA, and no remotely intelligent person (and even most stupid ones) seriously believes otherwise when they sign up, no matter how much outrage they might feign on Slashdot afterwards.

Finally, even most of the "unlimited" plans usually only talk about "unlimited downloads", not "unlimited bandwidth", which is my point - it's a measure of volume, not speed.

You're stuck.

[numbski]

Here's the thing - you have no choice. Do the shaping.

That said - form a compelling argument for doing the right thing, and present that to your boss. Don't defy him, but give him a reason to reconsider. In the meantime, do as you're told. You can always undo shaping. Don't screw your employment in the interim.

Re:You're stuck.

[ssj152]

Better read the current terms of service first - yanking the rug before changing the terms of service frequently leads to lawsuits. Be nice to the pointy-haired one, but point out the likelihood of legal problems here. Also, I liked the first responder 'seanadams' suggestion as an actual solution - if there is no way to actually get the bandwidth upped.

Re:You're stuck.

[Anonymous Coward]

Agree 100%. You're getting paid for this work. It doesn't matter how much you admire Casper the Friendly Geek, it's neither your right nor your job to contradict your boss's decisions. If your customers don't like the service, they'll find alternatives or drop his service, and then he'll either deal with the revenue loss or improve.

Make the business case for it. Feel free to refuse to do anything actually unethical or illegal that he asks you to do. This is neither, so suck it up. Or, alternately; you're suggesting that this is a really small market. I assume that means there's not that much tech know-how in the area. That gives you another option; if you really care that much about not throttling, quit. It'll take him a while to replace you, and that's months and months more that your users can illegally download Battlestar Galactica to their hearts' content.

Oh, man! I need to go watch the latest BSG I downloaded!

The choice is simple

[Bandman]

Petition for your boss to do the right thing.

While you're petitioning, do what your boss tells you.

If what your boss tells you to do is unethical, quit, and tell him why in your resignation letter.

Re:The choice is simple

[MeanMF]

Exactly... If there's a business case for buying more bandwidth, then write it up and show it to the boss. Are people dropping the service because they're fed up with slow speeds? Are there people who would be willing to pay more for higher bandwidth? Do the customers even notice or care that speeds are slow at times? Is 90% of the bandwidth being used up by 1% of the customers? If you don't know the answers to these questions, whining to the boss isn't going to get you anywhere.

Add a free period

[grahamsz]

I had a situation once where my bandwidth was metering during regular hours but free from midnight - 7am. Any smart heavy user will set up their downloads to happen during the free period and take the load off the network during peak hours. I've never understood why more ISPs don't do that.

If you just tell people they have a 40G cap then they'll feel entitled to use it whenever they want, and you really can't argue with that.

Striking a balance.....

[Computershack]

You honestly know in your heart that most of the P2P traffic is illegal so throttle it BUT only implement the throttling between the hours of say, 8am to 10pm or midnight. Send out an email to all customers stating that due to the abuse of a minority of users, P2P throttling will take place between the hours of 8am to 12 Midnight to ensure a high level of service to other users.

The P2P boys will quickly figure out what is going on and they can set their clients to download from Midnight to 8am. That way, there's plenty of bandwidth when Joe Average wants to check their Facebook and when businesses are operating and the bandwidth through the night which is mostly unused is utilised better. Everyone wins.

Re:Striking a balance.....

[z0idberg]

due to the abuse of a minority of users,

If they signed for and are paying for unlimited internet access then where exactly does the abuse part come into it?

Re:Striking a balance.....

[meerling]

As they don't know what the P2P traffic is, you can't say it's illegal. Statistically, it probably is violating a copyright, but that isn't sufficient justification for singling out the P2P traffic alone. That would be like sending everyone in your city with a drivers license a traffic ticket, because you just know that virtually all of them will speed, roll through a stop sign, or commit some other traffic violation this year.
Besides, he didn't even mention what kind of traffic was going on during peak hours, just that the company is (my interpretation) screwing customers by oversubscribing them 70:1 (his statement).
It's possible that their biggest traffic spike is youtubers. Until someone does an analysis, you just won't know.

Re:Striking a balance.....

[z0idberg]

Bullshit.

So what about someone that uses a media centre PC as their television input and watches Hulu and mlb.tv etc. rather than via cable or satellite or whatever during peak periods causing their ISP to hit its bandwidth limits? Is that abuse as well? Is that guy soaking up bandwidth or is he using what he is paying for?

Screw morality. Get pragmatic: prioritize traffic.

[hessian]

Morality is a tool for the herd to feel more important than their leaders. Instead, get pragmatic: how can you make this business work for most people?

You probably want heavy downloaders to use another service, anyway. You might even consider setting up two plans, one for ueber-users and one for normal users.

However, I would prioritize traffic. Email, web, SSH, et al come first; after that, all p2p protocols in order of usefulness.

You need to define your business audience. If it's people who are going to check the mail and web surf, and 5% of your customers are p2p users, cut out the p2p users and focus on the people you want to serve.

shape and/or prioritize that traffic

[itzdandy]

Im wondering what you have for backbone that you are 70:1 oversubscribed. If you deploy 768/256 connections with 400 customers sounds like a whopping 3 T1 lines (~4.5Mb/s). if you do a more standard 1.5MB thats 6 T1 lines(~9Mb/s).

Maybe you should look at your upstream provider and see if you can get a fractional T3 to replace the T1s if my math is anywhere near correct. You will likely have a longer contract to sign but you may be able to pull in 10Mb/s for less than you currently pay. Then you could try to match the current expense.

There are other ways to trim back your backbone usage. Consider a cluster of transparent proxy servers. You can get pretty aggressive with the cacheing mechanise in squid and you can easily balance the cluster with DNS and not have to worry about session awareness as clients also cache DNS temorarily so each client will use the same proxy for their browsing session.

Certainly some sort of QoS will work for you and lessen the need to directly throttle.

If you just throw some proxying in there and give http and https higher priority and do some packet inspection to sniff out the P2P traffic and drop it down a level you will put off the inevitable need to grow your bandwidth for a while.

if my math is correct on 1.5Mb/s cable, you look like you have a per users upstream cost of just $7.50 each. That is pretty low. Too low.

Morality?? HA!

[iminplaya]

There is no morality for throttling. It's done for either technical or business reasons.

Remember, you've got a job now

[davmoo]

What would I do? I'd start by doing what the boss says. This is a really bad time to have to look for employment elsewhere. If you don't do what the boss says, customers of your former employer are not going to start sending you money to live on because you did the "right" thing but lost your job.

Then after things have been at least temporarily taken care of, research better alternatives and present them to your boss.

BS.

[msauve]

He has no choice but to honor the contract they've made with customers.

If, as most cable companies do, they've contracted to provide "unlimited" service, at "xx Mbps rate", then that's what they need to provide.

If such is the case, then throttling anyone is fraud.

Re:Morality?

[schon]

That's not the issue. The issue is whether there is any legal P2P traffic. geekboy642 proved there was, and you didn't offer anything to refute it, so I guess that you agree with him.

Since you agree that there *is* legal P2P traffic, the argument that "it's illegal so there is no problem throttling it" is a non-sequitur.

Re:Tell your boss you quit ...

[intx13]

Ick.. who mods this stuff up? Since when is "quit unless you get your way" a good policy? Maybe the parent is filthy rich, hasn't read the news in 6 months, or is a child, but right now is not exactly the best economic climate in which to be clearing out your desk. There are requests a boss could make that might be so morally appalling that you feel the need to quit on the spot... but imposing throttling on some customers? Probably not one of those requests.

Second, what's so evil or innately wrong about throttling? So long as you don't violate your contract - and ethically, in my mind, don't violate the spirit of your contract either (i.e. tiny print doesn't make it ok) - then what's the problem? The parent is acting like the act of throttling is a "sin"; it's just a technology.

Is oversubscription really "evil"?

[Illusion]

Your details are a bit vague, but let's pretend "your pipe" is a single DS3 (45 megabits) out in the boonies somewhere and you are offering a mix of plans that average out to 7.8 megabits per customer (400 * 7.8 / 70 = 44.5).

Assuming you are in the US, 45 megabits of transit is unlikely to cost you more than ~$2k/month ($50/megabit transit is easy to come by, you can do way better if you shop and have access to many carriers), but due to the amazing power of phone company pricing, the DS3 to carry it could easily run $10k-40k/month depending on how far out of a major city you are. (Within a major city, DS3s are closer to $3k/month.) Let's use the low end of that range and call it $10000/mo for the DS3 and $2000/mo for the bandwidth, or $12000/mo total for 45 megabits or your total cost of ~$267/megabit.

If your customers were to demand no oversubscription (as most Slashdotters seem to), delivering a 10 meg cable connection would therefore cost you $2670/month to deliver to your customers. At standard retail markup (including maintaining the cable lines, buying routers, paying rent, paying salaries, etc) of ~2x, let's call it $5k/month per customer. This poses a problem, since no residential customer will pay $5k/month.

If you work it from the other angle, starting from what your customers will pay, let's pretend they are comfortable paying $80/month for their 10 meg cable connection. (This is high if they were in a city, but if this is their only option vs dialup, they'll buy it anyway.) Assuming you have some overhead and only half that can pay for bandwidth, you have $40/month for 10 megabits or $4/megabit.

How do you reconcile that your customers will only pay $4/megabit when your costs are $267/megabit? The magic of oversubscription.

These customers need to be willing to live with the idea that they are expected, on average, to use only 143Kbit/sec on their 10 meg pipe. If on average they want more than that, they have to be willing to pay for it, otherwise the ISP is just going to fold, and they can go back to dialup.

For some reason, Slashdotters see this as evil. Is it? How else can you make the numbers work? (Most of these numbers are ballpark since the posters details were so vague, but they real-ish.)

Re:Do it by usage, not by protocol.

[Cylix]

The lesson of the day is not to base critical business applications on consumer bandwidth.

This is why nearly every ISP I have dealt with or worked with offered a free for all business package. Sure, they cost a bit more, but it's usually worth it.

Re:I wouldn't be happy

[daveime]

Ah but you NEVER buy "6mbps" ... you always buy "up to 6mbps" or "maximum 6mbps", and then try to conveniently ignore the bits you don't like in the deal.

Please, cite me just one ISP who offers a "guaranteed 6mbps available 24/7", and I'll gladly admit I am wrong (right after I sign up with them).

Stop fretting and start analysing...

[DarkRecluse]

I think you need to make sure the traffic you are seeing is actually P2P. I would highly doubt it given your subscriber to bandwidth ratio. The majority of "normal" long flow traffic is actually http. Mostly flash video or http downloads. That said, you have such a high ratio that it's possible its not even downloads hitting up against your cap. If you have as flat a usage pattern as you say you have, it likely already sucks to be your customer doing anything at all at peak times. People would do better on dial-up....at least it would be consistent and they wouldn't get stuck with nil at certain intervals.

Confirm you have a P2P problem before you start shaping. If you tell your boss the traffic is mostly http no amount of packetshaping is going to fix this problem to anyone's satisfaction(unless it actually is all http downloads).

Since you're on a tight budget already, I recommend running nTop on a box connected to a mirror or span port. That would be an easy way to determine what's actually going on.

When presented with the fact that shaping is pointless your boss will either buy more bandwidth or do nothing at all. Either way you aren't forced to shape. If he chooses the second option your customers should make him uncomfortable or fix the problem altogether by moving to dial-up.

Shape, not throlle or cap

[AigariusDebian]

Shape, not throttle. If done correctly shaping is what makes a difference between a good ISP and a great ISP. It is not a problem to detect P2P traffic and shape it to a lower priority, provided that you shape important traffic as high priority - ACK's, Skype voice, game traffic (WoW, CS, ...), first 100k of any HTTP or HTTPS connection, SSH, ...

As a power user it is not that critically important that my torrents only come at 16kb/s during the day if my web, games and IM apps are snappy, but I would like to have the torrents saturate the pipe during off-peak.

Also, hard caps are overrated - you don't pay per Gb, why should we? Just prioritise traffic correctly and everyone will be happy.

Nah man...

[phatslaab]

In this current economic environment are you really surprised they are asking you to throttle instead of paying for bigger pipes? It is not your moral duty to ensure that people get the best internet experience. You do what you have to do to enforce your company goals and standards within the situation you've been given and ensure that they don't step across your moral standard - ie. lying, cheating, murder, etc. (If that is your particular moral standard. I once knew of a man who killed his wife yet felt morally bound to OSS for some reason. Hmmmm...) To throttle or not to throttle has little to do with your own morality.