Morality of Throttling a Local ISP?
An anonymous reader writes "I work for a small (400 customers) local cable ISP. For the company, the ISP is only a small side business, so my whole line of expertise lies in other areas, but since I know the most about Linux and networking I've been stuck into the role of part-time sysadmin. In examining our backbone and customer base I've found out that we are oversubscribed around 70:1 between our customers' bandwidth and our pipe. I've gone to the boss and showed him the bandwidth graphs of us sitting up against the limit for the better part of the day, and instead of purchasing more bandwidth, he has asked me to start implementing traffic shaping and packet inspection against P2P users and other types of large downloaders. Because this is in a certain limited market, the customers really only have the choice between my ISP and dial-up. I'm struggling with the desire to give the customers I'm administering the best experience, and the desire to do what my boss wants. In my situation, what would you do?"
This is not a hard problem. You can not maintain a reasonable oversell ratio unless you have low average usage. Yes, one way to get that is throttling, but it's difficult to do that in an effective way that won't piss off your customers.
What you should do is tell them they get 40G/mo or whatever, plus a usage fee above that, and let the customers throttle themselves if they want to. If you want to be a nice guy about it, you could give them the option of being auto-throttled or suspended if they approach the limit, so they don't get an unexpected bill. Of course whatever you do, you'll need to revise your terms of service.
Voila, you maintain low pricing and good performance for everyone, because the p2p guys will police themselves now. If you have customers that routinely transmit hundreds of GB because they're a professional video editor or something, then they won't mind paying for the bandwidth.
Here's the thing - you have no choice. Do the shaping.
That said - form a compelling argument for doing the right thing, and present that to your boss. Don't defy him, but give him a reason to reconsider. In the meantime, do as you're told. You can always undo shaping. Don't screw your employment in the interim.
Karma: Chameleon (mostly due to the fact that you come and go).
Petition for your boss to do the right thing.
While you're petitioning, do what your boss tells you.
If what your boss tells you to do is unethical, quit, and tell him why in your resignation letter.
Check out my sysadmin blog!
I had a situation once where my bandwidth was metering during regular hours but free from midnight - 7am. Any smart heavy user will set up their downloads to happen during the free period and take the load off the network during peak hours. I've never understood why more ISPs don't do that.
If you just tell people they have a 40G cap then they'll feel entitled to use it whenever they want, and you really can't argue with that.
The P2P boys will quickly figure out what is going on and they can set their clients to download from Midnight to 8am. That way, there's plenty of bandwidth when Joe Average wants to check their Facebook and when businesses are operating and the bandwidth through the night which is mostly unused is utilised better. Everyone wins.
I only please one person per day. Today is not your day. Tomorrow isn't looking good either. - Scott Adams
For a 400 user ISP, there is presumably only a dozen or so high traffic users...
Privately, encourage them to shift some of their activity to off times, such as late morning and middle of the night - explain to them it will help other users, plus help them too in they'll get better speed while helping to keep prices low.
If not enough voluntary compliance, then try enabling aggressive throttling / shaping during day / evening, but allow unthrottled speed during off-hours for high traffic users.
Presuming the ISP has access to multiple providers, then another option to consider is evaluating how much the ISP is paying for bandwidth - see if there are better options and/or if contracts can be renegotiated.
Ron
You don't want to punish customers for how much they download so much as when they download.
The guy who downloads 100Gb overnight when no one else is online? He isn't a problem.
The 100 users who all connect and download from together at peak hour? They are the problem.
So you want to allow people who don't use the net when everyone else is using it full-speed access. And you want those who use the net at peak hour to be slowed down.
The way to acheive this?
Do nothing and let congestion shape them.
Morality is a tool for the herd to feel more important than their leaders. Instead, get pragmatic: how can you make this business work for most people?
You probably want heavy downloaders to use another service, anyway. You might even consider setting up two plans, one for ueber-users and one for normal users.
However, I would prioritize traffic. Email, web, SSH, et al come first; after that, all p2p protocols in order of usefulness.
You need to define your business audience. If it's people who are going to check the mail and web surf, and 5% of your customers are p2p users, cut out the p2p users and focus on the people you want to serve.
Futurist Traditionalism
Your boss understands his customers and the contracts in place. Your boss understands the political consequences of changing his service under the feet of his existing customers. Your boss has lawyers that understand the legal ramifications of his decisions.
If this is an error in judgement, his customers will let him know by either (1) suing him, or (2) withholding payment, or (3) leaving the service. All three mean less revenue for him no matter the outcome.
Your job is to do what he asks within the law. If you think he is asking you to break the law, talk to your personal lawyer for advice. If you have a moral issue with him, gracefully resign.
Don't stick your neck in the guillotine.
Im wondering what you have for backbone that you are 70:1 oversubscribed. If you deploy 768/256 connections with 400 customers sounds like a whopping 3 T1 lines (~4.5Mb/s). if you do a more standard 1.5MB thats 6 T1 lines(~9Mb/s).
Maybe you should look at your upstream provider and see if you can get a fractional T3 to replace the T1s if my math is anywhere near correct. You will likely have a longer contract to sign but you may be able to pull in 10Mb/s for less than you currently pay. Then you could try to match the current expense.
There are other ways to trim back your backbone usage. Consider a cluster of transparent proxy servers. You can get pretty aggressive with the cacheing mechanise in squid and you can easily balance the cluster with DNS and not have to worry about session awareness as clients also cache DNS temorarily so each client will use the same proxy for their browsing session.
Certainly some sort of QoS will work for you and lessen the need to directly throttle.
If you just throw some proxying in there and give http and https higher priority and do some packet inspection to sniff out the P2P traffic and drop it down a level you will put off the inevitable need to grow your bandwidth for a while.
if my math is correct on 1.5Mb/s cable, you look like you have a per users upstream cost of just $7.50 each. That is pretty low. Too low.
There is no morality for throttling. It's done for either technical or business reasons.
What?
Or to get more BW.
By your description, you are The Man when it comes to this, he won't fire you, he is forced for more BW. He can't replace you because you refuse to teach your follower if it goes that route, and in effort he would loose the ISP business.
What stuns me, people are ALL UP FOR THROTTLING! Give me a break! Everyone here recommending it is either shooting themselves on their legs due to sheer ignorance or working for a anti-net neutrality party.
To really start saving BW, think about caching, you can rather easily implement transparect proxy using squid and simple routing rules, and your customers won't notice a thing even if WWW traffic is cached. On that size it sums up to quite considerable amount of data.
You can consider other caching methods too, but you can also implement QOS, prioritize SSH and WWW, and immediate increase in service quality achieved, given you use powerfull enough routers.
Any kind of throttling beyond mere QOS is plain and simply EVIL.
Pulsed Media Seedboxes
What would I do? I'd start by doing what the boss says. This is a really bad time to have to look for employment elsewhere. If you don't do what the boss says, customers of your former employer are not going to start sending you money to live on because you did the "right" thing but lost your job.
Then after things have been at least temporarily taken care of, research better alternatives and present them to your boss.
I want a new quote. One that won't spill. One that don't cost too much. Or come in a pill.
He has no choice but to honor the contract they've made with customers.
If, as most cable companies do, they've contracted to provide "unlimited" service, at "xx Mbps rate", then that's what they need to provide.
If such is the case, then throttling anyone is fraud.
"National Security is the chief cause of national insecurity." - Celine's First Law
The answer to this, and many such sticky situations in IT, is to update your resume` and leave town.
The way I see it, you're screwed if you throttle, and you're screwed if you don't throttle. Some of the solutions given sound good and well on paper. But then again, so does communism.
The game.
That's not the issue. The issue is whether there is any legal P2P traffic. geekboy642 proved there was, and you didn't offer anything to refute it, so I guess that you agree with him.
Since you agree that there *is* legal P2P traffic, the argument that "it's illegal so there is no problem throttling it" is a non-sequitur.
This is a prime example of why the telecommunications, medical, banking and the power industries just need to be nationalized. These people or not going to be told what is going on, there are going to be no changes to there terms of services and more than likely this guy is going to be fired. In the end they will throttle the entire network, put hard caps in, and close the accounts of people who make a fuss. And probably turn them in to the RIAA/MPAA.
Linux modi 2.6.26-2-parisc
If the bandwidth throttling negative affects ONE "legitimate user" (which is always subjectively based on your own personal judgment) then you are doing something wrong.
What I would do is research solutions from this page (I like the QoS/Caching ideas in particular). Then, I would prepare a lengthy, extremely technical, report to your boss which would explain that by implementing this new system it will:
1) Cost your company a ton of time and money
2) Never be completely possible to implement
3) Is unethical based on your principles
4) Will be extremely difficult to maintain, wasting more time and money perpetually
"In this economy" arguments in favor of becoming a corporate zombie are BULLSHIT. I will not sacrifice my beliefs for a paycheck. If you cannot find a way to make money with your computer skills, then you deserve to be flipping burgers or making tacos.
If you cannot outsmart your superiors, then what do you even spend your time doing all day?
you are having a *moral* problem with throttling p2p traffic? Huh?
Oh sure, mod me troll, and yeah, it's cliche', but a business has to play statistics and look at trends. The overwhelming majority of people using p2p for *legit* things aren't using it for such things day in and day out; they're torrenting a fedora dvd, or something like that. That's fine, works, etc. But if you see someone with a constant stream day in and day out...
...that person, on a general level, you feel morally obligated to protect? Really?
There are plenty of valid uses for p2p. Certainly. Just assume that's not the people who your boss is after; it shouldn't be difficult to determine the difference.
Your details are a bit vague, but let's pretend "your pipe" is a single DS3 (45 megabits) out in the boonies somewhere and you are offering a mix of plans that average out to 7.8 megabits per customer (400 * 7.8 / 70 = 44.5).
Assuming you are in the US, 45 megabits of transit is unlikely to cost you more than ~$2k/month ($50/megabit transit is easy to come by, you can do way better if you shop and have access to many carriers), but due to the amazing power of phone company pricing, the DS3 to carry it could easily run $10k-40k/month depending on how far out of a major city you are. (Within a major city, DS3s are closer to $3k/month.) Let's use the low end of that range and call it $10000/mo for the DS3 and $2000/mo for the bandwidth, or $12000/mo total for 45 megabits or your total cost of ~$267/megabit.
If your customers were to demand no oversubscription (as most Slashdotters seem to), delivering a 10 meg cable connection would therefore cost you $2670/month to deliver to your customers. At standard retail markup (including maintaining the cable lines, buying routers, paying rent, paying salaries, etc) of ~2x, let's call it $5k/month per customer. This poses a problem, since no residential customer will pay $5k/month.
If you work it from the other angle, starting from what your customers will pay, let's pretend they are comfortable paying $80/month for their 10 meg cable connection. (This is high if they were in a city, but if this is their only option vs dialup, they'll buy it anyway.) Assuming you have some overhead and only half that can pay for bandwidth, you have $40/month for 10 megabits or $4/megabit.
How do you reconcile that your customers will only pay $4/megabit when your costs are $267/megabit? The magic of oversubscription.
These customers need to be willing to live with the idea that they are expected, on average, to use only 143Kbit/sec on their 10 meg pipe. If on average they want more than that, they have to be willing to pay for it, otherwise the ISP is just going to fold, and they can go back to dialup.
For some reason, Slashdotters see this as evil. Is it? How else can you make the numbers work? (Most of these numbers are ballpark since the posters details were so vague, but they real-ish.)
Aaron
The lesson of the day is not to base critical business applications on consumer bandwidth.
This is why nearly every ISP I have dealt with or worked with offered a free for all business package. Sure, they cost a bit more, but it's usually worth it.
"You should always go to other people's funerals; otherwise, they won't come to yours." -- Yogi Berra
I would consider myself a fairly large user of my internet. I have a 30MB/sec connection, and last month I used approximately 50GB. Calculating that out, I am using my connection 1/197th of the time. A 1:70 ratio sounds pretty decent unless you have an abnormally large number of bandwidth hogs running bittorrent 24/7.
Ah but you NEVER buy "6mbps" ... you always buy "up to 6mbps" or "maximum 6mbps", and then try to conveniently ignore the bits you don't like in the deal.
Please, cite me just one ISP who offers a "guaranteed 6mbps available 24/7", and I'll gladly admit I am wrong (right after I sign up with them).
I think you need to make sure the traffic you are seeing is actually P2P. I would highly doubt it given your subscriber to bandwidth ratio. The majority of "normal" long flow traffic is actually http. Mostly flash video or http downloads. That said, you have such a high ratio that it's possible its not even downloads hitting up against your cap. If you have as flat a usage pattern as you say you have, it likely already sucks to be your customer doing anything at all at peak times. People would do better on dial-up....at least it would be consistent and they wouldn't get stuck with nil at certain intervals.
Confirm you have a P2P problem before you start shaping. If you tell your boss the traffic is mostly http no amount of packetshaping is going to fix this problem to anyone's satisfaction(unless it actually is all http downloads).
Since you're on a tight budget already, I recommend running nTop on a box connected to a mirror or span port. That would be an easy way to determine what's actually going on.
When presented with the fact that shaping is pointless your boss will either buy more bandwidth or do nothing at all. Either way you aren't forced to shape. If he chooses the second option your customers should make him uncomfortable or fix the problem altogether by moving to dial-up.
--"It's Bradford Company, slash your last name, dot your first name"
If this is as described a small ISP with 400 customers whose bandwidth use is right at the limit most of the time, then throttling is already implemented. Automatically. By the ISPs upstream provider. So if customers would be unhappy because of throttling, then they are unhappy already. If there are contract problems because unlimited service was promised, then these contract problems are already there.
And as described, this is a small sideline of the companies business, so anything that will keep their lawyers busy, like contract changes, won't fly. Anything that is a major investment most likely won't fly. The only thing that could fly is anything that either makes money, or significantly improves the reputation of the company which could have other positive side effects.
Since Megabits are limited in this situation, his boss is absolutely right that the only thing he can do is to maximise the number of _happy_ customers. And that would be maximised by throttling the heavy users, giviing low bandwidth users fast access whenever they need it.
From the user's point of view: As a group, they pay 400x dollars per month to the ISP, who for that money gives them a total bandwidth with some limit. As a group, they don't want to include anyone who uses tons more than their fair 1/400th share.
Shape, not throttle. If done correctly shaping is what makes a difference between a good ISP and a great ISP. It is not a problem to detect P2P traffic and shape it to a lower priority, provided that you shape important traffic as high priority - ACK's, Skype voice, game traffic (WoW, CS, ...), first 100k of any HTTP or HTTPS connection, SSH, ...
As a power user it is not that critically important that my torrents only come at 16kb/s during the day if my web, games and IM apps are snappy, but I would like to have the torrents saturate the pipe during off-peak.
Also, hard caps are overrated - you don't pay per Gb, why should we? Just prioritise traffic correctly and everyone will be happy.
Unfortunately you don't have the control over the data at the right places to do what you need here. The bottleneck isn't getting from the ISP to the user's house (unlike in adequately-funded ISPs) - it's the feed that that the ISP is getting in from their upstream. You might still do something like run Weighted RED to harass the FTP and BitTorrent traffic, but it's not as effective there as if you got your upstream to prioritize what they're sending you.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
In this current economic environment are you really surprised they are asking you to throttle instead of paying for bigger pipes? It is not your moral duty to ensure that people get the best internet experience. You do what you have to do to enforce your company goals and standards within the situation you've been given and ensure that they don't step across your moral standard - ie. lying, cheating, murder, etc. (If that is your particular moral standard. I once knew of a man who killed his wife yet felt morally bound to OSS for some reason. Hmmmm...) To throttle or not to throttle has little to do with your own morality.
Without the email/craigslist/banking net users subsidizing our massive bandwidth usage, our costs would go up immensely. You think guaranteed-rate 6mb lines are cheap? I'd MUCH rather pay a pittance for my bandwidth, and get in line behind the non-geeks.
As an example: I recently installed a proxying squid/pfsense-based firewall for my parent's home. They have 5 people living there, and monthly traffic of about 8GB. My traffic for my desktop system alone for the last 2 weeks is 26GB. I'm extremely happy to have them and those like them subsidizing my costs so I get cheaper bandwidth.
Just another "DOJ fascist authoritarian totalitarian bootlicker" -- Zeio