Slashdot Mirror
Morality of Throttling a Local ISP?
An anonymous reader writes "I work for a small (400 customers) local cable ISP. For the company, the ISP is only a small side business, so my whole line of expertise lies in other areas, but since I know the most about Linux and networking I've been stuck into the role of part-time sysadmin. In examining our backbone and customer base I've found out that we are oversubscribed around 70:1 between our customers' bandwidth and our pipe. I've gone to the boss and showed him the bandwidth graphs of us sitting up against the limit for the better part of the day, and instead of purchasing more bandwidth, he has asked me to start implementing traffic shaping and packet inspection against P2P users and other types of large downloaders. Because this is in a certain limited market, the customers really only have the choice between my ISP and dial-up. I'm struggling with the desire to give the customers I'm administering the best experience, and the desire to do what my boss wants. In my situation, what would you do?"
This is not a hard problem. You can not maintain a reasonable oversell ratio unless you have low average usage. Yes, one way to get that is throttling, but it's difficult to do that in an effective way that won't piss off your customers.
What you should do is tell them they get 40G/mo or whatever, plus a usage fee above that, and let the customers throttle themselves if they want to. If you want to be a nice guy about it, you could give them the option of being auto-throttled or suspended if they approach the limit, so they don't get an unexpected bill. Of course whatever you do, you'll need to revise your terms of service.
Voila, you maintain low pricing and good performance for everyone, because the p2p guys will police themselves now. If you have customers that routinely transmit hundreds of GB because they're a professional video editor or something, then they won't mind paying for the bandwidth.
Here's the thing - you have no choice. Do the shaping.
That said - form a compelling argument for doing the right thing, and present that to your boss. Don't defy him, but give him a reason to reconsider. In the meantime, do as you're told. You can always undo shaping. Don't screw your employment in the interim.
Karma: Chameleon (mostly due to the fact that you come and go).
start my own ISP, reselling third party bandwidth. If the market is that limited and poorly serviced, there is money to be made by providing a decent service. You will be happier and as the owner you also stand to make more money.
Petition for your boss to do the right thing.
While you're petitioning, do what your boss tells you.
If what your boss tells you to do is unethical, quit, and tell him why in your resignation letter.
Check out my sysadmin blog!
I had a situation once where my bandwidth was metering during regular hours but free from midnight - 7am. Any smart heavy user will set up their downloads to happen during the free period and take the load off the network during peak hours. I've never understood why more ISPs don't do that.
If you just tell people they have a 40G cap then they'll feel entitled to use it whenever they want, and you really can't argue with that.
Is throttling really cheaper?
Have you tried to compare the price of just buying more bandwidth with what it will cost you to setup and maintain the packed shaping?
The P2P boys will quickly figure out what is going on and they can set their clients to download from Midnight to 8am. That way, there's plenty of bandwidth when Joe Average wants to check their Facebook and when businesses are operating and the bandwidth through the night which is mostly unused is utilised better. Everyone wins.
I only please one person per day. Today is not your day. Tomorrow isn't looking good either. - Scott Adams
For a 400 user ISP, there is presumably only a dozen or so high traffic users...
Privately, encourage them to shift some of their activity to off times, such as late morning and middle of the night - explain to them it will help other users, plus help them too in they'll get better speed while helping to keep prices low.
If not enough voluntary compliance, then try enabling aggressive throttling / shaping during day / evening, but allow unthrottled speed during off-hours for high traffic users.
Presuming the ISP has access to multiple providers, then another option to consider is evaluating how much the ISP is paying for bandwidth - see if there are better options and/or if contracts can be renegotiated.
Ron
You don't want to punish customers for how much they download so much as when they download.
The guy who downloads 100Gb overnight when no one else is online? He isn't a problem.
The 100 users who all connect and download from together at peak hour? They are the problem.
So you want to allow people who don't use the net when everyone else is using it full-speed access. And you want those who use the net at peak hour to be slowed down.
The way to acheive this?
Do nothing and let congestion shape them.
Morality is a tool for the herd to feel more important than their leaders. Instead, get pragmatic: how can you make this business work for most people?
You probably want heavy downloaders to use another service, anyway. You might even consider setting up two plans, one for ueber-users and one for normal users.
However, I would prioritize traffic. Email, web, SSH, et al come first; after that, all p2p protocols in order of usefulness.
You need to define your business audience. If it's people who are going to check the mail and web surf, and 5% of your customers are p2p users, cut out the p2p users and focus on the people you want to serve.
Futurist Traditionalism
Im wondering what you have for backbone that you are 70:1 oversubscribed. If you deploy 768/256 connections with 400 customers sounds like a whopping 3 T1 lines (~4.5Mb/s). if you do a more standard 1.5MB thats 6 T1 lines(~9Mb/s).
Maybe you should look at your upstream provider and see if you can get a fractional T3 to replace the T1s if my math is anywhere near correct. You will likely have a longer contract to sign but you may be able to pull in 10Mb/s for less than you currently pay. Then you could try to match the current expense.
There are other ways to trim back your backbone usage. Consider a cluster of transparent proxy servers. You can get pretty aggressive with the cacheing mechanise in squid and you can easily balance the cluster with DNS and not have to worry about session awareness as clients also cache DNS temorarily so each client will use the same proxy for their browsing session.
Certainly some sort of QoS will work for you and lessen the need to directly throttle.
If you just throw some proxying in there and give http and https higher priority and do some packet inspection to sniff out the P2P traffic and drop it down a level you will put off the inevitable need to grow your bandwidth for a while.
if my math is correct on 1.5Mb/s cable, you look like you have a per users upstream cost of just $7.50 each. That is pretty low. Too low.
...blackjack and hookers would also be involved somehow?
Mit der Dummheit kämpfen Götter selbst vergebens
400 divided by 70 = 5.71.
I have no problem with you scheduling low-latency traffic over filesharing traffic, filtering, or whatever, but it seems a little short-sighted that it only takes 5.71 users to completely muck up your network. (I.E if you sell 1mbit connections, you could "theoretically" support 420 customers on a 6mibt pipe (6*70=420 at a 70:1 oversell ratio).
Is it sad that I am more likely to recognize you and your posts by your sig than your name or UID?
I think P2P is servers used this way are a great tool helping ISP's reduce their upstream bandwidth costs. My ISP does it and, for example, has mirrors of Fedora and Ubuntu update repositories plus a whole library of popular downloads that I don't get charged for if I use their servers to download (and it's faster too). Furthermore their servers will download files via P2P and make that available to all their other users.
My ism, it's full of beliefs.
There is no morality for throttling. It's done for either technical or business reasons.
What?
Or to get more BW.
By your description, you are The Man when it comes to this, he won't fire you, he is forced for more BW. He can't replace you because you refuse to teach your follower if it goes that route, and in effort he would loose the ISP business.
What stuns me, people are ALL UP FOR THROTTLING! Give me a break! Everyone here recommending it is either shooting themselves on their legs due to sheer ignorance or working for a anti-net neutrality party.
To really start saving BW, think about caching, you can rather easily implement transparect proxy using squid and simple routing rules, and your customers won't notice a thing even if WWW traffic is cached. On that size it sums up to quite considerable amount of data.
You can consider other caching methods too, but you can also implement QOS, prioritize SSH and WWW, and immediate increase in service quality achieved, given you use powerfull enough routers.
Any kind of throttling beyond mere QOS is plain and simply EVIL.
Pulsed Media Seedboxes
What would I do? I'd start by doing what the boss says. This is a really bad time to have to look for employment elsewhere. If you don't do what the boss says, customers of your former employer are not going to start sending you money to live on because you did the "right" thing but lost your job.
Then after things have been at least temporarily taken care of, research better alternatives and present them to your boss.
I want a new quote. One that won't spill. One that don't cost too much. Or come in a pill.
He has no choice but to honor the contract they've made with customers.
If, as most cable companies do, they've contracted to provide "unlimited" service, at "xx Mbps rate", then that's what they need to provide.
If such is the case, then throttling anyone is fraud.
"National Security is the chief cause of national insecurity." - Celine's First Law
The answer to this, and many such sticky situations in IT, is to update your resume` and leave town.
The way I see it, you're screwed if you throttle, and you're screwed if you don't throttle. Some of the solutions given sound good and well on paper. But then again, so does communism.
The game.
Check the contract your customers sign. there's usually (if the lawyer who wrote it up was worth his salt) would have a clause in the contract stating "The ISP can change he terms of the contract with 30 days notice." or words to that effect. All the OP needs to do is set up a mail shot to all subscribers telling them of the changes to the contract will come into force in 30 days and wait..... Then dump all the complaints on the boss's desk. The reality of him loosing about 10%-20% (pulled out of the air guestamate) of the customers might make him rethink and that's when you suggest a few alternatives (Just make sure you do a lot of fact finding and homework on the issues before you talk to the boss).
Laters Sol "Have you found the secrets of the universe? Asked Zebade "I'm sure I left them here somewhere"
I've never seen a cable-ISP contract that provided service at a specified rate in Mbps. You can get those contracts as a business user, but they're not the standard ones home users have. Usually home contracts say something along the lines of "up to xx Mbps; actual speeds may vary and are not guaranteed".
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
That's not the issue. The issue is whether there is any legal P2P traffic. geekboy642 proved there was, and you didn't offer anything to refute it, so I guess that you agree with him.
Since you agree that there *is* legal P2P traffic, the argument that "it's illegal so there is no problem throttling it" is a non-sequitur.
Easy solution... I did something like this a long time ago.
We used to split our upstream into "Priority" and "Non-priority" and all users went into "Non-Priority"
When we gave them a real-time "price" meter... It had a button and a small display that showed how much your bill was for the month.
Use the service at non-priority and the $$$ ticked over slowly.
But hit the "Turbo" button, it added your IPs to the priority stream and the $$$ scream over and you get a big speed boost. Great for businesses who used it.
We only ever tried it in beta while we had significant oversubscription due to limited availability of bandwidth at the time, but we noticed a few strange effects.
First, people just liked pressing the button. They would go on, off, on, off while waiting for anything.
Second, it was instant gratification - you hit th e button and your download speed goes straight up... Very effective and you know it's going faster because the $$$ tick over faster.
Thirdly, the level of satisfaction was directly influenced by the speed the $$$ ticked over... We accidently released a buggy version under Beta where the $$$ ticked over at ten times the rate.
It turned out to be the most popular and people started requesting it after we fixed the bug in the subsequent version... Seems that if they got charged more, the mental connection was that it was faster.
Anyway, then bandwidth prices came down and we just got more bandwidth, and all the beta testers moaned when we turned off their turbo buttons...
We weren't actually charging the beta testers for the button at the time, but they were all willing to pay for the service, because they loved being able to see at all times (through a small widget-like interface) exactly what they were spending.
GrpA
Enjoy science fiction? "Turing Evolved" - AI, Mecha, Androids and rail-gun battles. What more could you want?
This is a prime example of why the telecommunications, medical, banking and the power industries just need to be nationalized. These people or not going to be told what is going on, there are going to be no changes to there terms of services and more than likely this guy is going to be fired. In the end they will throttle the entire network, put hard caps in, and close the accounts of people who make a fuss. And probably turn them in to the RIAA/MPAA.
Linux modi 2.6.26-2-parisc
1.) If the action your boss is requesting is illegal, inform the proper authorities.
2.) Abuse is a matter of perspective: Is your boss asking you to shape everyone to 64k as a maximum? Our legal system is fucked up. A line could be sold with a 7 Mb maximum, 768 kb average. Said line would only become abuse when it avaergaes 7kb on purpose.
3.) The only reqard for having morals must come from within yourself. If you are looking for anyone to recognize that you have morals, be prepared to be disappointed
Ultimately, you are in charge of your own decision. I happen to agree here with many who say to convince your boss for the alternate solution first.
Unless the request specifically borders on Fraud, your morals are safe and sound.
I'd do what the boss says. He signs your paycheck. When traffic shaping STILL doesn't work, show him that you're still hitting the pipeline cap, and suggest more bandwidth again.
you are having a *moral* problem with throttling p2p traffic? Huh?
Oh sure, mod me troll, and yeah, it's cliche', but a business has to play statistics and look at trends. The overwhelming majority of people using p2p for *legit* things aren't using it for such things day in and day out; they're torrenting a fedora dvd, or something like that. That's fine, works, etc. But if you see someone with a constant stream day in and day out...
...that person, on a general level, you feel morally obligated to protect? Really?
There are plenty of valid uses for p2p. Certainly. Just assume that's not the people who your boss is after; it shouldn't be difficult to determine the difference.
In my situation, what would you do?
First, at 70:1 oversubscription there is no bandwidth shaping policy which will improve the user experience, so you'll piss off the top 10% of your users without making the other 90% any happier.
I'd explain to the boss that the accepted norm for residential oversubscription is 10:1 and that oversubscription rates in excess of 20:1 flat out don't work. You either need to increase your system bandwidth reduce your subscriber bandwidth. In other words, you either buy more T1s at the head end or you drop those 5 meg lines to 768kbps and be honest about it.
Next, implement traffic shaping for ports other than UDP 53, TCP 22, 25, 80 and 443 during the prime time hours on your graph. You'll piss off the torrent freaks in the top 10%, but oh well.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
In my opinion, the best solution is to strongly throttle large bandwidth usages (P2P, FTP and NNTP streams, etc) during the periods of near-capacity, and automatically relax the filtering during off hours.
That's one way... Here's another:
Instead of trying to choose which protocols are heaviest usage, traffic shape people based on what the actual criteria that you care about is: Too much overall usage over long periods.
In Linux terms, set up a HTB with a queue for every customer. Set the base rate to whatever your backbone speed is (1/70th of the customer's line rate), the ceil rate to their line rate, and give them a nice big bucket - say, 120 seconds times their line rate.
Then, people who are normal users - web surfing, downloading an occasional email attachment, etc - will go full bore, any time they want it. People who are bittorrenting will go full speed for a couple minutes, and then decrease down to whatever bandwidth is available. At night, if there's a lot of backbone free, it'll go fast. At 7 PM, they get best effort on whatever is available.
This is a very simplified example. You could additionally shape them so that their web and email will take priority over bittorrent when they're at the bottom of their token bucket, or other fine tuning...
The basic message I'd like to get across is: you don't have to shape based on protocol, because you care about the usage, not the protocol. Just shape based on usage, and let them work out which protocols they want to use.
Your details are a bit vague, but let's pretend "your pipe" is a single DS3 (45 megabits) out in the boonies somewhere and you are offering a mix of plans that average out to 7.8 megabits per customer (400 * 7.8 / 70 = 44.5).
Assuming you are in the US, 45 megabits of transit is unlikely to cost you more than ~$2k/month ($50/megabit transit is easy to come by, you can do way better if you shop and have access to many carriers), but due to the amazing power of phone company pricing, the DS3 to carry it could easily run $10k-40k/month depending on how far out of a major city you are. (Within a major city, DS3s are closer to $3k/month.) Let's use the low end of that range and call it $10000/mo for the DS3 and $2000/mo for the bandwidth, or $12000/mo total for 45 megabits or your total cost of ~$267/megabit.
If your customers were to demand no oversubscription (as most Slashdotters seem to), delivering a 10 meg cable connection would therefore cost you $2670/month to deliver to your customers. At standard retail markup (including maintaining the cable lines, buying routers, paying rent, paying salaries, etc) of ~2x, let's call it $5k/month per customer. This poses a problem, since no residential customer will pay $5k/month.
If you work it from the other angle, starting from what your customers will pay, let's pretend they are comfortable paying $80/month for their 10 meg cable connection. (This is high if they were in a city, but if this is their only option vs dialup, they'll buy it anyway.) Assuming you have some overhead and only half that can pay for bandwidth, you have $40/month for 10 megabits or $4/megabit.
How do you reconcile that your customers will only pay $4/megabit when your costs are $267/megabit? The magic of oversubscription.
These customers need to be willing to live with the idea that they are expected, on average, to use only 143Kbit/sec on their 10 meg pipe. If on average they want more than that, they have to be willing to pay for it, otherwise the ISP is just going to fold, and they can go back to dialup.
For some reason, Slashdotters see this as evil. Is it? How else can you make the numbers work? (Most of these numbers are ballpark since the posters details were so vague, but they real-ish.)
Aaron
Morality is a tool for the herd to feel more important than their leaders.
Who would have guessed that Dick Cheney was posting on slashdot?
When information is power, privacy is freedom.
IP packets let the sender specify which ones are important, via the QoS info. If I'm sending real-time game traffic and a big giant file, I want you to give priority to the game.
Ideally you both respect my QoS info and let me override that via a nice web admin interface that lets me specify ports that are important to me.
All of this is subject to my per-user throttling of course. You use it to select which of my packets get dropped first, not the number of my packets that get dropped.
Ah but you NEVER buy "6mbps" ... you always buy "up to 6mbps" or "maximum 6mbps", and then try to conveniently ignore the bits you don't like in the deal.
Please, cite me just one ISP who offers a "guaranteed 6mbps available 24/7", and I'll gladly admit I am wrong (right after I sign up with them).
I think you need to make sure the traffic you are seeing is actually P2P. I would highly doubt it given your subscriber to bandwidth ratio. The majority of "normal" long flow traffic is actually http. Mostly flash video or http downloads. That said, you have such a high ratio that it's possible its not even downloads hitting up against your cap. If you have as flat a usage pattern as you say you have, it likely already sucks to be your customer doing anything at all at peak times. People would do better on dial-up....at least it would be consistent and they wouldn't get stuck with nil at certain intervals.
Confirm you have a P2P problem before you start shaping. If you tell your boss the traffic is mostly http no amount of packetshaping is going to fix this problem to anyone's satisfaction(unless it actually is all http downloads).
Since you're on a tight budget already, I recommend running nTop on a box connected to a mirror or span port. That would be an easy way to determine what's actually going on.
When presented with the fact that shaping is pointless your boss will either buy more bandwidth or do nothing at all. Either way you aren't forced to shape. If he chooses the second option your customers should make him uncomfortable or fix the problem altogether by moving to dial-up.
--"It's Bradford Company, slash your last name, dot your first name"
A contention ratio of 70-1 is really high. What exactly are you selling your customers?
Most ISPs around my part run on an contention ratio of between 20-1 and 50-1. In practise it sits closer to the 20-1 than the 50-1. At 70-1 I'm not surprised that the pipe's constantly full: it's twice what it should be.
Unless, of course, you're selling an 'lite' package. But as you've got an monopoly, it sounds like you're probably selling an 'lite' package at 'premium' prices.
If this is as described a small ISP with 400 customers whose bandwidth use is right at the limit most of the time, then throttling is already implemented. Automatically. By the ISPs upstream provider. So if customers would be unhappy because of throttling, then they are unhappy already. If there are contract problems because unlimited service was promised, then these contract problems are already there.
And as described, this is a small sideline of the companies business, so anything that will keep their lawyers busy, like contract changes, won't fly. Anything that is a major investment most likely won't fly. The only thing that could fly is anything that either makes money, or significantly improves the reputation of the company which could have other positive side effects.
Since Megabits are limited in this situation, his boss is absolutely right that the only thing he can do is to maximise the number of _happy_ customers. And that would be maximised by throttling the heavy users, giviing low bandwidth users fast access whenever they need it.
From the user's point of view: As a group, they pay 400x dollars per month to the ISP, who for that money gives them a total bandwidth with some limit. As a group, they don't want to include anyone who uses tons more than their fair 1/400th share.
Shape, not throttle. If done correctly shaping is what makes a difference between a good ISP and a great ISP. It is not a problem to detect P2P traffic and shape it to a lower priority, provided that you shape important traffic as high priority - ACK's, Skype voice, game traffic (WoW, CS, ...), first 100k of any HTTP or HTTPS connection, SSH, ...
As a power user it is not that critically important that my torrents only come at 16kb/s during the day if my web, games and IM apps are snappy, but I would like to have the torrents saturate the pipe during off-peak.
Also, hard caps are overrated - you don't pay per Gb, why should we? Just prioritise traffic correctly and everyone will be happy.
Have you considered any other means of reducing network load? For example, Squid? A significant portion of your traffic is likely your users visiting the same content-rich websites, like MySpace, Facebook, Youtube, etc. If you can locally cache this content (especially the Flash stuff) you'll probably see a large drop in load.
Unfortunately you don't have the control over the data at the right places to do what you need here. The bottleneck isn't getting from the ISP to the user's house (unlike in adequately-funded ISPs) - it's the feed that that the ISP is getting in from their upstream. You might still do something like run Weighted RED to harass the FTP and BitTorrent traffic, but it's not as effective there as if you got your upstream to prioritize what they're sending you.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
In this current economic environment are you really surprised they are asking you to throttle instead of paying for bigger pipes? It is not your moral duty to ensure that people get the best internet experience. You do what you have to do to enforce your company goals and standards within the situation you've been given and ensure that they don't step across your moral standard - ie. lying, cheating, murder, etc. (If that is your particular moral standard. I once knew of a man who killed his wife yet felt morally bound to OSS for some reason. Hmmmm...) To throttle or not to throttle has little to do with your own morality.
Without the email/craigslist/banking net users subsidizing our massive bandwidth usage, our costs would go up immensely. You think guaranteed-rate 6mb lines are cheap? I'd MUCH rather pay a pittance for my bandwidth, and get in line behind the non-geeks.
As an example: I recently installed a proxying squid/pfsense-based firewall for my parent's home. They have 5 people living there, and monthly traffic of about 8GB. My traffic for my desktop system alone for the last 2 weeks is 26GB. I'm extremely happy to have them and those like them subsidizing my costs so I get cheaper bandwidth.
Just another "DOJ fascist authoritarian totalitarian bootlicker" -- Zeio
I an the senior network tech for a small ISP. We manage 10 sites. Some with as many as 2000 customers, some with as few as 40. All it takes is one abusive user to ruin the internet experience for all people in a site. We also face the problem of satellite delivery for our network content since most of our locations are unaccessible via terrestrial means. So bandwidth is not only limited, but very, very expensive. We also limit our DSL and cable modem services down to 256k because of the cost of delivery. We have implemented Packeteer Packetshapers and have filtered out all P2P traffic except bittorrent. And we have torrent traffic limited to a max rate of 10% of the pipe to an area. This is especially important to satellite as most p2p software streams without regard to satellite latency and bandwidth constraints and floods the link causing service outages for our sites. We have only had a few complaints over the years. And those folks we refer back to our ToS as we lay out the p2p restrictions in there. We have had to take the approach to penalize the few for the sake of the many. We would rather have one or two pissed off customers then have 1000.. We also utilize monitoring software to track overall bandwidth utilization of each client to find abusive users (users that peg their bandwidth 100% of the time) and penalize them if it is causing detrimental service to our other customers. We have learned over the years that you can never had enough bandwidth. The more you provide, the more the users utilize. And you will always have a few that push the envelope.
You probably would not see this post as it is hugging the bottom of a long pile of messages, but here are my two cents:
In small scale networks, as few as five to ten over utilizing customers can bring the whole structure to its knees. From ethical perspective, it is your duty to keep network as operational as possible for the whole customer base. So that it is OK in my book to shape traffic as long as you keep it as fair as possible for your customers' benefit. Also it is important to back your traffic shaping with a solid mathematical model, as some (usually below 1%) of your customers can complain, and even can claim that you are stealing their capacity...
FAP (Fair Access Policy) is a rolling average, leaky bucket traffic shaping algorithm. We are using HNS (Hughes Network Systems) implementation with great success for five years. As you are a cable operator HNS solution would not work for you, however it is well documented (by public, in public domain. HNS' own documentation sucks). If you ignore customer complaints about HNS services in USA (problem there is not FAP mechanism, but very tight parameters set by HNS operations team) and concentrate on the system you would learn a great deal about traffic shaping that is adapted to real life conditions.
As you would need an implementation to use, a single layer FAP (HNS implementation permits three layers) can be put in place by using basic traffic shaping parameters in Cisco. For multi layered approach, you can use a Linux firewall. If you have money to spend on this, Allot traffic shapers are very good Linux based devices.
Regards
What this means is that his 400 user ISP only has the capacity for less than SIX users to be running full tilt at a time! To me that's a bit extreme. I can see 5:1 or even 10:1 but 70:1??!!