Slashdot Mirror
Hunting the Mythical "Bandwidth Hog"
eldavojohn writes "Benoit Felten, an analyst in Paris, has heard enough of the elusive creature known as the bandwidth hog. Like its cousin the Boogie Man, the 'bandwidth hog' is a tale that ISPs tell their frightened users to keep them in check or to cut off whoever they want to cut off from service. And Felten's calling them out because he's certain that bandwidth hogs don't exist. What's actually happening is the ISPs are selecting the top 5% of users, by volume of bits that move on their wire, and revoking their service, even if they aren't negatively impacting other users. Which means that they are targeting 'heavy users' simply for being 'heavy users.' Felten has thrown down the gauntlet asking for a standardized data set from any telco that he can do statistical analysis on that will allow him to find any evidence of a single outlier ruining the experience for everyone else. Unlikely any telco will take him up on that offer but his point still stands." Felten's challenge is paired with a more technical look at how networks operate, which claims that TCP/IP by its design eliminates the possibility of hogging bandwidth. But Wes Felter corrects that mis-impression in a post to a network neutrality mailing list.
They are generally using UDP so the original assertion that degrading the other users experience should be true as UDP should break down long before TCP does. Though I do agree that if Comcast's system works as described it's probably the best solution for a network that can't implement QoS.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Because the operators pay for the bandwidth. The high bandwidth users are less profitable than the other ones.
They don't negatively impact operations in the sense of taking up a scarce resource that degrades other customers' performance. However, they do still use above-average amounts of bandwidth, which costs ISPs money. When offering a flat-rate, unlimited-use service, your economics come out ahead if you can find some way to skew your customers towards those who don't actually take advantage of your claimed "unlimited use".
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
I have personally witnessed hogging of bandwidth and, I'd wager, so have you. This term describes when an individual user uses more bandwidth resources than they were assumed to need.
Example: My brother moves in with two of his friends. His latency is horrible. When his roommate is not home, the internet is fine. When he's away at work it becomes unusable. He calls me to look at the situation, and we determine that one of his roomies is a heavy torrent user. Turns out the roommate was ramping up torrents of anime shows he wanted to watch while he was gone. He was aware of the impact to his own internet experience, and so ramped it back down when he wanted to use it himself.
If that's not hogging bandwidth, I'm not too sure what is.
If this doesn't scale, logically, up to the network at a whole, I'm not sure why.
Now, to be completely clear - I feel overselling bandwidth is wrong. I feel the proper response to issues like this on the larger network is guaranteed access to the full amount of bandwidth sold at all times. On the local scale, these men should have brought in another source of internet. On the larger scale, the telco should do the same.
Denying that the issue can happen, however, is stupid to the point of sabotage.
An end-user can download all his access line will sustain when the network is comparatively empty, but as soon as it fills up from other users' traffic, his own download (or upload) rate will diminish until it's no bigger than what anyone else gets.
So, if I understand this statement, if a user is hogging all the bandwidth until no one gets any connectivity - since it is all the same it is totally fair. One user can bottleneck the pipes, but since their stuff isn't fast any more either, we're all good?
How does an argument of this kind help anyone but a bandwidth hog?
Marge: We drove around until three in the morning looking for another open all-you-can-eat seafood restaurant.
Lionel Hutz: And when you couldn't find one?
Marge: [crying] We... went... fishing.
--
Lately I've had to deal with this problem. Our solution was rather simple. We use NTOP on an Ubuntu box at the internal switch. We replicate all the traffic coming into that switch to a port that the NTOP box listens on.
It may not be a perfect solution, but it can easily let us know who the top talkers are and give us a historical look at what they are doing.
From that report, we look for anyone uploading more than they download. We also look for people who upload/download a consistent amount every hour. If you see someone doing 80gb in traffic each day with 60gb uploaded, you probably have a file sharer. When you see the 24-hour reports for the user and see 2~3gb every hour on upload, you *know* you have a file sharer.
After that, it's as simple as going to the DNS server and locking their MAC address to an IP. Then, we drop all that traffic (access list extended is wonderful) to another Ubuntu box. That box has a web page explaining what we saw, why the user is banned, and the steps they need to take to get back online.
Most users are very apologetic. We help them to set up upload/download limits on their bittorrent client and then we put them back online.
I'd rather you do it wrong, than for me to have to do it at all.
During a Slashdotting, the problem is rarely network-related (aside from people who use a cheap host and have very low artificial bandwidth limitations, or are hosting their site on a low-end cable connection).
More often than not, the database goes down. MySQL is especially prone to just dying when put under any significant workload. That's why you'll often see error messages saying that the web front end can't connect to the database. You can still get to the site because the network can handle the volume of traffic just fine, and you can get the error message because the web server can also handle the volume of traffic just fine.
The next most common problem is the server-side web apps being unable to handle the load. I don't mean the web servers, as most of those can handle huge amounts of traffic, even on ancient hardware. I mean web apps implemented using PHP, Ruby on Rails, ASP.NET, JSP and so on. Many sites don't use PHP bytecode caching, for instance, nor do they do much data caching. So it just ends up taking too long to generate pages, and your browser times out.
I guess it's cheaper to sacrifice 5% of revenue than to have to undertake a network upgrade.
This mentality is part of why the U.S. lags so much in broadband.
I also go through my client list and drop those that consume more of my time and resources in favour of the easier clients who ultimately improve my business at a lesser cost. What's wrong with that? My company, my rules. "We reserve the right to refuse service to anyone" -- it's in every restaurant. Why would you expect a business to serve you? Why would you consider it a right?
I'm not saying he's wrong... quite possibly he's right, but seriously - how does someone's blog entry that doesn't provide one single data point to back up the claim make it to the front page?
The important thing that he's doing is trying to shift the burden of proof back onto the ISPs and telcos. They just declared that some people are bandwidth hogs and terminated their connection. They didn't give the public any proof that they were ruining the internet experience for anyone else ... nor did anyone come forward after the purge and say, "Gee, my internet sure is fast now that the bandwidth hogs are disconnected!"
So he calls for proof since he hasn't seen any. He has to say that there are no bandwidth hogs in order to get a response from the telcos. Saying someone might be wrong is not the same impact as calling someone a liar. Yes, he's basing this on an assumption but it's just the same that everyone assumed there were individuals out there ruining the experience. All of us just let the telcos terminate the service of whoever they wanted to and then we moved on with our lives.
I welcome his opposing viewpoint and challenge to "because we said so." They can release anonymous usage data without harming anyone so why not open it up to a request?
My work here is dung.
"We aren't getting the advertised bandwidth! Waaah!"
Yes, actually, false advertising is a problem. If an ISP tells me I can make unlimited use of my 10Mbps connection, I expect to be able to make unlimited use of it -- including sustaining 10Mbps or something reasonably close all day and all night. If such a level of service is impossible for an ISP to provide and remain profitable, why the hell are they advertising these plans?
If they are lying to consumers about the level of service they can provide, they should cover themselves by increasing the network capacity, or they should admit they lied, reduce the bandwidth they provide to users, and hope that nobody sues them over it. Kicking people off the network for trying to use what they paid for is not an appropriate response to overselling, and if the FCC had any spine they would kill the practice before it gets out of hand.
Palm trees and 8
First of all, I am, and always will be, a bandwidth hog. Why? Because I'm better at using the internet than everyone around me. That means I find more things, and bigger things, to download. If they someone banned P2P, I'd still have more streamed video than anyone I know. If they banned that, too, I'd still download more images. If they banned that, i'd still have more web traffic, email, IM, etc etc etc. I will always be a 'hog' in any environment. I was even told that I was "#1 abuser" of the "Unlimited" service when I was on dial-up in a small town and they tried to charge me an extra $300 that month. As someone else had just come into town, I switched, obviously.
I don't pay for the top tier of residential service to just let it sit idle. I'm going to -use- it.
I have absolutely no sympathy for people that sell me something and then get upset when I actually use it within the original limitations. I have only a small amount of sympathy for people that sell me something and I use it beyond their arbitrary limitations, even if I agreed to them.
Why?
America has -crap- for internet compared to other developed countries. We are quickly falling behind the rest of the world in terms of internet bandwidth. This is purely from greed and laziness on the part of the ISP. They refuse to upgrade and try to prevent competition at the same time. Sprint even has the nerve to advertise Pure and claim that it's faster than Cable internet, despite being 1/10th of the speed.
"If you make people think they're thinking, they'll love you; But if you really make them think, they'll hate you." - DM
One problem is by default many network devices/OSes do bandwidth distribution on a per _connection_ basis not on a per IP basis. So if there are only two users and one user has 1000 active connections and the other has just one active connection the first user will get about 1000 times more bandwidth than the second user.
;).
;) ), BUT even when they "log out" they _still_ get always-on internet access except it's just on a lower priority (but NO byte quota!). A customer might be restricted to say 10GBs at "priority" a month.
P2P clients typically have very very many connections open. Wheres other clients might not.
A much fairer way would be to share bandwidth amongst users on a per IP basis. That means if two users are active they get about 50% each, even if one user has 100 P2P connections and the other user has only one measly http connection.
Then within each customer's "per IP" queue, to improve the customer's experience you could prioritize latency or loss sensitive stuff like like dns, tcp acks, typical game connections, ssh, telnet and so on, over all the other less sensitive/important stuff.
Of course if you have oversubscribed too much, you will have way too many active users for your available bandwidth. A fair distribution of "not enough" will still be not enough.
If you have two people and you give each a fair slice of one banana, they each get half a banana. Maybe both are satisfied.
If you have 1000 people and you give each a fair slice of one banana, they each get 1/1000th of a banana. Not many are going to be satisfied
And that's where we come to the other problem.
The problem with P2P is many customers will often leave their P2P clients on 24/7, EVEN when some of them don't really care very much about how fast it goes (some do, but some don't). To revisit the banana analogy, what you have here is 1000 people, and 1000 of them ask for a slice of the banana, EVEN though some of them don't really care - they'll only really feel like having a slice next week, when they're back from their holiday!
So how do you figure out who cares and who doesn't care?
Right now what many ISPs do is have quota limits - they limit how much data can be transferred in total. When the quota runs out "stuff happens" (connections go slow, users get charged more etc). So the users have to manage it.
BUT this is PRIMITIVE, because if you can figure out when a user doesn't care about the speed etc, technology allows you to easily prioritize other traffic over that user's "who cares" traffic.
So what's a better way of figuring it out?
My proposal is to give the customers a "dialer" which allows users to "log on" to "priority Internet" (and then only something starts counting the bytes
The advantage of this method is:
1) There is no WASTED capacity - almost all the available bandwidth can be used all the time, without affecting the people who NEED "priority" internet access (and still have unused quota).
2) It allows a ISP to better figure out how much capacity to actually buy.
3) If there is insufficient capacity for "priority Internet" the ISP can actually inform the user and not put the user on "priority" (where the quota is counted). While the user might not be that happy, this is much fairer, than getting crappy access while having your quota still being deducted.
Perhaps this system is not needed and will never be needed in countries that don't seem to have big problems offering 100Mbps internet access to lots of people.
But it might still be useful in countries where the internet access and telcos are poorly regulated/managed. For example - you run a small ISP in one of those crappy countries and so you pay more for bandwidth from your providers- this system could allow you to make better use of your bandwidth and to be a more efficient competitor. And maybe even give your customers better internet service at the same time.
Yes the ISP could always buy enough bandwidth so that _everyone_ can get the offered amount even though not everyone really cares all the time (believe me this is true). But that could mean the ISP's internet access packages being much more expensive than they could be.