Slashdot Mirror
Hunting the Mythical "Bandwidth Hog"
eldavojohn writes "Benoit Felten, an analyst in Paris, has heard enough of the elusive creature known as the bandwidth hog. Like its cousin the Boogie Man, the 'bandwidth hog' is a tale that ISPs tell their frightened users to keep them in check or to cut off whoever they want to cut off from service. And Felten's calling them out because he's certain that bandwidth hogs don't exist. What's actually happening is the ISPs are selecting the top 5% of users, by volume of bits that move on their wire, and revoking their service, even if they aren't negatively impacting other users. Which means that they are targeting 'heavy users' simply for being 'heavy users.' Felten has thrown down the gauntlet asking for a standardized data set from any telco that he can do statistical analysis on that will allow him to find any evidence of a single outlier ruining the experience for everyone else. Unlikely any telco will take him up on that offer but his point still stands." Felten's challenge is paired with a more technical look at how networks operate, which claims that TCP/IP by its design eliminates the possibility of hogging bandwidth. But Wes Felter corrects that mis-impression in a post to a network neutrality mailing list.
They are generally using UDP so the original assertion that degrading the other users experience should be true as UDP should break down long before TCP does. Though I do agree that if Comcast's system works as described it's probably the best solution for a network that can't implement QoS.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Why would any business cancel paying customers that don't negatively impact operations?
All ideas^H^H^H^H^Hprocesses in this post are Patent Pending. (as well as the process of patenting all postings)
I have personally witnessed hogging of bandwidth and, I'd wager, so have you. This term describes when an individual user uses more bandwidth resources than they were assumed to need.
Example: My brother moves in with two of his friends. His latency is horrible. When his roommate is not home, the internet is fine. When he's away at work it becomes unusable. He calls me to look at the situation, and we determine that one of his roomies is a heavy torrent user. Turns out the roommate was ramping up torrents of anime shows he wanted to watch while he was gone. He was aware of the impact to his own internet experience, and so ramped it back down when he wanted to use it himself.
If that's not hogging bandwidth, I'm not too sure what is.
If this doesn't scale, logically, up to the network at a whole, I'm not sure why.
Now, to be completely clear - I feel overselling bandwidth is wrong. I feel the proper response to issues like this on the larger network is guaranteed access to the full amount of bandwidth sold at all times. On the local scale, these men should have brought in another source of internet. On the larger scale, the telco should do the same.
Denying that the issue can happen, however, is stupid to the point of sabotage.
An end-user can download all his access line will sustain when the network is comparatively empty, but as soon as it fills up from other users' traffic, his own download (or upload) rate will diminish until it's no bigger than what anyone else gets.
So, if I understand this statement, if a user is hogging all the bandwidth until no one gets any connectivity - since it is all the same it is totally fair. One user can bottleneck the pipes, but since their stuff isn't fast any more either, we're all good?
How does an argument of this kind help anyone but a bandwidth hog?
The fact is that what most telcos call hogs are simply people who overall and on average download more than others. Blaming them for network congestion is actually an admission that telcos are uncomfortable with the 'all you can eat' broadband schemes that they themselves introduced on the market to get people to subscribe. In other words, the marketing push to get people to subscribe to broadband worked, but now the telcos see a missed opportunity at price discrimination...
It's nice of him to declare that without evidence. Now I know it to be true.
I'm not saying he's wrong... quite possibly he's right, but seriously - how does someone's blog entry that doesn't provide one single data point to back up the claim make it to the front page?
Marge: We drove around until three in the morning looking for another open all-you-can-eat seafood restaurant.
Lionel Hutz: And when you couldn't find one?
Marge: [crying] We... went... fishing.
--
Lately I've had to deal with this problem. Our solution was rather simple. We use NTOP on an Ubuntu box at the internal switch. We replicate all the traffic coming into that switch to a port that the NTOP box listens on.
It may not be a perfect solution, but it can easily let us know who the top talkers are and give us a historical look at what they are doing.
From that report, we look for anyone uploading more than they download. We also look for people who upload/download a consistent amount every hour. If you see someone doing 80gb in traffic each day with 60gb uploaded, you probably have a file sharer. When you see the 24-hour reports for the user and see 2~3gb every hour on upload, you *know* you have a file sharer.
After that, it's as simple as going to the DNS server and locking their MAC address to an IP. Then, we drop all that traffic (access list extended is wonderful) to another Ubuntu box. That box has a web page explaining what we saw, why the user is banned, and the steps they need to take to get back online.
Most users are very apologetic. We help them to set up upload/download limits on their bittorrent client and then we put them back online.
I'd rather you do it wrong, than for me to have to do it at all.
During a Slashdotting, the problem is rarely network-related (aside from people who use a cheap host and have very low artificial bandwidth limitations, or are hosting their site on a low-end cable connection).
More often than not, the database goes down. MySQL is especially prone to just dying when put under any significant workload. That's why you'll often see error messages saying that the web front end can't connect to the database. You can still get to the site because the network can handle the volume of traffic just fine, and you can get the error message because the web server can also handle the volume of traffic just fine.
The next most common problem is the server-side web apps being unable to handle the load. I don't mean the web servers, as most of those can handle huge amounts of traffic, even on ancient hardware. I mean web apps implemented using PHP, Ruby on Rails, ASP.NET, JSP and so on. Many sites don't use PHP bytecode caching, for instance, nor do they do much data caching. So it just ends up taking too long to generate pages, and your browser times out.
You think you're being sarcastic, but has anyone ever seen a network go down in flames due to slashdotting, or has it always been the server?
If I have been able to see further than others, it is because I bought a pair of binoculars.
Then they will introduce the "friends and family" of ISP, some downloads and some sites will be "unmetered", and the sources will be the friends and family of the ISP. You know? the "partners" who provide "new and exciting" products and content to their "valued customers". Net neutrality will go down the tubes. ha ha.
Google needs the net to be open and neutral for it to freely access and index content. When the dot com bubble burst Google bought tons and tons of bandwidth, the dark fibers, the unlit strands of fiber optic lines. If the net fragments, I expect Google to step in, light up these strands and go head to head with the ISPs providing metro level WiFi. Since it is not a government project, it could not be sabotaged like Verizon and AT&T torpedoed municipal high peed networks.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
I also go through my client list and drop those that consume more of my time and resources in favour of the easier clients who ultimately improve my business at a lesser cost. What's wrong with that? My company, my rules. "We reserve the right to refuse service to anyone" -- it's in every restaurant. Why would you expect a business to serve you? Why would you consider it a right?
"We aren't getting the advertised bandwidth! Waaah!"
Yes, actually, false advertising is a problem. If an ISP tells me I can make unlimited use of my 10Mbps connection, I expect to be able to make unlimited use of it -- including sustaining 10Mbps or something reasonably close all day and all night. If such a level of service is impossible for an ISP to provide and remain profitable, why the hell are they advertising these plans?
If they are lying to consumers about the level of service they can provide, they should cover themselves by increasing the network capacity, or they should admit they lied, reduce the bandwidth they provide to users, and hope that nobody sues them over it. Kicking people off the network for trying to use what they paid for is not an appropriate response to overselling, and if the FCC had any spine they would kill the practice before it gets out of hand.
Palm trees and 8
First of all, I am, and always will be, a bandwidth hog. Why? Because I'm better at using the internet than everyone around me. That means I find more things, and bigger things, to download. If they someone banned P2P, I'd still have more streamed video than anyone I know. If they banned that, too, I'd still download more images. If they banned that, i'd still have more web traffic, email, IM, etc etc etc. I will always be a 'hog' in any environment. I was even told that I was "#1 abuser" of the "Unlimited" service when I was on dial-up in a small town and they tried to charge me an extra $300 that month. As someone else had just come into town, I switched, obviously.
I don't pay for the top tier of residential service to just let it sit idle. I'm going to -use- it.
I have absolutely no sympathy for people that sell me something and then get upset when I actually use it within the original limitations. I have only a small amount of sympathy for people that sell me something and I use it beyond their arbitrary limitations, even if I agreed to them.
Why?
America has -crap- for internet compared to other developed countries. We are quickly falling behind the rest of the world in terms of internet bandwidth. This is purely from greed and laziness on the part of the ISP. They refuse to upgrade and try to prevent competition at the same time. Sprint even has the nerve to advertise Pure and claim that it's faster than Cable internet, despite being 1/10th of the speed.
"If you make people think they're thinking, they'll love you; But if you really make them think, they'll hate you." - DM
There have been times when telephone networks, wireless networks, and utilities have been knocked offline due to too much demand. Most utilities have "turn off" agreements with heavy industrial users who can tolerate a shutdown in exchange for a concession, such as a lower rate, to ensure consumers still have service.
Even that doesn't work. Witness the jammed cell-phone lines after a major event or in days of yore jammed long-distance phone lines on holidays.
Cutting off or throttling heavy users during times when the network is saturated is sensible. A more sensible route is to charge by the GB or TB, as long as the charge is fair and reasonable.
A fair and reasonable charge would be "X" for your initial allowance plus Y for every block of bytes after that point. X would cover fixed costs such as the cost of billing. On a per-byte basis, X would be higher than Y. A fair and reasonable charge could include time-of-week-sensitive, bitrate-sensitive, and quality-of-service-sensitive billing. For example, there could be a 20% surcharge during afternoon and evening hours, a discount when you voluntarily throttle to a very low bitrate, or a discount when you accept a lower quality of service, such as for bulk file transfers.
Involuntary speed reductions or QOS reductions could be imposed on either heavy users or those users who volunteered for them in exchange for a discount when the network is saturated. These should not last more than the duration of the saturation, and the effect should be spread around in a fair way. Of course, since customers are paying by the bit and paying higher for higher-rate/higher-quality transmission, every time a service provider did this they would be hurting their revenue stream, at least a little.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
One problem is by default many network devices/OSes do bandwidth distribution on a per _connection_ basis not on a per IP basis. So if there are only two users and one user has 1000 active connections and the other has just one active connection the first user will get about 1000 times more bandwidth than the second user.
;).
;) ), BUT even when they "log out" they _still_ get always-on internet access except it's just on a lower priority (but NO byte quota!). A customer might be restricted to say 10GBs at "priority" a month.
P2P clients typically have very very many connections open. Wheres other clients might not.
A much fairer way would be to share bandwidth amongst users on a per IP basis. That means if two users are active they get about 50% each, even if one user has 100 P2P connections and the other user has only one measly http connection.
Then within each customer's "per IP" queue, to improve the customer's experience you could prioritize latency or loss sensitive stuff like like dns, tcp acks, typical game connections, ssh, telnet and so on, over all the other less sensitive/important stuff.
Of course if you have oversubscribed too much, you will have way too many active users for your available bandwidth. A fair distribution of "not enough" will still be not enough.
If you have two people and you give each a fair slice of one banana, they each get half a banana. Maybe both are satisfied.
If you have 1000 people and you give each a fair slice of one banana, they each get 1/1000th of a banana. Not many are going to be satisfied
And that's where we come to the other problem.
The problem with P2P is many customers will often leave their P2P clients on 24/7, EVEN when some of them don't really care very much about how fast it goes (some do, but some don't). To revisit the banana analogy, what you have here is 1000 people, and 1000 of them ask for a slice of the banana, EVEN though some of them don't really care - they'll only really feel like having a slice next week, when they're back from their holiday!
So how do you figure out who cares and who doesn't care?
Right now what many ISPs do is have quota limits - they limit how much data can be transferred in total. When the quota runs out "stuff happens" (connections go slow, users get charged more etc). So the users have to manage it.
BUT this is PRIMITIVE, because if you can figure out when a user doesn't care about the speed etc, technology allows you to easily prioritize other traffic over that user's "who cares" traffic.
So what's a better way of figuring it out?
My proposal is to give the customers a "dialer" which allows users to "log on" to "priority Internet" (and then only something starts counting the bytes
The advantage of this method is:
1) There is no WASTED capacity - almost all the available bandwidth can be used all the time, without affecting the people who NEED "priority" internet access (and still have unused quota).
2) It allows a ISP to better figure out how much capacity to actually buy.
3) If there is insufficient capacity for "priority Internet" the ISP can actually inform the user and not put the user on "priority" (where the quota is counted). While the user might not be that happy, this is much fairer, than getting crappy access while having your quota still being deducted.
Perhaps this system is not needed and will never be needed in countries that don't seem to have big problems offering 100Mbps internet access to lots of people.
But it might still be useful in countries where the internet access and telcos are poorly regulated/managed. For example - you run a small ISP in one of those crappy countries and so you pay more for bandwidth from your providers- this system could allow you to make better use of your bandwidth and to be a more efficient competitor. And maybe even give your customers better internet service at the same time.
Yes the ISP could always buy enough bandwidth so that _everyone_ can get the offered amount even though not everyone really cares all the time (believe me this is true). But that could mean the ISP's internet access packages being much more expensive than they could be.
X would cover fixed costs such as the cost of billing.
Billing is a very large cost. In the telco world, the cost of billing passed the cost of transmission about two decades ago. That's even more true for Internet transmission at the retail level. With proposed "economic" solutions, you have to factor in the costs of billing. Those costs apply both to the provider and the customer. If customers have to meter to control their expenses, that's a cost to the customer in attention, and drives business to competitors that require less attention.
Handling one phone call about a billing complaint eats up months of profits from selling the service. Complex billing means big call centers.
Except they are not "throttling" you, they are just giving you lower priority IF you use over 80% of your bandwidth for 15 minutes AND the whole segment is over 70% utilization. This means that grandma can still get her mail when you are seeding the new release of Ubuntu, but you "lose" bandwidth if you actually hit 100% congestion.