Slashdot Mirror
Hunting the Mythical "Bandwidth Hog"
eldavojohn writes "Benoit Felten, an analyst in Paris, has heard enough of the elusive creature known as the bandwidth hog. Like its cousin the Boogie Man, the 'bandwidth hog' is a tale that ISPs tell their frightened users to keep them in check or to cut off whoever they want to cut off from service. And Felten's calling them out because he's certain that bandwidth hogs don't exist. What's actually happening is the ISPs are selecting the top 5% of users, by volume of bits that move on their wire, and revoking their service, even if they aren't negatively impacting other users. Which means that they are targeting 'heavy users' simply for being 'heavy users.' Felten has thrown down the gauntlet asking for a standardized data set from any telco that he can do statistical analysis on that will allow him to find any evidence of a single outlier ruining the experience for everyone else. Unlikely any telco will take him up on that offer but his point still stands." Felten's challenge is paired with a more technical look at how networks operate, which claims that TCP/IP by its design eliminates the possibility of hogging bandwidth. But Wes Felter corrects that mis-impression in a post to a network neutrality mailing list.
They are generally using UDP so the original assertion that degrading the other users experience should be true as UDP should break down long before TCP does. Though I do agree that if Comcast's system works as described it's probably the best solution for a network that can't implement QoS.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Marge: We drove around until three in the morning looking for another open all-you-can-eat seafood restaurant.
Lionel Hutz: And when you couldn't find one?
Marge: [crying] We... went... fishing.
--
Lately I've had to deal with this problem. Our solution was rather simple. We use NTOP on an Ubuntu box at the internal switch. We replicate all the traffic coming into that switch to a port that the NTOP box listens on.
It may not be a perfect solution, but it can easily let us know who the top talkers are and give us a historical look at what they are doing.
From that report, we look for anyone uploading more than they download. We also look for people who upload/download a consistent amount every hour. If you see someone doing 80gb in traffic each day with 60gb uploaded, you probably have a file sharer. When you see the 24-hour reports for the user and see 2~3gb every hour on upload, you *know* you have a file sharer.
After that, it's as simple as going to the DNS server and locking their MAC address to an IP. Then, we drop all that traffic (access list extended is wonderful) to another Ubuntu box. That box has a web page explaining what we saw, why the user is banned, and the steps they need to take to get back online.
Most users are very apologetic. We help them to set up upload/download limits on their bittorrent client and then we put them back online.
I'd rather you do it wrong, than for me to have to do it at all.
I guess it's cheaper to sacrifice 5% of revenue than to have to undertake a network upgrade.
This mentality is part of why the U.S. lags so much in broadband.
I should point out that this sort of thing, while true, is often overstated because of poor local network configuration. When I first set up my new Vista machine a couple years back, I noticed that torrents on it would frequently interfere with internet connectivity on other networked devices in the house. I hadn't had this problem before and was curious as to the cause. I initially tried setting the bandwidth priorities by machine IP and by port, setting the desktop and specifically uTorrent's port to the lowest priority for traffic (similar to what ISPs do when they try to limit by protocol, but more accurate and without an explicit cap), but that actually made the situation worse; the torrents ran slower, and the other machines behaved even worse.
Turned out the problem was caused by the OS. Vista's TCP settings had QoS disabled, so when the router sent messages saying to slow down on the traffic, or just dropped the packets, the machine just ignored it and resent immediately, swamping the router's CPU resources used to filter and prioritize packets. The moment I turned on QoS the problem disappeared. The only network using device in my house that still has a problem is the VOIP modem, largely because QoS doesn't work quickly enough for the latency requirements of the phone, but it's not dropping calls or dropping voice anymore, it's just laggy (and capping the upload on uTorrent fixes it completely; the download doesn't need capping).
$_ = "wftedskaebjgdpjgidbsmnjgcdwatb"; tr/a-z/oh, turtleneck Phrase Jar!/; print
"If this doesn't scale, logically, up to the network at a whole, I'm not sure why."
Plenty of reasons why that won't scale up to the network as a whole. First and foremost, your ISP's network topology is a lot more effective for many users than the simple "star" topology most home router/switch combos give you. Beyond just the topology, the ISP uses better equipment that can cap bandwidth usage and dynamically shift priorities to maintain a minimum level of service for all users even in the presence of a very heavy user. The ISP also has much higher capacity links than what you have at home, and certainly more than the link they give you, and so even if there were a very poor topology and no switch level bandwidth management, it would be very difficult for a single user to severely diminish service for others.
I do not have any sympathy for ISPs when it comes to this issue. If they sell me broadband service and expect me to not use it, then they are supremely stupid, and retaliating against those users who actually make use of the bandwidth they are sold is just insulting. They oversold the bandwidth and they should suffer for it; blaming the users is just misguided.
Palm trees and 8
I also go through my client list and drop those that consume more of my time and resources in favour of the easier clients who ultimately improve my business at a lesser cost. What's wrong with that? My company, my rules. "We reserve the right to refuse service to anyone" -- it's in every restaurant. Why would you expect a business to serve you? Why would you consider it a right?
Your company's service isn't based on federal subsidies meant to provide internet access to all citizens.
"I zero-index my hamsters" - Willtor (147206)
First of all, I am, and always will be, a bandwidth hog. Why? Because I'm better at using the internet than everyone around me. That means I find more things, and bigger things, to download. If they someone banned P2P, I'd still have more streamed video than anyone I know. If they banned that, too, I'd still download more images. If they banned that, i'd still have more web traffic, email, IM, etc etc etc. I will always be a 'hog' in any environment. I was even told that I was "#1 abuser" of the "Unlimited" service when I was on dial-up in a small town and they tried to charge me an extra $300 that month. As someone else had just come into town, I switched, obviously.
I don't pay for the top tier of residential service to just let it sit idle. I'm going to -use- it.
I have absolutely no sympathy for people that sell me something and then get upset when I actually use it within the original limitations. I have only a small amount of sympathy for people that sell me something and I use it beyond their arbitrary limitations, even if I agreed to them.
Why?
America has -crap- for internet compared to other developed countries. We are quickly falling behind the rest of the world in terms of internet bandwidth. This is purely from greed and laziness on the part of the ISP. They refuse to upgrade and try to prevent competition at the same time. Sprint even has the nerve to advertise Pure and claim that it's faster than Cable internet, despite being 1/10th of the speed.
"If you make people think they're thinking, they'll love you; But if you really make them think, they'll hate you." - DM
I also go through my client list and drop those that consume more of my time and resources in favour of the easier clients who ultimately improve my business at a lesser cost. What's wrong with that? My company, my rules. "We reserve the right to refuse service to anyone" -- it's in every restaurant. Why would you expect a business to serve you? Why would you consider it a right?
Let's say you sell widgets.
You have 5 people come to you, each one wants to buy 1 widget. And another guy shows up and wants to buy 5 widgets.
You only have 5 widgets in stock, you need 10, but you really want their money. So you sell each of those people a coupon for their widgets, and tell them to pick it up at your warehouse. You figure they won't all run over there right now, and you'll probably have time to get a couple more widgets in stock before anybody notices.
Of course you don't tell your customers this. You don't tell them "I only have 5 right now, you'll have to wait 'til the next shipment" You just take their money and leave them with the impression that the widget is there, waiting for them, available for pickup whenever they want.
So all of them show up at the warehouse about 5 minutes later. All of them want their widgets now. But you don't have enough widgets to go around. So you call the guy who bought 5 widgets a "widget hog", cancel his order, and throw up a hastily-made sign that says "limit 1 per customer."
Legal? Yeah, I guess... Assuming you refund his money.
Right? Not so much. You should have clearly explained that you only have 5 widgets in stock, or that the coupon couldn't be redeemed for a week, or that there was a limit of 1 per customer, or something. You mis-represented what you were selling to your customers.
Likely to leave a good impression on your customers? Nope.
"Work is the curse of the drinking classes." -Oscar Wilde
One problem is by default many network devices/OSes do bandwidth distribution on a per _connection_ basis not on a per IP basis. So if there are only two users and one user has 1000 active connections and the other has just one active connection the first user will get about 1000 times more bandwidth than the second user.
;).
;) ), BUT even when they "log out" they _still_ get always-on internet access except it's just on a lower priority (but NO byte quota!). A customer might be restricted to say 10GBs at "priority" a month.
P2P clients typically have very very many connections open. Wheres other clients might not.
A much fairer way would be to share bandwidth amongst users on a per IP basis. That means if two users are active they get about 50% each, even if one user has 100 P2P connections and the other user has only one measly http connection.
Then within each customer's "per IP" queue, to improve the customer's experience you could prioritize latency or loss sensitive stuff like like dns, tcp acks, typical game connections, ssh, telnet and so on, over all the other less sensitive/important stuff.
Of course if you have oversubscribed too much, you will have way too many active users for your available bandwidth. A fair distribution of "not enough" will still be not enough.
If you have two people and you give each a fair slice of one banana, they each get half a banana. Maybe both are satisfied.
If you have 1000 people and you give each a fair slice of one banana, they each get 1/1000th of a banana. Not many are going to be satisfied
And that's where we come to the other problem.
The problem with P2P is many customers will often leave their P2P clients on 24/7, EVEN when some of them don't really care very much about how fast it goes (some do, but some don't). To revisit the banana analogy, what you have here is 1000 people, and 1000 of them ask for a slice of the banana, EVEN though some of them don't really care - they'll only really feel like having a slice next week, when they're back from their holiday!
So how do you figure out who cares and who doesn't care?
Right now what many ISPs do is have quota limits - they limit how much data can be transferred in total. When the quota runs out "stuff happens" (connections go slow, users get charged more etc). So the users have to manage it.
BUT this is PRIMITIVE, because if you can figure out when a user doesn't care about the speed etc, technology allows you to easily prioritize other traffic over that user's "who cares" traffic.
So what's a better way of figuring it out?
My proposal is to give the customers a "dialer" which allows users to "log on" to "priority Internet" (and then only something starts counting the bytes
The advantage of this method is:
1) There is no WASTED capacity - almost all the available bandwidth can be used all the time, without affecting the people who NEED "priority" internet access (and still have unused quota).
2) It allows a ISP to better figure out how much capacity to actually buy.
3) If there is insufficient capacity for "priority Internet" the ISP can actually inform the user and not put the user on "priority" (where the quota is counted). While the user might not be that happy, this is much fairer, than getting crappy access while having your quota still being deducted.
Perhaps this system is not needed and will never be needed in countries that don't seem to have big problems offering 100Mbps internet access to lots of people.
But it might still be useful in countries where the internet access and telcos are poorly regulated/managed. For example - you run a small ISP in one of those crappy countries and so you pay more for bandwidth from your providers- this system could allow you to make better use of your bandwidth and to be a more efficient competitor. And maybe even give your customers better internet service at the same time.
Yes the ISP could always buy enough bandwidth so that _everyone_ can get the offered amount even though not everyone really cares all the time (believe me this is true). But that could mean the ISP's internet access packages being much more expensive than they could be.
comcast = cable = coax style networking in modern form, no?
that is, its like going back to pre-hub style ethernet, where every computer is listening for the next millisecond of no signal on the coax so that it can hopefully push its next packet on there. There is a reason why this was quickly replaced with switches when said tech became available at acceptable prices...
No, No NO! For the love of God, NO! You're completely wrong, and you have no idea what you're talking about. There is no such thing as "coax style networking", and there never has been. And the network behavior of cable broadband connectivity has nothing whatsoever to do with the fact that some cable connections use coaxial wiring.
You are probably thinking of the old 10BASE2 Ethernet standard (http://en.wikipedia.org/wiki/10BASE2), which used coaxial cable with BNC connections and T-connectors to a shared cable bus medium. Cable broadband uses the DOCSIS protocol (http://en.wikipedia.org/wiki/DOCSIS) over coaxial cable with F connectors. The cable is the only really similar thing between the two technologies, everything else is pretty different.
10BASE2, like all Ethernet technologies, is a shared-medium, PURE collision-detection protocol. The hosts share the cable segment as a broadcast medium, so that a transmission by one host will be "heard" by all the rest. Each host makes its own decisions about when it wants to transmit, independent of the rest, and then transmits when it senses that the cable is "silent". If multiple hosts start transmitting at almost exactly the same time, they will all shortly detect the "collision". They all cease transmitting, and each picks a short random-length interval to wait before trying to transmit again, unless another host that picked a shorter timeout window starts transmitting, first. Statistically, it's unlikely that two hosts will pick the same random wait timeout, so most collisions resolve quickly unless the network is particularly congested.
DOCSIS uses a mixture of time-division, code-division, and collision-based contention behaviors (depending on the exact revision, too), but the impact of contention is really limited. From a bandwidth scheduling and congestion standpoint, it's nothing like 10BASE2, because the TDMA and CDMA elements of the protocol help each node sees a "fair share" of throughput. Plus, modern DOCSIS supports quality-of-service tags, which (if properly implemented) are pretty much a brick wall against congestion issues.
mostly to me it seems that the ISPs that cries highest are the ones that geared up when the net was mostly static webpages and ftp file transfers, able to handle the odd spike of traffic when someone clicked a link. But now the gear they have sitting around, and that they where banking on where not to be replaced for the next decade or so, baring hardware failure, is being swamped by continual "spikes". And the only way they can fix that at their end is by replacing the gear ahead of schedule, playing havoc with their earnings estimates. And rather then doing that, they break out the whip, trying to force the "cattle" back into the "pen".
I don't think you have any kind of real grasp on the technical implications of terms like "swamped" or "spike" in this context. You certainly understand the metaphor, and I bet you could analogize extensively comparing electrical, water, or highway systems to the Internet, but you don't seem to know too much about actual networking beyond setting up your home LAN.