Where The Bandwidth Goes

An anonymous reader writes "An often overlooked fact about network bandwidth utilization is that the bandwidth consumed on networks is more than the sum of the data exchanged at the highest level; it's data+overhead+upkeep. In the early 90's I worked for a large multi-national company whose software engineering department had a transatlantic x.25 circuit connection to it's European engineering headquarters. It was necessary that the connection be 'on' 24x7 due to the spanning of a large number of time zones, disparate working hours and tight contractual requirements. Very large data transfers were sometimes operationally essential. But the financial people used to scream constantly about the circuit costs (charged per packet, IIRC) of several thousand dollars/month. The sys admin realized that if he just reduced the frequency of keep-alives, he could shave something like 10% off the monthly bill. This article points out that p2p applications are greater bandwidth hogs than one might think because of the foregoing and more - they also search, accept pushed advertising and do other transactions that are transparent to most users, but add up. I doubt that developers of those free p2p applications have gave much thought to efficiency. This will be no surprise to many of you, but helps explain why ISP's rushing to put caps on transfers."

Comments

I doubt it.

[garcia]

it has nothing to do w/the advertising, the searches, etc. It has to do SOLELY w/the LARGE downloads that users of P2P networks do.

Over the summer (when no one was in this little college town) I was steadily get 250+k/s downloads (mostly updating Debian ;)) Now that everyone is back (and I assume loving Kazaa to it's limit) I average about 75 to 100k/s.

I am even tempted to call Road Runner and complain (I am just too lazy to fix Win98 and have it running so they can do their tests).

DiVX and MP3s are what kills the bandwith. Not the little "inefficiencies" that P2P authors added in.

Re:I doubt it.

[jbarr]

I really wouldn't necessarily characterize it as "chatter". After all, isn't the point of P2P to allow multiple hosts to "share the load"? Though you may not be downloading anything, many might be downloading from you. (This is, of course, assuming you have your client configured to share.) KaZaA, by default, puts its installable executable in your shared directory making it available for anyone to grab.

If you have ever "expanded" the downloding sources, it often shows the download being done from multiple sources. It could just be that your client is uploading part or all of some shared file.

Not that there aren't inefficiencies, though.

Re:I doubt it.

[Stephen Samuel]

so you are going to tell me that when I download 5 DiVX movies a day (700+mb each) that I am *NOT* hogging 3.5G of bandwith for that day?

Guh.
His point was that even when you're not downloading your 3G of files a day, you're still using a good bit of bandwidth. The ofiginal article mentions that 10% of his network's bandwidth could be chopped by simply cutting down on network keep-alive packets. If the grand-parent's comments are any indication, Some P2Ps may be well above that.

Or to put it another way: I'm not saying that you always look stupid -- just when you say things like that.

Doesn't explain anything

[afidel]

Actually P2P work does focus on efficiency because efficiency determines how large the network can scale on a give set of hardware (the users machines and comodity internet connections). ISP's want to cap bandwidth because their current business model demands that they oversubscribe their uplink by around 20-200 times depending on the type and pricing of the comodity connection. Besides caps are based on total bandwidth usage which includes networking overhead (the routers accounting program doesn't care about payload usually)

We need web caches

[Nicopa]

We need web caches... It's stupid to have files crossing the ocean thousands of times. Besides not using web caches causes that those who cannot afford bandwidth costs cannot put content in the web... Caches now!.

Web developers must not be afraid of web caches, since the HTTP/1.1 protocol allows them to precisely define how and when their content will be cached.

Re:We need web caches

[Cloudmark]

While caching does offer a lot of advantages, there are also pitfalls, particularly for those providing it.

Working for an ISP myself and specifically with the bandwidth tracking section, we deal with prety much every type of high bandwidth application out there and in many cases we could save an immense amount by caching. Unfortunately, if we cache and then illegal material is downloaded, we can be held responsible for that material. It's unfortunate that efficiency must be sacrificed but right now it's generally too dangerous for anyone to run a serious caching system.

The rule of thumb for ISPs, at least in North America, is generally that if it's on a client system (subscriber - your PC), then it's not our problem (legally). If a file resides on our cache, then we can be held responsible for it by law enforcement agencies.

As to the general suggestion that a great deal of bandwidth is consumed by overhead, I think there is some merit to it but that it's a fairly small amount compared to what is used by deliberate downloads and transfers. Systems are moving towards greater efficiency in order to improve speed and to work with lower bandwidth platforms (phones, PDAs, etc) but bandwidth is unlikely to be a major motivator. Most broadband subscribers either download too little to cause serious issues (6gb a month or so - limited overhead) or extreme volumes (100gb a month - overhead is dwarfed by content).

...but it'll bite them in the end.

[The Fanta Menace]

The more they cap usage, the less people will use (obviously). Then content providers such as streaming radio stations will start to drop off as it becomes more expensive for users to access them.

After that it becomes a vicious circle, with fewer content providers, there's no reason for users to keep their service. Then the ISPs go broke.

Take a look at the Australian example. Almost all broadband providers have a 3Gb monthly cap. The ABC has just started an internet-only radio station, but I really wonder why. It wouldn't take too many days of listening to it for a user to totally max out their cap. I predict the station will be closed due to lack of interest, within a year.

hrmm yeah i guess so

[digitalsushi]

as an ISP i can say that we make our money by a gamble that people use X amount of bandwidth. p2p breaks our precious little ratio of what we expect and what we need.

the geek it me though, says "waaa" and that things that dont evolve, die. and the things that dont die. p2p pushes the envelope right now, but all that encourages is more network growth. just think of p2p as those pains you had in your legs when you were 14. sure, it may not be the most efficient thing in the world, but the underlaying infrastructure has to take that into account, or get out of the way for one that can.

Re:hrmm yeah i guess so

[Fastolfe]

p2p breaks our precious little ratio of what we expect and what we need

Uhh, yah, except this is how they're determining how much they can charge you. If the ratio becomes permanently skewed, the way they "evolve" as you put it is to simply skew their prices to compensate. Though your end user connections may be effectively "unlimited", someone upstream pays for the bandwidth by how much data gets transferred. I guarantee the costs will filter down.

So as a business, what would you do? Raise your rates for all "unlimited" customers? Create a new class of DSL customer with a lower bandwidth cap and re-figure the ratio? Block P2P activity entirely? Write into the end user contract some soft usage caps and go after the top 1% of bandwidth consumers? All of the above?

I don't really think P2P is going to drive growth (i.e. more bandwidth for less cost) any more aggressively than the growth we're already seeing. I just think it's going to annoy ISP's and make them re-think some of their "unlimited" bandwidth plans.

Re:nope, he's not insane.

[EvanED]

>>most of us dont need the damn hungarian notation that MS has spreads like gospel truth

Why said anything about that? And besides, MS now discourages its use.

>>It makes for unreadable names that convey less meaning that a nice clear variable name.

Which 'fn' is not but 'FirstName' is.

Now, if you have a dynamically generated page, you could use constants that are set to short stuff like 'fn'. Less code to be transmitted while still keeping most of the readability of the original code. If you discover a bug, temporarily switch to a different set of constants ('FirstName' instead of 'fn') until you sort it out so the resultant HTML is more readable. (Same goes with whitespace: Make a constant ENDL or NEWLINE that is set to '\n' while debugging, then changes to '' for production.)

Comcast where I live..

[stratjakt]

..is airing this commercial of goofy testimonials for their broadband cable service. A kid says "Ever been in the belly of a whale? I have", another guy goes "I go to the moon and back twice a day", etc.. etc..

Now, one of them has some guy say "I collected everything Mozart ever did... In 10 minutes!"

To me that's comes through loud and clear as "*wink* *wink* *nudge* *nudge* napster(etc)!"

I would say p2p is the driving force behind non-geeks getting broadband. They don't need it for e-mail, or casual web-surfing. They don't play games, but I know many people eager for an alternative to the bland junk on the radio. (Plus due to geography, radio reception is poor here)

Same thing with the 'work from home' bunk they promote, and yet block VPN connections.

It's like dangling a carrot in front of a mule to get him to move, and he stupidly chases it not realising he'll never reach it. It works fine in cartoons, but eventually the mule becomes frustrated, kicks you, and refuses to move at all.

Someone is smart enough to figure a way to give out the bandwidth and make money at the same time. And, it won't be a monopoly. Maybe 802.11 will be our savior?

Wow

[Salamander]

The article itself was kind of ho-hum, but the following part of the Slashdot intro caught my attention:

I doubt that developers of those free p2p applications have gave much thought to efficiency.

Again...wow. One would need to search far and wide, even on Slashdot, to find another example of such absolutely astonishing cluelessness. Timothy has obviously never talked to a P2P developer in his life. Sometimes it seems like efficiency is just about the only thing P2P developers think about, unless someone's on a security/anonymity rant. Little things like robustness or usability get short shrift because so much of the focus is on efficiency. Hundreds of papers have been written about the bandwidth-efficiency of various P2P networks - especially Gnutella, which everyone who knows anything knows is "worst of breed" when it comes to broadcasting searches.

It's unfortunate that the most popular P2P networks seem to be the least efficient ones, and doubly unfortunate that so many vendors bundle spyware with their P2P clients, but to say that P2P developers don't give much thought to efficiency is absurd. They give a lot more thought to efficiency than Slashdot editors give to accuracy, that's for damn sure.