Where The Bandwidth Goes

An anonymous reader writes "An often overlooked fact about network bandwidth utilization is that the bandwidth consumed on networks is more than the sum of the data exchanged at the highest level; it's data+overhead+upkeep. In the early 90's I worked for a large multi-national company whose software engineering department had a transatlantic x.25 circuit connection to it's European engineering headquarters. It was necessary that the connection be 'on' 24x7 due to the spanning of a large number of time zones, disparate working hours and tight contractual requirements. Very large data transfers were sometimes operationally essential. But the financial people used to scream constantly about the circuit costs (charged per packet, IIRC) of several thousand dollars/month. The sys admin realized that if he just reduced the frequency of keep-alives, he could shave something like 10% off the monthly bill. This article points out that p2p applications are greater bandwidth hogs than one might think because of the foregoing and more - they also search, accept pushed advertising and do other transactions that are transparent to most users, but add up. I doubt that developers of those free p2p applications have gave much thought to efficiency. This will be no surprise to many of you, but helps explain why ISP's rushing to put caps on transfers."

Comments

This has been known for a long time...

[fortinbras47]

With gnutella, QueryHit packets can make up as little as 1% of traffic (by numbers of packets, not size) while Ping and Pong packets can be well over 50% of packets. Check out this article to see more detail.

Gnutella is not one of the more advanced protocols, but most of it's problems are present at varying levels in other p2p systems. It's not really surprising that P2P software which spends so much time trying to connect to computers, connect to a computer to start a download etc... and search in a geometric spiderring fashion are quite inefficient.

Re:Text compresses

[EnderWiggnz]

no, there arent better solutions.

as i stated earlier, go to google and look at the source - one letter field names, and no line breaks.

why? TO SAVE BANDWIDTH.

every byte is sacred... every byte is great...

and if a byte is wasted, CFO's get quite irate...

Re:AT&T Blocking P2P?

[martissimo]

there is a big thread thats been going about this situation over at dslreports.

if you dig a page or two back into the thread quite a few users seemed to have success by using some http tunnelling software so they were on a non throttled port. might just find an answer to your problem in there if ya look

Streaming Real TV

[silas_moeckel]

As an engineer that made a network to do that that tanked (Nothing like lies from sales people) it's possible 512 kbit a sec looks pretty nice but the bandwith costs on the sending end are about 50 bucks a month before servers people etc (thats sending all month) why because it's all unicast because NO ISP wants mcast working outside of itself they dont know how to bill for it. A satalite at 500 an hour is much cheaper than delivering over the internet and inherently multicast. I wonder when somebody will come up with a multicast service that is delivered to a majority of ISP's.

Gnutella IS inefficient

[PotatoError]

If you take a look at the spec for the Gnutella Protocol, you will understand where all this "extra" traffic is coming from.

I've been messing about with gnutella on and off for about 3 months now, I hope to make an open source functional client eventually. It's quite an interesting area because there is so much work to be done on security and efficiency.

The only problem is that P2P networks are never going to be as efficient as centralised server networks and certainly never as fast. I suppose a cynic (like me!) could blame the entertainment industry for forcing out server based file sharing networks.

But I believe the death of server based file sharing is a good thing. The bad side of the server-client model is that it can (and usually is) controlled by an authority and its security is often obscurity based (the obsure bit being hiden on the server). Peer to Peer networks however, offer total anonominity as well as giving users access to the whole component.

Peer to Peer networks are the next step in securing freedom of information on the internet and preventing government control.

It's when Peer to Peer mobile phone networks are produced that things will really get interesting....

Re:compression

[ShaunC]

It would simply be a matter of Apache and IIS supporting it

Apache does support it, it's called mod_gzip and Slashdot already uses it. The IIS equivalent (sort of) is called PipeBoost.

Shaun

Efficient search in peer networks

[PureFiction]

I doubt that developers of those free p2p applications have gave much thought to efficiency

Some of us have. Search is much of the bandwidth in peer networks is wasted (downloads are downloads, but search can eat up a lot of bandwidth for little return)

There are some efficient, effective peer network search apps currently in development. Hopefully we can eventually leave gnutella and kazaa in the past and move on to more open, efficient networks...

Re:nope, he's not insane.

[kaphka]

Got a reference for that?

RTFM.

• "
• Do not prefix parameter names with Hungarian type notation.
• "

mod_gzip for Apache

[jsjacob]

A new HTTP command is not necessary because HTTP 1.1 supports compression as a content encoding (the "Content-Encoding" HTTP header). The mod_gzip module enables compression for Apache. As you suggested, mod_gzip can be configured to compress or not compress certain files matching given criteria.