Where The Bandwidth Goes

An anonymous reader writes "An often overlooked fact about network bandwidth utilization is that the bandwidth consumed on networks is more than the sum of the data exchanged at the highest level; it's data+overhead+upkeep. In the early 90's I worked for a large multi-national company whose software engineering department had a transatlantic x.25 circuit connection to it's European engineering headquarters. It was necessary that the connection be 'on' 24x7 due to the spanning of a large number of time zones, disparate working hours and tight contractual requirements. Very large data transfers were sometimes operationally essential. But the financial people used to scream constantly about the circuit costs (charged per packet, IIRC) of several thousand dollars/month. The sys admin realized that if he just reduced the frequency of keep-alives, he could shave something like 10% off the monthly bill. This article points out that p2p applications are greater bandwidth hogs than one might think because of the foregoing and more - they also search, accept pushed advertising and do other transactions that are transparent to most users, but add up. I doubt that developers of those free p2p applications have gave much thought to efficiency. This will be no surprise to many of you, but helps explain why ISP's rushing to put caps on transfers."

Comments

I doubt it.

[garcia]

it has nothing to do w/the advertising, the searches, etc. It has to do SOLELY w/the LARGE downloads that users of P2P networks do.

Over the summer (when no one was in this little college town) I was steadily get 250+k/s downloads (mostly updating Debian ;)) Now that everyone is back (and I assume loving Kazaa to it's limit) I average about 75 to 100k/s.

I am even tempted to call Road Runner and complain (I am just too lazy to fix Win98 and have it running so they can do their tests).

DiVX and MP3s are what kills the bandwith. Not the little "inefficiencies" that P2P authors added in.

Re:I doubt it.

[Anonymous Coward]

uh, no. the actual bandwidth hogs there are not the files themselves, but the packet overhead of transferring those files. Keep in mind that searches are broadcast, so when one person searches next to you, they send out N search packets, one of which goes to you. You send out N search requests on their behalf to all the people you're connected with, and so on. So if all N of your peers are searching simultaneously (this is on a closed network of just you and N people, keep in mind), you're forwarding somewhere like N^2 packets.

At N=16 and average packet size of 128Bytes, that's 16*16*128 = 32KB. 32KB for just you and 16 friends, nobody else. As soon as you add more people to the topology, the math gets trickier and the numbers get much, much larger. Also keep in mind that the 128 is "ideal", not including the overhead of TCP establishing sessions, etc.

There were a lot of papers published on this about a year ago, i don't have any references to them, though.

Re:I doubt it.

[Deagol]

It has nothing to do w/the advertising, the searches, etc. It has to do SOLELY w/the LARGE downloads that users of P2P networks do.

IMO, you're very wrong.

The university I work for has it's spies watching the border routers, logging streams. Daily, they release a "top talker" list to a select few individuals (not myself) who notify the admins of aberrant hosts. This is to stop blatant abuse, as well as cut off possibly compromised hosts.

Occasionally, I would leave gnut running in a shell when I left for the day. I'd usually end up on "top talkers" with 1-2GB of traffic when no downloads were running! This was solely the chatter of the gnutellanet in action.

Of course, I do configure my client to talk to a rather wide neighborhood, but still...

Re:I doubt it.

[Anonymous Coward]

Every time I visit a web page using my cable modem, I feel a pang of guilt. By visiting a web page, I:

• drive up bandwidth costs for the webmaster that are not covered by advertising
• consume my ISP's bandwidth
• consume shared bandwidth, slowing down my neighbor's computer slightly.

Bandwidth is a finite resource which we should all conserve. One day, eventually, the Internet will run out of bandwidth.

Re:I doubt it.

[garcia]

so you are going to tell me that when I download 5 DiVX movies a day (700+mb each) that I am *NOT* hogging 3.5G of bandwith for that day?

Sure, traffic hogs it, but so do the files. When you have 85% of your cable modem subscribers downloading 700+mb files daily, all day, everyday, you are going to see a significant drop in your overall bandwith.

Again, I will point out that when no one was here in this college town (25000 person difference) that the bandwith was WIDELY available. Once RR started getting everyone back on, the bandwith went to hell.

Re:I doubt it.

[jbarr]

I really wouldn't necessarily characterize it as "chatter". After all, isn't the point of P2P to allow multiple hosts to "share the load"? Though you may not be downloading anything, many might be downloading from you. (This is, of course, assuming you have your client configured to share.) KaZaA, by default, puts its installable executable in your shared directory making it available for anyone to grab.

If you have ever "expanded" the downloding sources, it often shows the download being done from multiple sources. It could just be that your client is uploading part or all of some shared file.

Not that there aren't inefficiencies, though.

Re:I doubt it.

[SerpentMage]

Here in Europe I have to pay for bandwidth. And so long as I do not run anything like Gnutella I have no bandwidth problems. I can share with Kazza and no problems. But the moment I share with GNUTella, my bandwidth shot through the roof. I kept it running a week and have never started it again.

Re:I doubt it.

[Deagol]

No, I don't allow downloads. I usually hop on to find some obscure file, then hop right off. I don't share files much these days, certainly not with my work machine.

I don't even allow caching -- too gray an area for my tastes, especially on my employer's equipment.

Re:I doubt it.

[Deagol]

Perhaps I responded a bit too harshly. Your original point seemed to imply that nothing other than downloads acounted for a non-trivial use of overall bandwidth. I responded with anecdotal evidence that this is not always the case.

With the "always on" mentality of broadband users, and the fact that most clients simply hide in the system tray when you click the top-right "x" on the window rather than shut down, it wouldn't surprise me if a substantial amount of bandwidth wasn't directly related to a particular client downloading.

Re:I doubt it.

[dattaway]

Bandwidth is a finite resource which we should all conserve. One day, eventually, the Internet will run out of bandwidth.

Consider joining the Bandwidth Conservation Society.

Re:I doubt it.

[gwernol]

Though you may not be downloading anything, many might be downloading from you. (This is, of course, assuming you have your client configured to share.) KaZaA, by default, puts its installable executable in your shared directory making it available for anyone to grab.

You're missing the point - go back and read the article linked in the story. The point is that excluding uploads and downloads these P2P networks are producing a lot of nework traffic. The example quoted is up to 1.6GB a day just for running the client. Again, this is excluding the bandwidth required for uploads and downloads to/from your machine. This is just the overheads of communication, searches and ad pushes.

Not that there aren't inefficiencies, though.

The point being the inefficiencies are so large that just having a few hundred P2P machines running on your network can amount to a significant bandwidth drain, even before they share a single file.

Re:I doubt it.

[Stephen Samuel]

so you are going to tell me that when I download 5 DiVX movies a day (700+mb each) that I am *NOT* hogging 3.5G of bandwith for that day?

Guh.
His point was that even when you're not downloading your 3G of files a day, you're still using a good bit of bandwidth. The ofiginal article mentions that 10% of his network's bandwidth could be chopped by simply cutting down on network keep-alive packets. If the grand-parent's comments are any indication, Some P2Ps may be well above that.

Or to put it another way: I'm not saying that you always look stupid -- just when you say things like that.

Re:I doubt it.

[putzin]

It's not just p2p either. I run Ethereal every now and then on my DSL router to keep track of those intrepid hackers using my wireless connection (the girl two floors down apparently thinks her computer is actually "on" the internet) and to see if anyone has decided it is time to break in. Just watch a typical email session on hotmail. 80%+ of the traffic is solely advertising and hotmail related extra crap. There were 1000 plus packets before I even saw the first message header (I thought someone was doing something naughty). I couldn't believe it at first, but really, that's just amazing. Now the point.

Which is, we really don't notice all of the extra traffic generated every time we hit a website or fire up Morpheus. Generally, you expect the downloads, but you don't expect the protocol overhead, or the ads, or keepalives, or whatever else might be bundled in. This is where we could save bandwidth if we wanted to. But, we don't want to. I would freak if Ameritech imposed a bandwidth restriction.

Re:I doubt it.

[Magila]

Gnutella is a bad example since it's about the worst possible senerio in terms of wasted bandwidth. Something like kazaa (assuming you're not a supernode) or edonkey use a trivial amount of bandwith while idle since they're not constantly receiving and sending searches (i.e. they keep servers and clients separate). I've said many times that gnutella is a steaming pile because it wastes so much bandwidth on searches, your example is proof just how bad it is.

Re:I doubt it.

[DrMaurer]

re: The "always on" mentality:

Well, I don't know if you pay attention to stuff or not, but the advertisements for the cable modems here seem to push that kind of ideal.

"Always on. No busy signals. Fast downloads of multimedia."

Then they complain when we do exactly that. Don't advertise it unless you're willing to deliver it.

But, the question: Is it (bandwidth) a "right?"

I think so. Maybe. Hard to say. It's not a right like life or something that vital, but the limiting of servers and other "bandwidth issues" makes me concerned because it starts to make the internet a one way conduit of information instead of allowing all users to serve their own content, which is really not what the Internet is capable of or, IMO, the ideal usage of the 'net is.

Beyond those ideals, which I admit, probably aren't shared by C*O's around the world, it's important that companies that get into this broadband invest in increasing their own bandwidth. The bandwidth "limit" companies complain about is fixable, and it's only going to get worse if they refuse to make that bandwidth grow with their hopeful customer base.

There is money to be made with the promise of freedom. Are you more likely to go with an (otherwise idential) ISP that allows/encourages (even simply does not dis-allow) servers for your content as opposed to one that disallows & tries to block their use? Freedom is important, and people, especially now, are conginzant (sp) of that.

Yes, I use P2P, not gnutella or kazaa or popular ones but private hubs; take your salt as recommended by your physician.

Optimize html

[Winterblink]

I wonder how much bandwidth could be saved annually if people who developed webpages maybe optimized their html a little better? Removing extraneous spacings, simplifying form field namings ("fn" instead of "FirstName"), that kind of thing. Especially sites that get insane amounts of traffic. You know, like Slashdot. :)

Re:Optimize html

[heyeq]

If you're anywhere near being a k0der you'll be squeezing your head in digust. for fifteen+ years we've been battling against asm-style two/three letter variable declarations, and finally have languages that have helped us define naming conventions and the like, and you want us to go BACK to TLA's??? (TLA = two/three letter acronym)

are you insane?

Re:Optimize html

[stefanlasiewski]

Or, as an alternative, how much bandwidth could be saved if webservers used compression, like mod_gzip?

Re:Optimize html

[!splut]

I tend to think that poorly-optomized html is just a drop in the bucket. If there were one thing to optomize, it ought to be images. The average page uses at least an order of magnitude more data for its images than the html. Using smaller images, or just saving things in more efficient formats, such as jpg, or lossless compression formats, would be a big step in the right direction.

Re:Optimize html

[J'raxis]

<paragraph> Would you rather write HTML like <emphasis>this</emphasis>? </paragraph>

<paragraph> Sure, &open-quotation-mark; TLAs &close-quotation-mark; may be annoying to read, but they are certainly OK to use if they are understandable enough. </paragraph>

</hypertext-markup-language>

Re:Optimize html

[Sean Clifford]

Good point. However, readability is important. I do a lot of ASP (yeah, I know..., migrating to perl) coding and optimize my code for scalability and speed; however, I won't sacrifice readability. I leave my comments in. Yes, I know I can strip them out for the production server, but I prefer to leave my comments in and use vbcrlf line feeds to break up the html into something readable. Helps with debugging on the development server.

That said, text compresses pretty damned well. So I'm less concerned with text than with images. I much prefer png to jpeg & gif, but gotta make sure the client supports it (dynamically feed whichever).

For sites with massive traffic, you're right - it behooves you to optimize everything you can - but not to the point of making life difficult for the development team:

Monkey 1: "Hey, what the hell is 'lnm'?"
Monkey 2: "Hell if I know."
Monkey 1: "You wrote it!"
Monkey 2: "Yeah, but that was like 4 months ago. What do the comments say?"
Monkey 1: "DOH!"

I don't like my HTML generated in one humumgous unbroken string when I look at the source.

Re:Optimize html

[silentbozo]

My question is, what does mod_gzip break? Hell, my crappy HTML/1.0 client supports gzip content (Netscape 4.08), so why aren't more people using it?

Not only does it save bandwidth on the server side, but text pages render instantaneously on the client side when the server supports it (I'm using Netscape on an old Mac, so this kind of stuff is important.) I say everyone should plug it in, and we'll fix all the broken stuff afterwards...

Re:Optimize html

[Hast]

Does Slashdot do GZIP? In any case I bet that if you added GZIP to Slashdot the transfered size would shrink a lot. (But you wouldn't see that if you looked at the size of the source, as it's decompressed.)

And no optimizations in the world are going to remove the big shunk of data on Slashdot. Since that is in the posts.

Re:Optimize html

[SCHecklerX]

Too bad there isn't a 'mod_gzip' for mail. That would be something that could save bandwidth.

Re:Optimize html

[Sean Clifford]

I didn't think you were suggesting server-side; just that I prefer readability on the client side. But as you and several others pointed out, saving 25%+ bandwidth means that I should optimize client output on the production server.

I can't flame you for ASP development - that's how I make most of my living and what the clients want.

Doesn't explain anything

[afidel]

Actually P2P work does focus on efficiency because efficiency determines how large the network can scale on a give set of hardware (the users machines and comodity internet connections). ISP's want to cap bandwidth because their current business model demands that they oversubscribe their uplink by around 20-200 times depending on the type and pricing of the comodity connection. Besides caps are based on total bandwidth usage which includes networking overhead (the routers accounting program doesn't care about payload usually)

Re:Doesn't explain anything

[Deluge]

Another thing you can do to help yourself is the DNS cache. Make it HUGE. Watch the number of requests that come out of clients. You will be amazed. I would guess on a average 2-3 hour surf for me. I goto 10-15 pages. Those pages in turn hit 10-15 OTHER links. It gets rather large quickly. A nice way to speed up my connection is to turn on a decent DNS cache. Windows is rather pittfully small

You can edit the DNS Client service settings in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Servic es\DNSCache\Parameters. Obviously this applies to NT/2k/XP only, but who uses 9x for anything other than games anymore anyway.

We need web caches

[Nicopa]

We need web caches... It's stupid to have files crossing the ocean thousands of times. Besides not using web caches causes that those who cannot afford bandwidth costs cannot put content in the web... Caches now!.

Web developers must not be afraid of web caches, since the HTTP/1.1 protocol allows them to precisely define how and when their content will be cached.

Re:We need web caches

[Cloudmark]

While caching does offer a lot of advantages, there are also pitfalls, particularly for those providing it.

Working for an ISP myself and specifically with the bandwidth tracking section, we deal with prety much every type of high bandwidth application out there and in many cases we could save an immense amount by caching. Unfortunately, if we cache and then illegal material is downloaded, we can be held responsible for that material. It's unfortunate that efficiency must be sacrificed but right now it's generally too dangerous for anyone to run a serious caching system.

The rule of thumb for ISPs, at least in North America, is generally that if it's on a client system (subscriber - your PC), then it's not our problem (legally). If a file resides on our cache, then we can be held responsible for it by law enforcement agencies.

As to the general suggestion that a great deal of bandwidth is consumed by overhead, I think there is some merit to it but that it's a fairly small amount compared to what is used by deliberate downloads and transfers. Systems are moving towards greater efficiency in order to improve speed and to work with lower bandwidth platforms (phones, PDAs, etc) but bandwidth is unlikely to be a major motivator. Most broadband subscribers either download too little to cause serious issues (6gb a month or so - limited overhead) or extreme volumes (100gb a month - overhead is dwarfed by content).

Re:We need web caches

[Salamander]

Unfortunately, if we cache and then illegal material is downloaded, we can be held responsible for that material.

That's not really true. Even under the DMCA you can qualify as a safe harbor and avoid that liability. The requirements are too burdensome for an invididual, and that IMO is only one flaw among many in the DMCA, but for an ISP that employs full-time staff it's entirely doable and many ISPs have done it. You'll need to find another excuse.

Re:We need web caches

[Sloppy]

I agree. But I must be wrong about something, because ISPs don't seem to be deploying caches. ISPs would seem to have the most to gain from caches, and they are also at a very natural and sensible point for it.

I know there are ISP slashdotters. Any of you guys want to explain why web caches aren't worthwhile?

Re:We need web caches

[ender81b]

What about specifing a really, really short 'life' for it to be cached.

I know in squid you can specifiy that anything over x minutes is to be discarded. Then again I'm not sure if squid can handle an entire ISP's worth of traffic (probably though).

That would solve your problem with caching illegal content if it was just discarded after, oh, say 90 minutes.

Re:We need web caches

[Sloppy]

Have you considered maybe some sort of whitelist cache? For example, when 100 people are all downloading a RedHat ISO or Microsoft service pack, surely you'd like to move it across your outside pipe only once, and there isn't much chance that RedHat and Microsoft are distributing kiddie porn.

Or is "isn't much chance" not good enough? Argh.

This sucks. If you're protected from liability on how your wires are used, that protection should extend to caches. It's just common sense. Sounds like we've got a fucked up law or something.

Re:We need web caches

[Salamander]

And where is the law or precedent that would be used to prosecute an ISP for caching things that are illegal for other than copyright reasons? Common-carrier laws have been around for a long time, and have been used many times to shield the people who provide a communications medium from culpability for what is transmitted over that medium. The reason I mentioned the DMCA is that, in the specific case of copyrighted material, it attempts to supplant existing law and precedent; in cases other than copyright, that law and precedent is still valid and likely to protect the ISP as a common carrier.

Then again, law moves in mysterious ways. Are you aware of a recent case that would lead to a different conclusion?

Re:We need web caches

[Cato]

There is a lot of support in the HTTP caching headers for controlling the timeliness of cached copies - you can say 'keep this for just 5 minutes then check for a new version'. The trouble is that web developers don't know much about caching.

Of course, your browser caches already, and can easily be more aggressive than proxy caches, so eliminating proxy caches is not going to solve anything. If someone using a web-based application is so clueless they don't know about the Refresh button, AND the application developer managed to do a cacheable dynamic page, that is just lack of training, or using the wrong tools.

Defeating caching is very easy - most CGI-based sites are not cacheable, and many commercial sites turn caching off in order to see all hits.

Re:We need web caches

[greenrd]

Exactly. It doesn't make any logical sense whatsoever. It's completely absurd. It was obviously dreamt up by a lawyer with no clue about how email, Usenet etc. work.

Not that new

[RollingThunder]

"This 4000% overhead is annoying but tolerable on lightly loaded networks."

(From RFC896)

Of course, that's talking about bytes of overhead vs bytes of real data - there would be much less than 4000 packets per packet containing real data.

...but it'll bite them in the end.

[The Fanta Menace]

The more they cap usage, the less people will use (obviously). Then content providers such as streaming radio stations will start to drop off as it becomes more expensive for users to access them.

After that it becomes a vicious circle, with fewer content providers, there's no reason for users to keep their service. Then the ISPs go broke.

Take a look at the Australian example. Almost all broadband providers have a 3Gb monthly cap. The ABC has just started an internet-only radio station, but I really wonder why. It wouldn't take too many days of listening to it for a user to totally max out their cap. I predict the station will be closed due to lack of interest, within a year.

xolox was a huge offender

[teamhasnoi]

Back when xolox (for windows) was in ver 1.2, it would requery *forever*. Althogh this worked great for getting results, I am sure it killed the network.

They have seemingly 'fixed' this in the new release, but it now has banner ads and popups all through it. Ug.

It's pretty good, even though they have some catching up to do. (They went down for awhile for fear of getting sued alá napster.)

hrmm yeah i guess so

[digitalsushi]

as an ISP i can say that we make our money by a gamble that people use X amount of bandwidth. p2p breaks our precious little ratio of what we expect and what we need.

the geek it me though, says "waaa" and that things that dont evolve, die. and the things that dont die. p2p pushes the envelope right now, but all that encourages is more network growth. just think of p2p as those pains you had in your legs when you were 14. sure, it may not be the most efficient thing in the world, but the underlaying infrastructure has to take that into account, or get out of the way for one that can.

Re:hrmm yeah i guess so

[Fastolfe]

p2p breaks our precious little ratio of what we expect and what we need

Uhh, yah, except this is how they're determining how much they can charge you. If the ratio becomes permanently skewed, the way they "evolve" as you put it is to simply skew their prices to compensate. Though your end user connections may be effectively "unlimited", someone upstream pays for the bandwidth by how much data gets transferred. I guarantee the costs will filter down.

So as a business, what would you do? Raise your rates for all "unlimited" customers? Create a new class of DSL customer with a lower bandwidth cap and re-figure the ratio? Block P2P activity entirely? Write into the end user contract some soft usage caps and go after the top 1% of bandwidth consumers? All of the above?

I don't really think P2P is going to drive growth (i.e. more bandwidth for less cost) any more aggressively than the growth we're already seeing. I just think it's going to annoy ISP's and make them re-think some of their "unlimited" bandwidth plans.

Burned our office once

[RollingThunder]

The office I'm at used to have a contract with a monthly cap - a mere 20GB, with fairly hefty per-GB fees after that.

One Monday morning, I came in, and glanced at the MRTG graphs over the weekend. Keeripes! Somebody had been pushing data at about 250Kbps from Friday night until about 6 PM on Sunday, sustained.

I did a quick calculation, and then informed the bosses that we were going to be paying a lot more than usual this month, and asked if they wanted me to find out why. Of course they did.

Turned out it was one of said managers. He fired up Limewire, grabbed something on Friday, and forgot to shut it off. Seeing our nice low-latency, high capacity link (E10 or thereabouts, just with a really low traffic cap), it went supernode... and we paid about twice the usual for it.

Re:Burned our office once

[afidel]

We had the same problem. Although our multi-T1 connection is not metered we did have it brought to its knees for almost a day when one person set kazaa to be a supernode. I got a very angry call from the wanops people to go tell this user to knock it off and that they would recomend disciplinary action if it happened again.

Re:Burned our office once

[mr_exit]

we had someone visit a site with a really badly set up ad system, it kept refreshing the image every 14 mili seconds..
he managed to pull 8 gig overnight... and at 8c a Meg that gets expensive

nope, he's not insane.

[EnderWiggnz]

go look at the html code from google - notice how they abbreviate every object name to ONE letter in the interest of bandwidth.

i'm sorry that you learned how to code sloppily, and are bitching about streamlining code for efficiency, and cost savings.

most of us dont need the damn hungarian notation that MS has spreads like gospel truth. It makes for unreadable names that convey less meaning that a nice clear variable name.

oh - and i know when to use a goto to streamline code, too :-)

Re:nope, he's not insane.

[EvanED]

>>most of us dont need the damn hungarian notation that MS has spreads like gospel truth

Why said anything about that? And besides, MS now discourages its use.

>>It makes for unreadable names that convey less meaning that a nice clear variable name.

Which 'fn' is not but 'FirstName' is.

Now, if you have a dynamically generated page, you could use constants that are set to short stuff like 'fn'. Less code to be transmitted while still keeping most of the readability of the original code. If you discover a bug, temporarily switch to a different set of constants ('FirstName' instead of 'fn') until you sort it out so the resultant HTML is more readable. (Same goes with whitespace: Make a constant ENDL or NEWLINE that is set to '\n' while debugging, then changes to '' for production.)

Re:nope, he's not insane.

[Tony Hoyle]

>>>most of us dont need the damn hungarian notation that MS has spreads like gospel truth

>Who said anything about that? And besides, MS now discourages its use.

Got a reference for that? It's shoved down our throats whether we like it or not simply because 'it's the MS way'. It's damned impossible to debug something called 'lpszglname' especially when it isn't even a string any more because it was changed years ago...

Re:nope, he's not insane.

[notsoanonymouscoward]

I wouldn't consider most of the programmers i know to be human. we all have lexers, parsers, and symbol tables built into our brains for various programming languages. problem solved.

Re:nope, he's not insane.

[kaphka]

Got a reference for that?

RTFM.

• "
• Do not prefix parameter names with Hungarian type notation.
• "

Re:nope, he's not insane.

[CyberKnet]

go look at the html code from google - notice how they abbreviate every object name to ONE letter in the interest of bandwidth

That's all very well if you only have a low number of objects on each page, but when the objects move into the hundreds you're likely to run into a little more trouble whilst keeping track of all said objects. After all, not every application is a search engine.

i'm sorry that you learned how to code sloppily, and are bitching about streamlining code for efficiency, and cost savings.

That's highly subjective, at best.

There are benefits other than bandwidth saving to take into consideration. I.E. The time it takes a new coder to become familiar with a new (Web) application, ease of documentation, and debugging.

most of us dont need the damn hungarian notation that MS has spreads like gospel truth. It makes for unreadable names that convey less meaning that a nice clear variable name.

Explain to me exactly how hungarian notation makes for unreadable names that convey less meaning?

Hungarian notation would add scope and type (depending on the strain) information (in lower case) before your "Clear Variable Name" (in title case), and not take anything away from it at all...

I've personally have never seen Microsoft spread hungarian notation as gospel truth. I find this most evident by the fact that Microsoft documentation / code samples in MSDN sways from using it partially, completely or not at all.

However ... despite all this, if you have any factual sources or meaningful (read: backed up) statistics to confirm what is so far just libel and slander, please do post again...

This has been known for a long time...

[fortinbras47]

With gnutella, QueryHit packets can make up as little as 1% of traffic (by numbers of packets, not size) while Ping and Pong packets can be well over 50% of packets. Check out this article to see more detail.

Gnutella is not one of the more advanced protocols, but most of it's problems are present at varying levels in other p2p systems. It's not really surprising that P2P software which spends so much time trying to connect to computers, connect to a computer to start a download etc... and search in a geometric spiderring fashion are quite inefficient.

Re:This has been known for a long time...

[krmt]

That article is from March 2001, which is very out of date. Since then, a lot of development has occurred, most notably the development of supernodes by Limewire, gnucleus, et al. In addition, a new ping and pong scheme has been proposed.

Bandwidth Division

[T-Kir]

Damn dialup crappyness, retype whole message time :-(

Question:

Isn't the excuse of capping broadband connections a moot point, because the general broadband thing is that it is a shared resource? So X ISP saying that 1% of their users are hogging 60-70% of the available bandwidth, then they use that to say 'Right, we're raising prices', in the sense that if there is any load balancing, then the other 99% of users would be able to level up the bandwidth if they needed more, so it divides up (theoretically equally)??

Although I'm spoilt rotten living with my brother in CT, cos the Optimum Online connection (around 5Mbit at it's fastest) has been no trouble at all, damn UK rural 'broadband' (or lack of). Oh well, I suppose I don't have many new Farscape eps to download when they come out :-(.

Just my 2 pence.

Re:Bandwidth Division

[Alan Shutko]

Maybe, if you're a Nielson family. If you aren't, how you watch a show will have no impact

X.25 hax0rz

[drwho]

Back in the heyday of "X.25" networks, there were a lot of illegitimate users. There was inadequate technology to protect and track.

It is rumored that there are accounts on public x.25 networks, belonging to large corporations, that have worked for over 13 years.

The Real issue...

[Quasar1999]

ISPs are putting bandwidth caps on accounts because they see it as a source of revenue. Plain and simple. The crap about how 5% of the users use 95% of the bandwidth is really starting to piss me off... they advertised always on, unlimited bandwidth when I signed up, and now they have enough customers used to the speed, so they essentially upped the price (just like soup companies reduced the size of their cans of soup, but kept the price the same, if you want more, buy a larger can...) if you want more bandwidth, upgrade your package, or better yet, pay $7.95 a GB/Month over our generous 3 GB/month...

Isn't there a law against doing this sorta crap? They said always on, unlimited bandwidth... now they're charging through the nose, claiming crappy stats on usage, and blaiming it on p2p networks... I can't even download my legitimate MSDN ISO images without going over my monthly bandwidth limit, let alone actually doing anything else on the net...

End rant... :P

Re:The Real issue...

[jbolden]

And during the time they advertised unlimited bandwidth they delivered it. They didn't advertise "unlimited bandwidth for life never to be revoked". And I know for a fact that MSDN will send you CDs monthly so there are some gigs you can save right there.

Anyway there is no magic bullet here. A small number of customers are using way more bandwidth then they are paying for. The ISPs can either raise prices for everyone or get those customers to pay more.

As for $8 per gig, that is insane and I'd talk to the PUC about it; that's well over their costs.

Re:so lots of applications...

[Kintanon]

Maybe when ALL THAT BANDWIDTH I'll finally be able to get real TV streaming to me? I mean, sputnik7.com is great, but all they have is anime and their selection doesn't change very often...
Where can I go to get streaming Comedy Central or SciFi or Cartoon Network or something? I'd pay a few bucks per channel for that, if the quality was excellent.

Kintanon

Re:Text compresses

[EnderWiggnz]

no, there arent better solutions.

as i stated earlier, go to google and look at the source - one letter field names, and no line breaks.

why? TO SAVE BANDWIDTH.

every byte is sacred... every byte is great...

and if a byte is wasted, CFO's get quite irate...

Re:AT&T Blocking P2P?

[martissimo]

there is a big thread thats been going about this situation over at dslreports.

if you dig a page or two back into the thread quite a few users seemed to have success by using some http tunnelling software so they were on a non throttled port. might just find an answer to your problem in there if ya look

Lies! All lies!

[joebp]

The sys admin realized that if he just reduced the frequency of keep-alives, he could shave something like 10% off the monthly bill.

Translated: he cut down on pr0n.

Comcast where I live..

[stratjakt]

..is airing this commercial of goofy testimonials for their broadband cable service. A kid says "Ever been in the belly of a whale? I have", another guy goes "I go to the moon and back twice a day", etc.. etc..

Now, one of them has some guy say "I collected everything Mozart ever did... In 10 minutes!"

To me that's comes through loud and clear as "*wink* *wink* *nudge* *nudge* napster(etc)!"

I would say p2p is the driving force behind non-geeks getting broadband. They don't need it for e-mail, or casual web-surfing. They don't play games, but I know many people eager for an alternative to the bland junk on the radio. (Plus due to geography, radio reception is poor here)

Same thing with the 'work from home' bunk they promote, and yet block VPN connections.

It's like dangling a carrot in front of a mule to get him to move, and he stupidly chases it not realising he'll never reach it. It works fine in cartoons, but eventually the mule becomes frustrated, kicks you, and refuses to move at all.

Someone is smart enough to figure a way to give out the bandwidth and make money at the same time. And, it won't be a monopoly. Maybe 802.11 will be our savior?

Re:Comcast where I live..

[xenoweeno]

Now, one of them has some guy say "I collected everything Mozart ever did... In 10 minutes!"

To me that's comes through loud and clear as "*wink* *wink* *nudge* *nudge* napster(etc)!"

One bit of bad news: Mozart's work is in the public domain, and has been for, ah, quite a long time.

Re:Comcast where I live..

[ChadN]

But specific recordings of Mozart's work may be copyrighted (and most likely are, if it came from a CD originally).

Re:Comcast where I live..

[Jeppe Salvesen]

I would say p2p is the driving force behind non-geeks getting broadband. They don't need it for e-mail, or casual web-surfing. Hardly. Have you tried suring on a 56k lately? The web sites are becoming ever more image-laden and bloated. Image swapping from their digital cameras. Plus - the fact that broadband doesn't block your phone line like a plain old modem.

When the MPAA gets their heads out of their asses and start providing decent pay-per-view and/or pay-per-download solutions, broadband will become ever more popular.

Quite Sane really

[ACNiel]

I couldn't disagree more.

I just got into an amazingly poorly written program after about a year, and was bewildered by the names, and what they really meant. And it was my code. And yet, I couldn't disagree with you more.

The streamlining that was discussed by the parent isn't for the sake of the coder, it is for the sake of the user. Mostly your argument is founded in long names really don't hurt anything. And if that is true, than long, descriptive names do their job. Here is a prime example of where they do make a difference. Here the variable names (for variables, and even javascript, or vb script embedded in a page) could make a tremendous amount of difference. And that is wht you would be stream lining for.

As for superfluous naming, well that can be just as bad, and unreadable as short names. Addled is addled, and you can use short (maybe more than 3, this isn't RPG afterall) descriptive names, without having to type an entire sentence.

junk1, junk2, junk3 will never be a good idea, but if your form has 4 variables, and you name them
FNm, LNm, MI, and Age, I don't think anyone will be confused.

I can totally understand.

[Arcaeris]

I can totally understand the limitations of bandwidth in the face of 2p2 software.

I was in my first year of college (living in the dorms) when Napster became popular. That same year, they banned it from all campus computers. The IT guys here said that of the estimated 7200 dorm room computers on campus, a minimum of 6500 were running Napster at any given time. They were forced to ban it because the bandwidth usage was taking away from vital staff/faculty related web-based tools and network services that needed to be maintained. In fact, nothing else could be run on the network.

Now Napster's gone, and I haven't lived on campus since Kazaa and such became popular. I'm pretty sure I know how they're dealing with it.

If one university had to do it, then imagine what the average cable/DSL provider has to deal with. Granted, they don't have as much essential network stuff.

Re:I can totally understand.

[stratjakt]

Yes but, (if your University was like mine), you don't pay for it. Access to the universities private network was a privelege of living on campus. They were bound by a fixed budget that came out of our tuitions/res fees and had to accomodate everyone. Our house, our rules..

I pay for my cable powered internet. I don't see their right to tell me what I can and can't do with it, it was part of no contract I signed, save some ambiguous crap about removing "abusive" users at their discretion.

I made another post in this forum about how they use p2p and VPN as incentives to sell the service. Bait and switch.

The business model in short (and not a lame SP troll):

Split a 10mbit pipe over 1000 users. Most only know how to read e-mail and read dilbert cartoons so they'll never notice we oversold ourselves. Kick the few that will off, cite bandwidth abuse as the reason. (How you 'abuse' something they sold you unlimited access to still escapes me)

The 'stupid sheep' they counted on forking 40-100 bucks a month for something they'd never use, found something to use it for.

Re:so lots of applications...

[duren686]

It would be nice from a purely "would be nice" point of view, but in effect you would get charged less for using more bandwidth, which wouldn't fly with anyone who provides bandwidth.

Re:Where did my bandwidth go?

[linuxwrangler]

Aha - now I know where it went! Microsoft Word must have autocorrected everyone's 10MBs to 10Mbs.

Streaming Real TV

[silas_moeckel]

As an engineer that made a network to do that that tanked (Nothing like lies from sales people) it's possible 512 kbit a sec looks pretty nice but the bandwith costs on the sending end are about 50 bucks a month before servers people etc (thats sending all month) why because it's all unicast because NO ISP wants mcast working outside of itself they dont know how to bill for it. A satalite at 500 an hour is much cheaper than delivering over the internet and inherently multicast. I wonder when somebody will come up with a multicast service that is delivered to a majority of ISP's.

Re:Four? Or Millions?

[MxTxL]

The 4 machines you are connected to perform your search then pass it on to the computers they're connected to and so on... i think there is a setting you can change, but generally searches go something like 8 levels deep.

I know where a lot of bandwidth goes...

[NineNine]

I know where at least a few hundred gig a week of bandwidth goes to...

sounds like

[Gavitron_zero]

the big hog on network bandwidth is TCP/IP...big surprise there.

Wow

[Salamander]

The article itself was kind of ho-hum, but the following part of the Slashdot intro caught my attention:

I doubt that developers of those free p2p applications have gave much thought to efficiency.

Again...wow. One would need to search far and wide, even on Slashdot, to find another example of such absolutely astonishing cluelessness. Timothy has obviously never talked to a P2P developer in his life. Sometimes it seems like efficiency is just about the only thing P2P developers think about, unless someone's on a security/anonymity rant. Little things like robustness or usability get short shrift because so much of the focus is on efficiency. Hundreds of papers have been written about the bandwidth-efficiency of various P2P networks - especially Gnutella, which everyone who knows anything knows is "worst of breed" when it comes to broadcasting searches.

It's unfortunate that the most popular P2P networks seem to be the least efficient ones, and doubly unfortunate that so many vendors bundle spyware with their P2P clients, but to say that P2P developers don't give much thought to efficiency is absurd. They give a lot more thought to efficiency than Slashdot editors give to accuracy, that's for damn sure.

Re:Wow

[Have Blue]

Hundreds of papers have been written about the bandwidth-efficiency of various P2P networks

I beg to differ. Hundreds of papers have been written on the bandwidth-inefficiency of P2P networks, and the fact that they are still being written is evidence that they are still inefficient.

Re:Wow

[Salamander]

Nonetheless, people have obviously thought about it a lot, and timothy is still full of crap for saying otherwise. :-P

Re:Wow

[Salamander]

I think the infoAnarchy Wiki covers it better than I could.

Re:Wow

[greenrd]

No, he didn't write any of the italic text. Can you read English? "An anonymous submitter writes" sounds unambiguous enough to me.

compression

[Twillerror]

Remeber back in the good ol' modem days. I remember getting 10 k a second on some transfer even with a 56.6.

If a P2P network protocol is text based, say like XML, it should compress pretty well and keep some of this extra bandwith down.

If HTTP would actually support compression natively we could save tons of bandwith in those HTML transfers. The page I'm typing this comment on is 11.1 k. zipped it is 3.5, and I think I have fast compression on. I'm sure the main slashdot page would save even more. Slashdot could litterally save megs a day.

It would simply be a matter of Apache and IIS supporting it. And maybe a new GETC command in HTTP that works the same. The browser would ask if the server supports it, and then go from there. Or try it and if it failed, try it normally. Apache or IIS would be smart enough to not try and compress JPEG, GIF, and other pre-compressed files.

Everything from FTP to SMTP could save a little here and there, which adds up quick.

Perhaps the real answer is to write it into the next version of TCP and have it hardware accelerated.

Re:compression

[bedessen]

The page I'm typing this comment on is 11.1 k. zipped it is 3.5, and I think I have fast compression on. I'm sure the main slashdot page would save even more. Slashdot could litterally save megs a day.

It's not a free lunch, you have to consider the resulting increase in server CPU load. It's probably not an issue for low traffic sites but it's definitely a concern for sites like slashdot.

Re:compression

[ShaunC]

It would simply be a matter of Apache and IIS supporting it

Apache does support it, it's called mod_gzip and Slashdot already uses it. The IIS equivalent (sort of) is called PipeBoost.

Shaun

Re:compression

[cduffy]

If a P2P network protocol is text based, say like XML, it should compress pretty well and keep some of this extra bandwith down.

In terms of compressed size, there's no (theoretical) advantage to a compressed XML stream over a compressed binary stream. The reason that XML compresses so well is that there's obvious redundancy that's easy to "squeeze out". A good binary protocol will have less easily removed redundancy -- so it'll be smaller to start with, but won't compress as well. If both represent the same protocol, in theory they should (post-compression) come down only to the same size.

Also, as mentioned, mod_gzip is available and used today.

Once again...

[_Knots]

I say P2P mesh networks (ala Gnutella) need to have intelligent meshing algorithms so that the network tries to minimize the number of mesh links crossing a given physical uplink or a given backbone segment.

Such a scheme would return optimized search results because your net neighbors would know of your query before somebody on the other side of an uplink (and, as there is less routing between you, can transfer files faster in theory).

On top of that, with such a router-aware network the wasted bandwith of broadcast packets multiply crossing a given line due to reflection by peers on the other side would be virtually gone once the network became aware of the layout - ideally each node wouldn't have to learn but could get some kind of topological information from a node it connected to ("You are in the same /x block as a.b.c.d - please connect to that node and drop this connection") or maybe even ask the remote node to preform some kind of query for it ("who wants a.b.c.e, because I don't?"). Our current "host caches" like router.limewire.com could gain some intelligence for whom they introduced to whom.

Instead of capping upload and download capacities as much as done now, perhaps those limits should be relaxed but a P2P "introduction" program installed on the ISP's router so that clients behind the firewall mesh with each other before a few of them send meshing links spanning the uplink.

Yes, downloads will still follow the usual TCP/IP pathways - which we presume are most efficient already. But the broadcast discovery packets which now ricochet around the network would, with an intelligent meshing algorithm, span as few uplinks as possible to query hosts as network-close as possible. All in all this would reduce traffic.

Somebody want to blow holes in this for me?

--Knots;

Re:Once again...

[Salamander]

Actually, it's a really good idea and IMO an area where I believe not enough work has been done. Which is not to say no work has been done. There are several projects aimed at finding the best way to determine network distance, and several more that seek to use that information to create more optimal connection topologies. I don't have my links to that stuff handy right now, but if you send me email I should be able to dig them up.

Part of the problem is that many P2P networks have dependencies on some model or other of the higher-level abstract topology through which they route search queries, and it can be difficult to map (for example) a hypercube onto the actual IP-network topology. Lacking a good solution to that problem of mapping one topology onto another, many P2P developers punt; they try to minimize hops through the overlay network, and vaguely hope that by doing so they'll make up for the extra hops through the underlying network. In many cases it even seems to work, because the search algorithms that operate on the higher-level topology can be extremely efficient.

Nonetheless, if someone could figure out a way to reconcile those advanced search algorithms with a more "reality-based" topology, that would be great. If you think you have ideas on how to do that, by all means explore them. The more the merrier.

Re:Once again...

[PureFiction]

An excellent idea in theory. Now try and code such a beast :-)

Trying to build efficient structure into peer networks is like building a house of sand. They are extremely volatile.

You need to take into account numerous factors, like many orders of magnitude in bandwidth capacity between peers, NAT and unNAT'ed hosts, high churn rates, volatile peer groups, etc, etc.

Most people who have tried to overlay fragile yet elegant topologies on top of peer networks have seen them crumble under volatile real world scenarios.

This is not to say it is impossible, but that it is much harder to implement such a network in today's internet environment than it first appears.

Re:Once again...

[_Knots]

Nat vs unNAT: Treat the NAT as an uplink that we should try to limit connections through.

High churn rates / volatile peer groups: yes, there's a lot of changeover in everything, but I'd wager that copying intelligence on connection ("Here's everything I know about the network around me") would endow newcomers with a good base to start off with.

There's nothing fragile about this topology: it's a runtime dynamic mesh topology - exactly like Gnutella's now. The sole difference is that groups of peers would try to actually group themselves by network-proximity (probably IP range, or for things like Road Runner or at a university, reverse DNS mappings might help). Yes, it might take some more effort from users to specify how to identify members of their local group and get it right. But there are surely some decent ways (IP ranges, as stated) of getting it *usually* right.

It shouldn't hurt the network - it should be an option to turn it off, it should turn itself off if it detects its being unhelpful. The incentive towards the users (inside a university or on a local cable loop) would be much faster downloads due to less routing overhead.

Really you could think of this as ultrapeers agreeing amongst themselves as to which of them will actually route outside a group.

--Knots;

Re:Once again...

[PureFiction]

Nat vs unNAT: Treat the NAT as an uplink that we should try to limit connections through.

What do you mean? NAT users only upload? NAT users only perform outgoing connections? There is a lot of documentation on the experiences developers have had when dealing with NAT. The majority of broadband users are behind NAT, and so using their connections effectively is critical to making a functional network.

NAT also comes into play not only for creating connections, but also maintaining connection state in the translation tables. You cannot even assume connections will stay open for arbitrary preiods of time or else you will inevitably start losing connections...

I'd wager that copying intelligence on connection ("Here's everything I know about the network around me") would endow newcomers with a good base to start off with.

That sounds like a good way to eat up bandwidth. One of the characteristics of most mesh routing protocols is that they do not lower overhead, they simply transition it from the overlay broadcast part of a protocol to the mesh maintenance layer. For example, maintaining optimal mesh topologies consumes a lot more bandwidth than the simple connect to a peer and send a query, pass it on style. So in this manner you are using much less bandwidth for queries, but much more bandwidth maintaining the mesh network over which queries are sent.

There's nothing fragile about this topology: it's a runtime dynamic mesh topology - exactly like Gnutella's now.

Gnutella is incredibly fragile. Gnutella sucks. Gnutella is incredibly easy to cripple with DoS attacks that flood queries into the network (among many other possible methods). So I do not think this is a valid comparison.

If you are relying on volatile peers to forward data through a mesh network you also need to provide robust recovery that does not consume extravagent amounts of bandwidth. Again, this is much harder than you think.

The sole difference is that groups of peers would try to actually group themselves by network-proximity

There is a lot of good info about subtle side effects of assuming Really you could think of this as ultrapeers agreeing amongst themselves as to which of them will actually route outside a group.

Then you are talking about a super node architecture, which is used in a number of peer networks and is much more efficient than broadcast & forward protocols like gnutella. But this is also a far cry from a true mesh routing protocol optimized for efficient bandwidth utilization.

Re:Once again...

[_Knots]

What do you mean?

Well, on my DSL line we were given 10.n.n.n addresses and the NATting was done at some hop down the line - we, two nodes behind the NATting router, could talk to each other as 10.n.n.n as well as our true IPs. Clearly doing the former is much more efficient as we don't have to span the router twice for nothing (in effect - yes, we're probably just hitting its routing tables once, but still, the routers before it probably could have handled the 10.n.n.n instead of the other form).

That sounds like a good way to eat up bandwidth.

Yes, I am aware that it's a risky proposition. Though each node knows very little - its position, its neighbors, and whether it is acting as a ultrapeer for its group. Maybe a little more state information (all members of the group?) but not much. And intelligence would only migrate *after* acceptance into the group by at least one member and/or one ultrapeer member (yes, there can be many).

Gnutella is incredibly fragile.

Well, in some ways yes, but in some ways no. It tries its damnest to stay around and with the ultrapeering technology already present, in addition to LimeWire (and presumably others?) ability to drop most DoS-style packets, it's as much an arms race as the internet in general.

Where Gnutella suffers, though, is its "broadcast to the world" idea. With an intelligent meshing protocol, one could concievably implement a much more efficient "local" routing scheme through some clever manipulation of TTLs inside the mesh and/or some more routing information (at the very least "drop this if you've seen it before," which I think Gnutella does already). This is all optional and maybe not even worth persuing. But the intelligent meshing protocol would prevent a DoS on Gnutella from scaling to a DoS on the backbone segments, ISPs, etc. involved because it would try as hard as possible to have as few reflective connections across a given link.

A supernode architecture, sure. And why not? Gnutella v0.6 already is one, yet its still possibly best described as a mesh with supernodes at the fringes in places.

Alright, I'm really tired. So if none of this makes sense, assume it's my fault.

--Knots;

Re:Once again...

[PureFiction]

Well, on my DSL line we were given 10.n.n.n addresses ...

I hope this was all your internal subnet. No broadband connections that I am aware of provision with internal IP's (except for a snafu at att broadband which transposed internal modem IP's with external client IP's)

In the real world everyone gets a public IP from their provider, and the NAT is usually a linksys cable modem / DSL router. In rare cases you might see linux or bsd systems used as NAT gateways.

You can work around NAT problems, but it is not exactly trivial for TCP.

in addition to LimeWire (and presumably others?) ability to drop most DoS-style packets, it's as much an arms race as the internet in general.

It is vastly different because with a small amount of DoS traffic injected into a gnutella network you can waste very large amounts of bandwidth overall. For most internet DoS attacks you need at least as much traffic to DoS as the pain imposed. I.e. it is a linear relationship, whereas DoS attacks in Gnutella are exponential. The gnutella clients have come a long way, but they will never be perfect because the protocol itself is flawed, and prevents robust resistance to DoS attacks (how can you tell if a given query packet is valid or not? You can't if it is done with any degree of stealth)

A supernode architecture, sure. And why not?

No reason at all. In fact, the super peer networks currently appear to have the optimal balance of decetralization and centralization mixed into one. The limited centralization improves query efficiency and reduces bandwidth, while the decentralized nature of ad-hoc super peer delegation allows for much greater scalability with very little dedicated infrastructure.

These will probably be the "sweet spot" topology/architecture for a number of years...

But this is a good bit different than a true mesh topology designed to optimize routes based on routing locality, etc, that you seemed to mention in the very first post. I suppose you could consider a super peer architecture as something akin to a mesh topology, but it is a stretch.

Alright, I'm really tired. So if none of this makes sense, assume it's my fault.

Peer networks are deceivingly simple at first glance, and wickedly complicated in the details. It all makes sense, but that doesn't mean it cant be confusing at times. :-)

Re:Once again...

[PureFiction]

How do you engineer a data transport protocol that prevents a DoS?

You can never eliminate it completely (anyone can still flood you off the network given enough bandwidth)

But what you can do is make it difficult. There are a few characteristics that should be implemented.

1. Make any kind of DoS attack require at least as much bandwidth on the DoS source as the bandwidth it interrupts. Meaning, the only usefull DoS is a flood, which no one can protect against anyway. In gnutella, a DoS can have a huge impact on wasted bandwidth by using very little sending bandwidth (the network effect of relaying bogus queries for example).

2. Make communication direct, so that all nodes know whom a given packet is coming from. In gnutella, packets are forwarded, and you never know if it is legitimate or not. Is this person malicious? Or are they relaying data from a peer downstream that is malicious? Having communication direct allowsd you to terminate connections to those you do not want to communicate with.

Relaying nodes wouldn't have to compute this because ...

Any node which relays traffic is a fundamental weakness. I do not like overlay relay networks at all for this reason. You never know (really) where a packet originated from, and this makes it very hard to protect against certain kinds of attacks (from well connected peers injecting bogus traffic)

Where the all bandwith goes? It is easy

[tandr]

Ready?

Slashdot.org and traffic redirected from its links.

I wonder

[A5un]

How much of current network traffic (data/voice) are really just protocol? I mean all the way down to physical layer (yes, the 1 and 0's). Seems like every layer of abstraction tags a protocol header on to the real payload. Is there any study done in this? I won't be surprised if more than 50% of network traffic are just protocols (IP headers, TCP signals, SONET header or even CRC bits).

Testing bandwidth does it too...

[kawika]

I run a site that has a bandwidth test, and there are people who run big multi-megabyte tests every hour or less to "see if there are any problems" in their connection. Multiply this times lots of people and lots of bandwidth test sites and I'm convinced that a lot of the bandwidth on the Internet is wasted in testing connection speed!

If this were legal, it could be far more efficient

[Animats]

If this "file-sharing" stuff were legal, it would be easy to do it efficiently. Each new song would go out on USENET into some binaries group, traverse each link no more than once, and reside in a nearby news server. A modest-sized disk drive per ISP could hold MP3 versions of the entire catalog of popular music. No problem.

So that's the standard with which P2P networks must be compared on an efficiency basis. It's not looking good right now. Current P2P architectures scale badly. This is well known in the networking community, but not widely realized by end users.

A big problem is that it's hard for a program to tell "how far away", in some sense, another IP address is. You can measure latency and bandwidth, but those are only hints. If many programs are doing this, the overall result is suboptimal. There's been considerable work on efficient self-organizing networks, mainly for military use. That's where to look for better architectures.

Gnutella IS inefficient

[PotatoError]

If you take a look at the spec for the Gnutella Protocol, you will understand where all this "extra" traffic is coming from.

I've been messing about with gnutella on and off for about 3 months now, I hope to make an open source functional client eventually. It's quite an interesting area because there is so much work to be done on security and efficiency.

The only problem is that P2P networks are never going to be as efficient as centralised server networks and certainly never as fast. I suppose a cynic (like me!) could blame the entertainment industry for forcing out server based file sharing networks.

But I believe the death of server based file sharing is a good thing. The bad side of the server-client model is that it can (and usually is) controlled by an authority and its security is often obscurity based (the obsure bit being hiden on the server). Peer to Peer networks however, offer total anonominity as well as giving users access to the whole component.

Peer to Peer networks are the next step in securing freedom of information on the internet and preventing government control.

It's when Peer to Peer mobile phone networks are produced that things will really get interesting....

Not all P2P networks are created equal

[0x0d0a]

Freenet is more efficient than, say, the Web would be. Those DiVXes don't need to cross your ISPs downstream connection at all.

Gnutella is noisy, but that's not the fault of the creators. Blame the RIAA -- the first P2P applications were centralized. If you can give up the requirement that there be no single, trusted point of failure, it's much easier to make an efficient network. They attacked Napster, and now people have moved to mostly less efficient approaches.

AOL is what makes us afraid

[sdjunky]

AOL is what makes us afraid since it shows that content providers don't care about refresh dates or content but rather cutting the bandwidth. Even if it means compressing images and caching a page for extended periods of time.

Efficient search in peer networks

[PureFiction]

I doubt that developers of those free p2p applications have gave much thought to efficiency

Some of us have. Search is much of the bandwidth in peer networks is wasted (downloads are downloads, but search can eat up a lot of bandwidth for little return)

There are some efficient, effective peer network search apps currently in development. Hopefully we can eventually leave gnutella and kazaa in the past and move on to more open, efficient networks...

PNG currently works only for still GIF images

[yerricde]

"Web browsers don't support" is not an excuse anymore

As you correctly point out, that's true for non-animated images. However, what do you propose for simple animations? What if animated .gif is the only way I can get advertisers to buy space on my site? Is there a way to hold off Unisys for the last nine months of the life of U.S. Patent 4,558,302? Mozilla (and Netscape 7) is the only popular browser to support Multiple-image Network Graphics, the animated extension to PNG and free alternative to animated GIF images. Excluding IE users is not an option. Or should I try to find (or write) a tool to convert animated .gif to .swf?

unless you have alpha (specifically multiple levels of transparency), which some of the older browsers didn't handle so well.

Even IE 6 doesn't handle alpha very well. (Mozilla does.) However, any PNG image converted from a still GIF image will work fine.

The good part of the DMCA

[yerricde]

Unfortunately, if we cache and then illegal material is downloaded, we can be held responsible for that material.

Not necessarily. A rider on the DMCA allows service providers in the United States to cache web pages, provided that they meet certain criteria (which are easy with HTTP/1.1) and designate one of their employees as a DMCA agent. Read more on this page.

Re:That's "MUTE" point not "moot" point

[Rosonowski]

Actually, there is such a word as "moot" if you look into it.

Google tells us there is a venture capital company
And a game by that name; oddly on the "nuances of the english language. (don't even complain about mine, I won't claim to have perfect grammar, since english is an evolving language.)

I also have heard it used in reference to a gathering of were-creatures.

Not to mention, the last name "Moot" is a very real one indeed.

So yes, the word "Moot" does exist, even if people use it wrong.

duh

[cr@ckwhore]

I might be missing the point, but why complain about cost? If the cost of a big private line is a problem, then you should consider VPN.

Two things..

[Restil]

First off all, the P2P networks by design will generate far more traffic than necessary. Necessary, being of course a single set of central servers that collect data from the entire network and serve out that data ONCE to anyone requesting it. However, Napster as we all know died a painful death because there was a single point of failure. Kazaa, gnutella, and others have no head you can cut off. Even if the company that sponsors Kazaa were to be sued/prosecuted into oblivion, the network would remain. The downside, of course, is an excessive amount of unnecessary traffic.

The second big problem is the fact that as far as I can tell, none of the P2P networks take advantage of the teired nature of the internet, attempting to search local networks first, and searching further ONLY when something can't be found closer. Bandwidth is always more scarce (and therefore more expensive) the closer you reach for the backbone. Any effort to keep the traffic within the local network of the ISP costs THEM less, which means they would be far more willing to promote those types of networks, or at the very least not attempt to restrict them.

The network admins for universities were especially outspoken against Napster at the height of that craze, since that single program was consuming all the upstream bandwidth, where there is a DAMN good chance that with a student population in the tens of thousands, there's probably a 99% chance that anything a student was searching for could be found somewhere on the university network, which typically has much larger pipes than the internet upstream.

-Restil

Re:Two things..

[HamNRye]

MY bandwidth just went to turning the plasma ball off... Ho Hum.

~Ham

"always on" bigger problem when sharing

[Trepidity]

At my school at least, the biggest use of bandwidth seems to be people who leave filesharing programs on all the time, which ends up sharing their download directories by default, even if they haven't configured them to share additional things. Having even a few dozen people sharing DivX movies on a high-speed pipe uses up a large percentage of the school's bandwidth, far more than the network chatter does (we're talking on the order of 30-40 GB/day for a single host).

Re:If this were legal, it could be far more effici

[base3]

If you thing peer-to-peer networks offer total anonymity, try sharing some pr0n that's illegal in your locale, along with some realistic-appearing stories about killing $HEAD_OF_STATE for a month or two, 24x7, and get back to us about that anonymity thing.

The current apps (other than Freenet/GNUnet) all either connect to or request a TCP connection from the machine sharing the material. When the client retrieving what you're sharing connects to or is connected from your machine, your IP address is known and that's one level of indirection from your identity (barring use of an open proxy).

Although many of the clients, particularly some of the Gnutella ones like Limewire attempt to obfuscate the addresses a little at times, the protocol is open, and $THREE_LETTER_AGENCY or $COPYRIGHT_CARTEL is free to write a client to reap the IP addresses of those sharing certain content (q.v. Ranger).

Re:HTTP itself isn't byte-optimized

[Hast]

If HTTP was encoded down to be bit level, transfers would be so much smaller!

Not really actually. A typical HTTP request result in a HTTP header which is 100B. (Well, the local one I just tested did at least.) Both TCP and IP has headers of 20B each. So those layers has the same order of overhead. And If you have a large file which you send you only get one HTTP header over several TCP/IP packets.

And compared to the amount of data transfered in the data part of a HTTP packet the header is very small.

As youself pointed out, it's nice to have a human readable protocol when you're debugging. (Try debugging a TCP/IP stack for a while and you'll be very happy that HTTP is text based.)

Re:Text compresses

[spectecjr]

The best compromise is still compression. A fast compression algorithm like LZO (or even gzip) can reduce bandwidth even more than twinking out your variable names and spaces AND still provides readability.

No, the best solution is to use ASN.1 encoding, with an editor that understands ASN.1 as its backend format.

This problem was solved years and years and years ago.

Simon

Re:well...

[Hast]

the issue is saving bandwidth - and for extremely high traffic sites, i bet you see this type of coding a lot.

Yes most likely. And most coders also feel that they "optimize" the program when they write fewer characters. It doesn't have much impact (at least none that is good) however.

Humans shouldn't try to optimize stuff that a computer can do better. It's a waste of time, and a source for frustration. (When it breaks the code.)

just like an extremely small percentage of web sites need to have shortened names of objects.

I don't think anyone disagrees with you that compressing pages are good if you have a lot of traffic. But I sincerely doubt that anyone is doing it by hand. (Except stupid people with too much time on their hands.)

The "right" solution is to either preprocess the template HTML files or to add a GZIP module to the web server.

The point is...

[Steveftoth]

that it's not the movies as much as the protocol.

I bet that running a ftp server that has the same content will result in less traffic even if the movie is downloaded more often. Why? Because of the crosstalk inherit in the p2p protocols.

mod_gzip for Apache

[jsjacob]

A new HTTP command is not necessary because HTTP 1.1 supports compression as a content encoding (the "Content-Encoding" HTTP header). The mod_gzip module enables compression for Apache. As you suggested, mod_gzip can be configured to compress or not compress certain files matching given criteria.

Depends upon your use of "efficient"

[0x0d0a]

Freenet has extremely high latency, yes. Request a file, and it might be a while before you get a response. Try browsing the Web-on-Freenet, and you'll get a less-than-optimal experience.

However, Freenet has efficient file transfers in terms of bandwidth usage, and avoids killing any single point on the network. Freenet is network friendly.

Hey, in Australia we pay 13.5c...

[Goonie]

per *megabyte* for excess bandwidth usage from most of the broadband providers.

Not many Aussies run gnutella AFAIK :)

We can *always* blame the RIAA :-)

[0x0d0a]

*shrug* I understand that this sounds loony, but I assure you that most P2P coders (well, at least *I* would :-) ) would love nothing better than the ability to use centralized, stable, trusted servers. It'd make their life *much* easier to make solid, easy-to-use (remember the get-a-host game played on Gnutella?), efficient apps.

Gnutella and friends only came about because Napster was under attack by the RIAA. The only way for file sharing to survive was to mutate and scatter, go not just P2P, but fully distributed, with *no* central points of failure.

The RIAA has unhesitatingly attacked any P2P services that have a central server. Given the environment, P2P evolved to be much more distributed.

And in doing so, became less efficient.

Things will probably pick up eventually -- P2P research is under full steam, and is a popular thesis subject now. As P2P and scalable, distributed and untrusted storage becomes a better understood problem, efficiency will improve. At the moment, however, file sharing has been pushed into a raw area of research. And yes, I do blame the RIAA for this.

Incidently, it may turn out to be a good thing in the long term -- distributed, failsafe, untrusted networks have a lot of potential for the future, and it's unlikely that they would have been popularized nearly as soon without the RIAA.

Conflict brings evolution -- World War II brought us atomic power and the programmable computer, and it looks like the RIAA is heading to bring us into the next era of worldwide telecommunications.

Re:We can *always* blame the RIAA :-)

[Dthoma]

"Incidently, it may turn out to be a good thing in the long term -- distributed, failsafe, untrusted networks have a lot of potential for the future, and it's unlikely that they would have been popularized nearly as soon without the RIAA."

You're using such a network right now, which has been popularised with virtually no help from the RIAA whatsoever.

OT

[offpath3]

Dude, you were running a rogue DHCP? That'll get you banned from the network for the whole year at my school! Heck, the guy that lived in my dorm room before me did it and I had to talk with the network support for days just to get them to re-enable my jack after I moved in.

Song reference

[doublem]

Where has all the band width gone?

Do, da do, da do.

indeed

[twitter]

Quoth the article

And if people are using the advertising-supported versions of those programs, there is even more traffic generated as the ads are "pushed" at the user.

It's amazing but true that the same thing can be said of advertising-supported web pages. It seems that an amazing amount of bandwith is taken up by advertisments and images I did not request. Indeed, as much as 90% of all bandwith used by comercial sites is composed of such "network chatter" as X-10 suggesting I check out the girl next door who just lost 90 pounds on the ginsing diet. And it blinks. All that just to get about four kilobytes of text and four kilobytes of image that I actually want to see. The ratio over conventional media is reversed: on the internet, most content is crap, whereas in conventional media all the content is crap.

Bandwith must be conserved on the internet so more crap we don't want can be pushed on more of us.

A New Low

[whereiswaldo]

Obviously the person who submitted the story doesn't know what he is talking about.
Efficiency is a major focus of large P2P apps. When you are making hundreds of connections, you need to be efficient, or it won't work worth a damn. Coming up with an efficient enough algorithm is probably the hardest part!

As far as using up international bandwidth, the reason it is so expensive in the first place is because not enough of it is used. Telling people not to use more of it is saying that it should always remain expensive.

The story submitter is totally off-base.

60% wasted howz that?

[inquisitive]

My office folks were eating up 60% of the BW with Kazaa, I had no option but block at the FW, Now things are back to normal...

Re:Where did my bandwidth go?

[Pig Hogger]

"Never let your sense of morals prevent you from doing what is right" - Hari Seldon

Er, it was actually Salvor Hardin who said that.

Let the ISP run a supernode....

[anonymous cupboard]

It seems to me that perhaps the best answer would be for the ISP to run a super-node. It sounds strange, but the decrease in off-network bandwidth could more than compensate for the cost of running the node. Use of port filters can ensure that the only supernode seen by people on that network would be that of the ISP.

I guess the ISP would then be potentially in trouble from the RIAA even if they didn't store anything themselves.

Use QoS?

[Cato]

Putting all P2P traffic into a 'low priority' queue on all routers, and HTTP traffic and everything else into a 'normal priority' queue, would help this. Actually some sort of bandwidth allocation (WFQ, CBQ, etc) could be used rather than priority queuing. P2P apps would get the whole pipe if no higher priority traffic is around, but just X% if there is other traffic.

Of course, this is wildly impractical given the complete lack of uptake of QoS in the Internet - but since bandwidth hogs such as Pointcast and P2P drove earlier adoption of single-point QoS boxes such as Packeteer, it is not beyond the bounds of possibility. ISPs could deploy this without cooperation from other ISPs, just as a way of giving better service to non-P2P traffic within their network.

Of course, some would say that P2P should not be segregated - in which case, perhaps they could buy a premium service that puts P2P into 'normal priority'...