Slashdot Mirror
Where The Bandwidth Goes
An anonymous reader writes "An often overlooked fact about network bandwidth utilization is that the bandwidth consumed on networks is more than the sum of the data exchanged at the highest level; it's data+overhead+upkeep. In the early 90's I worked for a large multi-national company whose software engineering department had a transatlantic x.25 circuit connection to it's European engineering headquarters. It was necessary that the connection be 'on' 24x7 due to the spanning of a large number of time zones, disparate working hours and tight contractual requirements. Very large data transfers were sometimes operationally essential. But the financial people used to scream constantly about the circuit costs (charged per packet, IIRC) of several thousand dollars/month. The sys admin realized that if he just reduced the frequency of keep-alives, he could shave something like 10% off the monthly bill. This article points out that p2p applications are greater bandwidth hogs than one might think because of the foregoing and more - they also search, accept pushed advertising and do other transactions that are transparent to most users, but add up. I doubt that developers of those free p2p applications have gave much thought to efficiency. This will be no surprise to many of you, but helps explain why ISP's rushing to put caps on transfers."
Actually P2P work does focus on efficiency because efficiency determines how large the network can scale on a give set of hardware (the users machines and comodity internet connections). ISP's want to cap bandwidth because their current business model demands that they oversubscribe their uplink by around 20-200 times depending on the type and pricing of the comodity connection. Besides caps are based on total bandwidth usage which includes networking overhead (the routers accounting program doesn't care about payload usually)
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
We need web caches... It's stupid to have files crossing the ocean thousands of times. Besides not using web caches causes that those who cannot afford bandwidth costs cannot put content in the web... Caches now!.
Web developers must not be afraid of web caches, since the HTTP/1.1 protocol allows them to precisely define how and when their content will be cached.
uh, no. the actual bandwidth hogs there are not the files themselves, but the packet overhead of transferring those files. Keep in mind that searches are broadcast, so when one person searches next to you, they send out N search packets, one of which goes to you. You send out N search requests on their behalf to all the people you're connected with, and so on. So if all N of your peers are searching simultaneously (this is on a closed network of just you and N people, keep in mind), you're forwarding somewhere like N^2 packets.
At N=16 and average packet size of 128Bytes, that's 16*16*128 = 32KB. 32KB for just you and 16 friends, nobody else. As soon as you add more people to the topology, the math gets trickier and the numbers get much, much larger. Also keep in mind that the 128 is "ideal", not including the overhead of TCP establishing sessions, etc.
There were a lot of papers published on this about a year ago, i don't have any references to them, though.
IMO, you're very wrong.
The university I work for has it's spies watching the border routers, logging streams. Daily, they release a "top talker" list to a select few individuals (not myself) who notify the admins of aberrant hosts. This is to stop blatant abuse, as well as cut off possibly compromised hosts.
Occasionally, I would leave gnut running in a shell when I left for the day. I'd usually end up on "top talkers" with 1-2GB of traffic when no downloads were running! This was solely the chatter of the gnutellanet in action.
Of course, I do configure my client to talk to a rather wide neighborhood, but still...
Method of processing duck feet
the geek it me though, says "waaa" and that things that dont evolve, die. and the things that dont die. p2p pushes the envelope right now, but all that encourages is more network growth. just think of p2p as those pains you had in your legs when you were 14. sure, it may not be the most efficient thing in the world, but the underlaying infrastructure has to take that into account, or get out of the way for one that can.
slashdot: where everyone yells sarcastic metaphors to themselves to understand the issue
The office I'm at used to have a contract with a monthly cap - a mere 20GB, with fairly hefty per-GB fees after that.
One Monday morning, I came in, and glanced at the MRTG graphs over the weekend. Keeripes! Somebody had been pushing data at about 250Kbps from Friday night until about 6 PM on Sunday, sustained.
I did a quick calculation, and then informed the bosses that we were going to be paying a lot more than usual this month, and asked if they wanted me to find out why. Of course they did.
Turned out it was one of said managers. He fired up Limewire, grabbed something on Friday, and forgot to shut it off. Seeing our nice low-latency, high capacity link (E10 or thereabouts, just with a really low traffic cap), it went supernode... and we paid about twice the usual for it.
Gnutella is not one of the more advanced protocols, but most of it's problems are present at varying levels in other p2p systems. It's not really surprising that P2P software which spends so much time trying to connect to computers, connect to a computer to start a download etc... and search in a geometric spiderring fashion are quite inefficient.
Every time I visit a web page using my cable modem, I feel a pang of guilt. By visiting a web page, I:
Bandwidth is a finite resource which we should all conserve. One day, eventually, the Internet will run out of bandwidth.
>>most of us dont need the damn hungarian notation that MS has spreads like gospel truth
Why said anything about that? And besides, MS now discourages its use.
>>It makes for unreadable names that convey less meaning that a nice clear variable name.
Which 'fn' is not but 'FirstName' is.
Now, if you have a dynamically generated page, you could use constants that are set to short stuff like 'fn'. Less code to be transmitted while still keeping most of the readability of the original code. If you discover a bug, temporarily switch to a different set of constants ('FirstName' instead of 'fn') until you sort it out so the resultant HTML is more readable. (Same goes with whitespace: Make a constant ENDL or NEWLINE that is set to '\n' while debugging, then changes to '' for production.)
no, there arent better solutions.
as i stated earlier, go to google and look at the source - one letter field names, and no line breaks.
why? TO SAVE BANDWIDTH.
every byte is sacred... every byte is great...
and if a byte is wasted, CFO's get quite irate...
... hi bingo
..is airing this commercial of goofy testimonials for their broadband cable service. A kid says "Ever been in the belly of a whale? I have", another guy goes "I go to the moon and back twice a day", etc.. etc..
Now, one of them has some guy say "I collected everything Mozart ever did... In 10 minutes!"
To me that's comes through loud and clear as "*wink* *wink* *nudge* *nudge* napster(etc)!"
I would say p2p is the driving force behind non-geeks getting broadband. They don't need it for e-mail, or casual web-surfing. They don't play games, but I know many people eager for an alternative to the bland junk on the radio. (Plus due to geography, radio reception is poor here)
Same thing with the 'work from home' bunk they promote, and yet block VPN connections.
It's like dangling a carrot in front of a mule to get him to move, and he stupidly chases it not realising he'll never reach it. It works fine in cartoons, but eventually the mule becomes frustrated, kicks you, and refuses to move at all.
Someone is smart enough to figure a way to give out the bandwidth and make money at the same time. And, it won't be a monopoly. Maybe 802.11 will be our savior?
I don't need no instructions to know how to rock!!!!
I can totally understand the limitations of bandwidth in the face of 2p2 software.
I was in my first year of college (living in the dorms) when Napster became popular. That same year, they banned it from all campus computers. The IT guys here said that of the estimated 7200 dorm room computers on campus, a minimum of 6500 were running Napster at any given time. They were forced to ban it because the bandwidth usage was taking away from vital staff/faculty related web-based tools and network services that needed to be maintained. In fact, nothing else could be run on the network.
Now Napster's gone, and I haven't lived on campus since Kazaa and such became popular. I'm pretty sure I know how they're dealing with it.
If one university had to do it, then imagine what the average cable/DSL provider has to deal with. Granted, they don't have as much essential network stuff.
The article itself was kind of ho-hum, but the following part of the Slashdot intro caught my attention:
Again...wow. One would need to search far and wide, even on Slashdot, to find another example of such absolutely astonishing cluelessness. Timothy has obviously never talked to a P2P developer in his life. Sometimes it seems like efficiency is just about the only thing P2P developers think about, unless someone's on a security/anonymity rant. Little things like robustness or usability get short shrift because so much of the focus is on efficiency. Hundreds of papers have been written about the bandwidth-efficiency of various P2P networks - especially Gnutella, which everyone who knows anything knows is "worst of breed" when it comes to broadcasting searches.
It's unfortunate that the most popular P2P networks seem to be the least efficient ones, and doubly unfortunate that so many vendors bundle spyware with their P2P clients, but to say that P2P developers don't give much thought to efficiency is absurd. They give a lot more thought to efficiency than Slashdot editors give to accuracy, that's for damn sure.
Slashdot - News for Herds. Stuff that Splatters.
Remeber back in the good ol' modem days. I remember getting 10 k a second on some transfer even with a 56.6.
If a P2P network protocol is text based, say like XML, it should compress pretty well and keep some of this extra bandwith down.
If HTTP would actually support compression natively we could save tons of bandwith in those HTML transfers. The page I'm typing this comment on is 11.1 k. zipped it is 3.5, and I think I have fast compression on. I'm sure the main slashdot page would save even more. Slashdot could litterally save megs a day.
It would simply be a matter of Apache and IIS supporting it. And maybe a new GETC command in HTTP that works the same. The browser would ask if the server supports it, and then go from there. Or try it and if it failed, try it normally. Apache or IIS would be smart enough to not try and compress JPEG, GIF, and other pre-compressed files.
Everything from FTP to SMTP could save a little here and there, which adds up quick.
Perhaps the real answer is to write it into the next version of TCP and have it hardware accelerated.
I really wouldn't necessarily characterize it as "chatter". After all, isn't the point of P2P to allow multiple hosts to "share the load"? Though you may not be downloading anything, many might be downloading from you. (This is, of course, assuming you have your client configured to share.) KaZaA, by default, puts its installable executable in your shared directory making it available for anyone to grab.
If you have ever "expanded" the downloding sources, it often shows the download being done from multiple sources. It could just be that your client is uploading part or all of some shared file.
Not that there aren't inefficiencies, though.
My mom always said, "Jim, you're 1 in a million." Given the current population, there are 7000 of me. God help us all!
I say P2P mesh networks (ala Gnutella) need to have intelligent meshing algorithms so that the network tries to minimize the number of mesh links crossing a given physical uplink or a given backbone segment.
/x block as a.b.c.d - please connect to that node and drop this connection") or maybe even ask the remote node to preform some kind of query for it ("who wants a.b.c.e, because I don't?"). Our current "host caches" like router.limewire.com could gain some intelligence for whom they introduced to whom.
Such a scheme would return optimized search results because your net neighbors would know of your query before somebody on the other side of an uplink (and, as there is less routing between you, can transfer files faster in theory).
On top of that, with such a router-aware network the wasted bandwith of broadcast packets multiply crossing a given line due to reflection by peers on the other side would be virtually gone once the network became aware of the layout - ideally each node wouldn't have to learn but could get some kind of topological information from a node it connected to ("You are in the same
Instead of capping upload and download capacities as much as done now, perhaps those limits should be relaxed but a P2P "introduction" program installed on the ISP's router so that clients behind the firewall mesh with each other before a few of them send meshing links spanning the uplink.
Yes, downloads will still follow the usual TCP/IP pathways - which we presume are most efficient already. But the broadcast discovery packets which now ricochet around the network would, with an intelligent meshing algorithm, span as few uplinks as possible to query hosts as network-close as possible. All in all this would reduce traffic.
Somebody want to blow holes in this for me?
--Knots;
Anarchy$ dd if=/dev/random of=~/.signature bs=120 count=1
Here in Europe I have to pay for bandwidth. And so long as I do not run anything like Gnutella I have no bandwidth problems. I can share with Kazza and no problems. But the moment I share with GNUTella, my bandwidth shot through the roof. I kept it running a week and have never started it again.
"You can't make a race horse of a pig"
"No," said Samuel, "but you can make very fast pig"
Freenet is more efficient than, say, the Web would be. Those DiVXes don't need to cross your ISPs downstream connection at all.
Gnutella is noisy, but that's not the fault of the creators. Blame the RIAA -- the first P2P applications were centralized. If you can give up the requirement that there be no single, trusted point of failure, it's much easier to make an efficient network. They attacked Napster, and now people have moved to mostly less efficient approaches.
May we never see th
I doubt that developers of those free p2p applications have gave much thought to efficiency
Some of us have. Search is much of the bandwidth in peer networks is wasted (downloads are downloads, but search can eat up a lot of bandwidth for little return)
There are some efficient, effective peer network search apps currently in development. Hopefully we can eventually leave gnutella and kazaa in the past and move on to more open, efficient networks...
Putting all P2P traffic into a 'low priority' queue on all routers, and HTTP traffic and everything else into a 'normal priority' queue, would help this. Actually some sort of bandwidth allocation (WFQ, CBQ, etc) could be used rather than priority queuing. P2P apps would get the whole pipe if no higher priority traffic is around, but just X% if there is other traffic.
Of course, this is wildly impractical given the complete lack of uptake of QoS in the Internet - but since bandwidth hogs such as Pointcast and P2P drove earlier adoption of single-point QoS boxes such as Packeteer, it is not beyond the bounds of possibility. ISPs could deploy this without cooperation from other ISPs, just as a way of giving better service to non-P2P traffic within their network.
Of course, some would say that P2P should not be segregated - in which case, perhaps they could buy a premium service that puts P2P into 'normal priority'...