Slashdot Mirror
Where The Bandwidth Goes
An anonymous reader writes "An often overlooked fact about network bandwidth utilization is that the bandwidth consumed on networks is more than the sum of the data exchanged at the highest level; it's data+overhead+upkeep. In the early 90's I worked for a large multi-national company whose software engineering department had a transatlantic x.25 circuit connection to it's European engineering headquarters. It was necessary that the connection be 'on' 24x7 due to the spanning of a large number of time zones, disparate working hours and tight contractual requirements. Very large data transfers were sometimes operationally essential. But the financial people used to scream constantly about the circuit costs (charged per packet, IIRC) of several thousand dollars/month. The sys admin realized that if he just reduced the frequency of keep-alives, he could shave something like 10% off the monthly bill. This article points out that p2p applications are greater bandwidth hogs than one might think because of the foregoing and more - they also search, accept pushed advertising and do other transactions that are transparent to most users, but add up. I doubt that developers of those free p2p applications have gave much thought to efficiency. This will be no surprise to many of you, but helps explain why ISP's rushing to put caps on transfers."
it has nothing to do w/the advertising, the searches, etc. It has to do SOLELY w/the LARGE downloads that users of P2P networks do.
;)) Now that everyone is back (and I assume loving Kazaa to it's limit) I average about 75 to 100k/s.
Over the summer (when no one was in this little college town) I was steadily get 250+k/s downloads (mostly updating Debian
I am even tempted to call Road Runner and complain (I am just too lazy to fix Win98 and have it running so they can do their tests).
DiVX and MP3s are what kills the bandwith. Not the little "inefficiencies" that P2P authors added in.
Actually P2P work does focus on efficiency because efficiency determines how large the network can scale on a give set of hardware (the users machines and comodity internet connections). ISP's want to cap bandwidth because their current business model demands that they oversubscribe their uplink by around 20-200 times depending on the type and pricing of the comodity connection. Besides caps are based on total bandwidth usage which includes networking overhead (the routers accounting program doesn't care about payload usually)
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
We need web caches... It's stupid to have files crossing the ocean thousands of times. Besides not using web caches causes that those who cannot afford bandwidth costs cannot put content in the web... Caches now!.
Web developers must not be afraid of web caches, since the HTTP/1.1 protocol allows them to precisely define how and when their content will be cached.
The more they cap usage, the less people will use (obviously). Then content providers such as streaming radio stations will start to drop off as it becomes more expensive for users to access them.
After that it becomes a vicious circle, with fewer content providers, there's no reason for users to keep their service. Then the ISPs go broke.
Take a look at the Australian example. Almost all broadband providers have a 3Gb monthly cap. The ABC has just started an internet-only radio station, but I really wonder why. It wouldn't take too many days of listening to it for a user to totally max out their cap. I predict the station will be closed due to lack of interest, within a year.
-- Even if a god did exist, why the fsck should I worship it?
the geek it me though, says "waaa" and that things that dont evolve, die. and the things that dont die. p2p pushes the envelope right now, but all that encourages is more network growth. just think of p2p as those pains you had in your legs when you were 14. sure, it may not be the most efficient thing in the world, but the underlaying infrastructure has to take that into account, or get out of the way for one that can.
slashdot: where everyone yells sarcastic metaphors to themselves to understand the issue
The office I'm at used to have a contract with a monthly cap - a mere 20GB, with fairly hefty per-GB fees after that.
One Monday morning, I came in, and glanced at the MRTG graphs over the weekend. Keeripes! Somebody had been pushing data at about 250Kbps from Friday night until about 6 PM on Sunday, sustained.
I did a quick calculation, and then informed the bosses that we were going to be paying a lot more than usual this month, and asked if they wanted me to find out why. Of course they did.
Turned out it was one of said managers. He fired up Limewire, grabbed something on Friday, and forgot to shut it off. Seeing our nice low-latency, high capacity link (E10 or thereabouts, just with a really low traffic cap), it went supernode... and we paid about twice the usual for it.
Gnutella is not one of the more advanced protocols, but most of it's problems are present at varying levels in other p2p systems. It's not really surprising that P2P software which spends so much time trying to connect to computers, connect to a computer to start a download etc... and search in a geometric spiderring fashion are quite inefficient.
>>most of us dont need the damn hungarian notation that MS has spreads like gospel truth
Why said anything about that? And besides, MS now discourages its use.
>>It makes for unreadable names that convey less meaning that a nice clear variable name.
Which 'fn' is not but 'FirstName' is.
Now, if you have a dynamically generated page, you could use constants that are set to short stuff like 'fn'. Less code to be transmitted while still keeping most of the readability of the original code. If you discover a bug, temporarily switch to a different set of constants ('FirstName' instead of 'fn') until you sort it out so the resultant HTML is more readable. (Same goes with whitespace: Make a constant ENDL or NEWLINE that is set to '\n' while debugging, then changes to '' for production.)
no, there arent better solutions.
as i stated earlier, go to google and look at the source - one letter field names, and no line breaks.
why? TO SAVE BANDWIDTH.
every byte is sacred... every byte is great...
and if a byte is wasted, CFO's get quite irate...
... hi bingo
..is airing this commercial of goofy testimonials for their broadband cable service. A kid says "Ever been in the belly of a whale? I have", another guy goes "I go to the moon and back twice a day", etc.. etc..
Now, one of them has some guy say "I collected everything Mozart ever did... In 10 minutes!"
To me that's comes through loud and clear as "*wink* *wink* *nudge* *nudge* napster(etc)!"
I would say p2p is the driving force behind non-geeks getting broadband. They don't need it for e-mail, or casual web-surfing. They don't play games, but I know many people eager for an alternative to the bland junk on the radio. (Plus due to geography, radio reception is poor here)
Same thing with the 'work from home' bunk they promote, and yet block VPN connections.
It's like dangling a carrot in front of a mule to get him to move, and he stupidly chases it not realising he'll never reach it. It works fine in cartoons, but eventually the mule becomes frustrated, kicks you, and refuses to move at all.
Someone is smart enough to figure a way to give out the bandwidth and make money at the same time. And, it won't be a monopoly. Maybe 802.11 will be our savior?
I don't need no instructions to know how to rock!!!!
I can totally understand the limitations of bandwidth in the face of 2p2 software.
I was in my first year of college (living in the dorms) when Napster became popular. That same year, they banned it from all campus computers. The IT guys here said that of the estimated 7200 dorm room computers on campus, a minimum of 6500 were running Napster at any given time. They were forced to ban it because the bandwidth usage was taking away from vital staff/faculty related web-based tools and network services that needed to be maintained. In fact, nothing else could be run on the network.
Now Napster's gone, and I haven't lived on campus since Kazaa and such became popular. I'm pretty sure I know how they're dealing with it.
If one university had to do it, then imagine what the average cable/DSL provider has to deal with. Granted, they don't have as much essential network stuff.
>>>most of us dont need the damn hungarian notation that MS has spreads like gospel truth
>Who said anything about that? And besides, MS now discourages its use.
Got a reference for that? It's shoved down our throats whether we like it or not simply because 'it's the MS way'. It's damned impossible to debug something called 'lpszglname' especially when it isn't even a string any more because it was changed years ago...
As an engineer that made a network to do that that tanked (Nothing like lies from sales people) it's possible 512 kbit a sec looks pretty nice but the bandwith costs on the sending end are about 50 bucks a month before servers people etc (thats sending all month) why because it's all unicast because NO ISP wants mcast working outside of itself they dont know how to bill for it. A satalite at 500 an hour is much cheaper than delivering over the internet and inherently multicast. I wonder when somebody will come up with a multicast service that is delivered to a majority of ISP's.
No sir I dont like it.
The article itself was kind of ho-hum, but the following part of the Slashdot intro caught my attention:
Again...wow. One would need to search far and wide, even on Slashdot, to find another example of such absolutely astonishing cluelessness. Timothy has obviously never talked to a P2P developer in his life. Sometimes it seems like efficiency is just about the only thing P2P developers think about, unless someone's on a security/anonymity rant. Little things like robustness or usability get short shrift because so much of the focus is on efficiency. Hundreds of papers have been written about the bandwidth-efficiency of various P2P networks - especially Gnutella, which everyone who knows anything knows is "worst of breed" when it comes to broadcasting searches.
It's unfortunate that the most popular P2P networks seem to be the least efficient ones, and doubly unfortunate that so many vendors bundle spyware with their P2P clients, but to say that P2P developers don't give much thought to efficiency is absurd. They give a lot more thought to efficiency than Slashdot editors give to accuracy, that's for damn sure.
Slashdot - News for Herds. Stuff that Splatters.
Remeber back in the good ol' modem days. I remember getting 10 k a second on some transfer even with a 56.6.
If a P2P network protocol is text based, say like XML, it should compress pretty well and keep some of this extra bandwith down.
If HTTP would actually support compression natively we could save tons of bandwith in those HTML transfers. The page I'm typing this comment on is 11.1 k. zipped it is 3.5, and I think I have fast compression on. I'm sure the main slashdot page would save even more. Slashdot could litterally save megs a day.
It would simply be a matter of Apache and IIS supporting it. And maybe a new GETC command in HTTP that works the same. The browser would ask if the server supports it, and then go from there. Or try it and if it failed, try it normally. Apache or IIS would be smart enough to not try and compress JPEG, GIF, and other pre-compressed files.
Everything from FTP to SMTP could save a little here and there, which adds up quick.
Perhaps the real answer is to write it into the next version of TCP and have it hardware accelerated.
I say P2P mesh networks (ala Gnutella) need to have intelligent meshing algorithms so that the network tries to minimize the number of mesh links crossing a given physical uplink or a given backbone segment.
/x block as a.b.c.d - please connect to that node and drop this connection") or maybe even ask the remote node to preform some kind of query for it ("who wants a.b.c.e, because I don't?"). Our current "host caches" like router.limewire.com could gain some intelligence for whom they introduced to whom.
Such a scheme would return optimized search results because your net neighbors would know of your query before somebody on the other side of an uplink (and, as there is less routing between you, can transfer files faster in theory).
On top of that, with such a router-aware network the wasted bandwith of broadcast packets multiply crossing a given line due to reflection by peers on the other side would be virtually gone once the network became aware of the layout - ideally each node wouldn't have to learn but could get some kind of topological information from a node it connected to ("You are in the same
Instead of capping upload and download capacities as much as done now, perhaps those limits should be relaxed but a P2P "introduction" program installed on the ISP's router so that clients behind the firewall mesh with each other before a few of them send meshing links spanning the uplink.
Yes, downloads will still follow the usual TCP/IP pathways - which we presume are most efficient already. But the broadcast discovery packets which now ricochet around the network would, with an intelligent meshing algorithm, span as few uplinks as possible to query hosts as network-close as possible. All in all this would reduce traffic.
Somebody want to blow holes in this for me?
--Knots;
Anarchy$ dd if=/dev/random of=~/.signature bs=120 count=1
Ready?
Slashdot.org and traffic redirected from its links.
Freenet is more efficient than, say, the Web would be. Those DiVXes don't need to cross your ISPs downstream connection at all.
Gnutella is noisy, but that's not the fault of the creators. Blame the RIAA -- the first P2P applications were centralized. If you can give up the requirement that there be no single, trusted point of failure, it's much easier to make an efficient network. They attacked Napster, and now people have moved to mostly less efficient approaches.
May we never see th
I doubt that developers of those free p2p applications have gave much thought to efficiency
Some of us have. Search is much of the bandwidth in peer networks is wasted (downloads are downloads, but search can eat up a lot of bandwidth for little return)
There are some efficient, effective peer network search apps currently in development. Hopefully we can eventually leave gnutella and kazaa in the past and move on to more open, efficient networks...
MSK
Putting all P2P traffic into a 'low priority' queue on all routers, and HTTP traffic and everything else into a 'normal priority' queue, would help this. Actually some sort of bandwidth allocation (WFQ, CBQ, etc) could be used rather than priority queuing. P2P apps would get the whole pipe if no higher priority traffic is around, but just X% if there is other traffic.
Of course, this is wildly impractical given the complete lack of uptake of QoS in the Internet - but since bandwidth hogs such as Pointcast and P2P drove earlier adoption of single-point QoS boxes such as Packeteer, it is not beyond the bounds of possibility. ISPs could deploy this without cooperation from other ISPs, just as a way of giving better service to non-P2P traffic within their network.
Of course, some would say that P2P should not be segregated - in which case, perhaps they could buy a premium service that puts P2P into 'normal priority'...