Slashdot Mirror
Where The Bandwidth Goes
An anonymous reader writes "An often overlooked fact about network bandwidth utilization is that the bandwidth consumed on networks is more than the sum of the data exchanged at the highest level; it's data+overhead+upkeep. In the early 90's I worked for a large multi-national company whose software engineering department had a transatlantic x.25 circuit connection to it's European engineering headquarters. It was necessary that the connection be 'on' 24x7 due to the spanning of a large number of time zones, disparate working hours and tight contractual requirements. Very large data transfers were sometimes operationally essential. But the financial people used to scream constantly about the circuit costs (charged per packet, IIRC) of several thousand dollars/month. The sys admin realized that if he just reduced the frequency of keep-alives, he could shave something like 10% off the monthly bill. This article points out that p2p applications are greater bandwidth hogs than one might think because of the foregoing and more - they also search, accept pushed advertising and do other transactions that are transparent to most users, but add up. I doubt that developers of those free p2p applications have gave much thought to efficiency. This will be no surprise to many of you, but helps explain why ISP's rushing to put caps on transfers."
Spyware is such a joke. The legality of most p2p networks is questionable, and yet they choose to include more software to make them seem even more dubious than they actually are. Then there are wannabe's like Kazaalite that get rid of the spyware, but keep the ads.
My suggestion for those of you who love your p2p, but hate the added junk?
WinMX: http://www.winmx.com/
No spyare and no ads. If only it were open-source...
Take Care
A1miras
uh, no. the actual bandwidth hogs there are not the files themselves, but the packet overhead of transferring those files. Keep in mind that searches are broadcast, so when one person searches next to you, they send out N search packets, one of which goes to you. You send out N search requests on their behalf to all the people you're connected with, and so on. So if all N of your peers are searching simultaneously (this is on a closed network of just you and N people, keep in mind), you're forwarding somewhere like N^2 packets.
At N=16 and average packet size of 128Bytes, that's 16*16*128 = 32KB. 32KB for just you and 16 friends, nobody else. As soon as you add more people to the topology, the math gets trickier and the numbers get much, much larger. Also keep in mind that the 128 is "ideal", not including the overhead of TCP establishing sessions, etc.
There were a lot of papers published on this about a year ago, i don't have any references to them, though.
IMO, you're very wrong.
The university I work for has it's spies watching the border routers, logging streams. Daily, they release a "top talker" list to a select few individuals (not myself) who notify the admins of aberrant hosts. This is to stop blatant abuse, as well as cut off possibly compromised hosts.
Occasionally, I would leave gnut running in a shell when I left for the day. I'd usually end up on "top talkers" with 1-2GB of traffic when no downloads were running! This was solely the chatter of the gnutellanet in action.
Of course, I do configure my client to talk to a rather wide neighborhood, but still...
Method of processing duck feet
The office I'm at used to have a contract with a monthly cap - a mere 20GB, with fairly hefty per-GB fees after that.
One Monday morning, I came in, and glanced at the MRTG graphs over the weekend. Keeripes! Somebody had been pushing data at about 250Kbps from Friday night until about 6 PM on Sunday, sustained.
I did a quick calculation, and then informed the bosses that we were going to be paying a lot more than usual this month, and asked if they wanted me to find out why. Of course they did.
Turned out it was one of said managers. He fired up Limewire, grabbed something on Friday, and forgot to shut it off. Seeing our nice low-latency, high capacity link (E10 or thereabouts, just with a really low traffic cap), it went supernode... and we paid about twice the usual for it.
go look at the html code from google - notice how they abbreviate every object name to ONE letter in the interest of bandwidth.
:-)
i'm sorry that you learned how to code sloppily, and are bitching about streamlining code for efficiency, and cost savings.
most of us dont need the damn hungarian notation that MS has spreads like gospel truth. It makes for unreadable names that convey less meaning that a nice clear variable name.
oh - and i know when to use a goto to streamline code, too
... hi bingo
ISPs are putting bandwidth caps on accounts because they see it as a source of revenue. Plain and simple. The crap about how 5% of the users use 95% of the bandwidth is really starting to piss me off... they advertised always on, unlimited bandwidth when I signed up, and now they have enough customers used to the speed, so they essentially upped the price (just like soup companies reduced the size of their cans of soup, but kept the price the same, if you want more, buy a larger can...) if you want more bandwidth, upgrade your package, or better yet, pay $7.95 a GB/Month over our generous 3 GB/month...
:P
Isn't there a law against doing this sorta crap? They said always on, unlimited bandwidth... now they're charging through the nose, claiming crappy stats on usage, and blaiming it on p2p networks... I can't even download my legitimate MSDN ISO images without going over my monthly bandwidth limit, let alone actually doing anything else on the net...
End rant...
---
Programming is like sex... Make one mistake and support it the rest of your life.
I couldn't disagree more.
I just got into an amazingly poorly written program after about a year, and was bewildered by the names, and what they really meant. And it was my code. And yet, I couldn't disagree with you more.
The streamlining that was discussed by the parent isn't for the sake of the coder, it is for the sake of the user. Mostly your argument is founded in long names really don't hurt anything. And if that is true, than long, descriptive names do their job. Here is a prime example of where they do make a difference. Here the variable names (for variables, and even javascript, or vb script embedded in a page) could make a tremendous amount of difference. And that is wht you would be stream lining for.
As for superfluous naming, well that can be just as bad, and unreadable as short names. Addled is addled, and you can use short (maybe more than 3, this isn't RPG afterall) descriptive names, without having to type an entire sentence.
junk1, junk2, junk3 will never be a good idea, but if your form has 4 variables, and you name them
FNm, LNm, MI, and Age, I don't think anyone will be confused.
I can totally understand the limitations of bandwidth in the face of 2p2 software.
I was in my first year of college (living in the dorms) when Napster became popular. That same year, they banned it from all campus computers. The IT guys here said that of the estimated 7200 dorm room computers on campus, a minimum of 6500 were running Napster at any given time. They were forced to ban it because the bandwidth usage was taking away from vital staff/faculty related web-based tools and network services that needed to be maintained. In fact, nothing else could be run on the network.
Now Napster's gone, and I haven't lived on campus since Kazaa and such became popular. I'm pretty sure I know how they're dealing with it.
If one university had to do it, then imagine what the average cable/DSL provider has to deal with. Granted, they don't have as much essential network stuff.
Remeber back in the good ol' modem days. I remember getting 10 k a second on some transfer even with a 56.6.
If a P2P network protocol is text based, say like XML, it should compress pretty well and keep some of this extra bandwith down.
If HTTP would actually support compression natively we could save tons of bandwith in those HTML transfers. The page I'm typing this comment on is 11.1 k. zipped it is 3.5, and I think I have fast compression on. I'm sure the main slashdot page would save even more. Slashdot could litterally save megs a day.
It would simply be a matter of Apache and IIS supporting it. And maybe a new GETC command in HTTP that works the same. The browser would ask if the server supports it, and then go from there. Or try it and if it failed, try it normally. Apache or IIS would be smart enough to not try and compress JPEG, GIF, and other pre-compressed files.
Everything from FTP to SMTP could save a little here and there, which adds up quick.
Perhaps the real answer is to write it into the next version of TCP and have it hardware accelerated.
I say P2P mesh networks (ala Gnutella) need to have intelligent meshing algorithms so that the network tries to minimize the number of mesh links crossing a given physical uplink or a given backbone segment.
/x block as a.b.c.d - please connect to that node and drop this connection") or maybe even ask the remote node to preform some kind of query for it ("who wants a.b.c.e, because I don't?"). Our current "host caches" like router.limewire.com could gain some intelligence for whom they introduced to whom.
Such a scheme would return optimized search results because your net neighbors would know of your query before somebody on the other side of an uplink (and, as there is less routing between you, can transfer files faster in theory).
On top of that, with such a router-aware network the wasted bandwith of broadcast packets multiply crossing a given line due to reflection by peers on the other side would be virtually gone once the network became aware of the layout - ideally each node wouldn't have to learn but could get some kind of topological information from a node it connected to ("You are in the same
Instead of capping upload and download capacities as much as done now, perhaps those limits should be relaxed but a P2P "introduction" program installed on the ISP's router so that clients behind the firewall mesh with each other before a few of them send meshing links spanning the uplink.
Yes, downloads will still follow the usual TCP/IP pathways - which we presume are most efficient already. But the broadcast discovery packets which now ricochet around the network would, with an intelligent meshing algorithm, span as few uplinks as possible to query hosts as network-close as possible. All in all this would reduce traffic.
Somebody want to blow holes in this for me?
--Knots;
Anarchy$ dd if=/dev/random of=~/.signature bs=120 count=1
How much of current network traffic (data/voice) are really just protocol? I mean all the way down to physical layer (yes, the 1 and 0's). Seems like every layer of abstraction tags a protocol header on to the real payload. Is there any study done in this? I won't be surprised if more than 50% of network traffic are just protocols (IP headers, TCP signals, SONET header or even CRC bits).
Here in Europe I have to pay for bandwidth. And so long as I do not run anything like Gnutella I have no bandwidth problems. I can share with Kazza and no problems. But the moment I share with GNUTella, my bandwidth shot through the roof. I kept it running a week and have never started it again.
"You can't make a race horse of a pig"
"No," said Samuel, "but you can make very fast pig"
With the "always on" mentality of broadband users, and the fact that most clients simply hide in the system tray when you click the top-right "x" on the window rather than shut down, it wouldn't surprise me if a substantial amount of bandwidth wasn't directly related to a particular client downloading.
Method of processing duck feet
Freenet is more efficient than, say, the Web would be. Those DiVXes don't need to cross your ISPs downstream connection at all.
Gnutella is noisy, but that's not the fault of the creators. Blame the RIAA -- the first P2P applications were centralized. If you can give up the requirement that there be no single, trusted point of failure, it's much easier to make an efficient network. They attacked Napster, and now people have moved to mostly less efficient approaches.
May we never see th
The point is you didnt oversubscribe enough. Or you may want to try things like proxy servers. Everything on your side of the network is 'free'. Yet you can still use that inside number to cap people. This will help 'some'. You can also 'limit' certian ports with QOS type things. Like port 1425 or what ever kazaa uses only gets 5-20% of the bandwidth. More than likely your TOS have wesaly wording in it that lets you do that. You may even have 'no servers' in it. Well each one of those is a server... Put the math on peoples bills. Or on a web page where they could see what they could be saving. But if you are making extra off em and they are happy with it...
:). Offer versions of gnut that look for local machines before outside ones. Remember inside bw is cheeper. Encourage people to not turn on supernode with kazaa.
My isp (roadrunner) uses a sloped type bw limiter. It will ramp up to 800kbytes per second very quickly. Then will come back down to 200-250kbytes. So that way small things go fast. Other bigger transfers do not swamp the network. You could also use something like this to 'take out' the more serious people. While keeping the rest of your customers cruising along.
Another thing you can do to help yourself is the DNS cache. Make it HUGE. Watch the number of requests that come out of clients. You will be amazed. I would guess on a average 2-3 hour surf for me. I goto 10-15 pages. Those pages in turn hit 10-15 OTHER links. It gets rather large quickly. A nice way to speed up my connection is to turn on a decent DNS cache. Windows is rather pittfully small.
Find out if any of your customers is going out to other USENET servers on the net. Remember you are going to dl what they want anyways. They will also be paying someone else to use your bw for this. So you might as well make it on your terms. If you put a decent USENET server inside your domain. Then set rules that people can live by. You can remove some of your outside bw, and bring it inside your network. I am not saying grab ALL of alt.* just the ones customers want. Have a nice way for them to request sites. That way you get only the sites your customers want and you can save money by not downloading all of alt.*.
Guide people in how to setup these programs that help you and them. If they must use them, and they must
Now if you still have bw hogs use your TOS to your advantage. You can write your TOS to flip customers that exceed bw for so many months in a row to a metered type acount. Make sure they are aware of it up front. You set your meter usage to be the same as if they used exactly your max. You will see their usuage rates change dramaticly when they see they are going to PAY for that bw. It seems people do not like metered use it to your advantage.
Another thing you can do to help yourself is to properly configure your routers. Make it so only certian routers will talk to certian ip addresses. Do not assume they will be good little customers and not do this sort of thing. They will. This will clip alot of trojans. Encourage your customers to get ZoneAlarm or the like. Tell your customers you will be port sweeping them. And do it. Trojan sort of trafic is not wanted on your network by you or your customer. Get them discounts on mcafee or trend. Like 5-20% off. Virus companies would probably drool over things like this to sell more copies. Respond to alerts from people telling you that they are seeing crap come out of your network. Take it out. It is unwanted bandwidth and costs you money.
My office folks were eating up 60% of the BW with Kazaa, I had no option but block at the FW, Now things are back to normal...
Putting all P2P traffic into a 'low priority' queue on all routers, and HTTP traffic and everything else into a 'normal priority' queue, would help this. Actually some sort of bandwidth allocation (WFQ, CBQ, etc) could be used rather than priority queuing. P2P apps would get the whole pipe if no higher priority traffic is around, but just X% if there is other traffic.
Of course, this is wildly impractical given the complete lack of uptake of QoS in the Internet - but since bandwidth hogs such as Pointcast and P2P drove earlier adoption of single-point QoS boxes such as Packeteer, it is not beyond the bounds of possibility. ISPs could deploy this without cooperation from other ISPs, just as a way of giving better service to non-P2P traffic within their network.
Of course, some would say that P2P should not be segregated - in which case, perhaps they could buy a premium service that puts P2P into 'normal priority'...