Where The Bandwidth Goes

An anonymous reader writes "An often overlooked fact about network bandwidth utilization is that the bandwidth consumed on networks is more than the sum of the data exchanged at the highest level; it's data+overhead+upkeep. In the early 90's I worked for a large multi-national company whose software engineering department had a transatlantic x.25 circuit connection to it's European engineering headquarters. It was necessary that the connection be 'on' 24x7 due to the spanning of a large number of time zones, disparate working hours and tight contractual requirements. Very large data transfers were sometimes operationally essential. But the financial people used to scream constantly about the circuit costs (charged per packet, IIRC) of several thousand dollars/month. The sys admin realized that if he just reduced the frequency of keep-alives, he could shave something like 10% off the monthly bill. This article points out that p2p applications are greater bandwidth hogs than one might think because of the foregoing and more - they also search, accept pushed advertising and do other transactions that are transparent to most users, but add up. I doubt that developers of those free p2p applications have gave much thought to efficiency. This will be no surprise to many of you, but helps explain why ISP's rushing to put caps on transfers."

Comments

I doubt it.

[garcia]

it has nothing to do w/the advertising, the searches, etc. It has to do SOLELY w/the LARGE downloads that users of P2P networks do.

Over the summer (when no one was in this little college town) I was steadily get 250+k/s downloads (mostly updating Debian ;)) Now that everyone is back (and I assume loving Kazaa to it's limit) I average about 75 to 100k/s.

I am even tempted to call Road Runner and complain (I am just too lazy to fix Win98 and have it running so they can do their tests).

DiVX and MP3s are what kills the bandwith. Not the little "inefficiencies" that P2P authors added in.

Re:I doubt it.

[Anonymous Coward]

You do realize that while "everyone was gone", there was no advertising, no searches were being performed, etc, etc, right?

Re:I doubt it.

[Anonymous Coward]

uh, no. the actual bandwidth hogs there are not the files themselves, but the packet overhead of transferring those files. Keep in mind that searches are broadcast, so when one person searches next to you, they send out N search packets, one of which goes to you. You send out N search requests on their behalf to all the people you're connected with, and so on. So if all N of your peers are searching simultaneously (this is on a closed network of just you and N people, keep in mind), you're forwarding somewhere like N^2 packets.

At N=16 and average packet size of 128Bytes, that's 16*16*128 = 32KB. 32KB for just you and 16 friends, nobody else. As soon as you add more people to the topology, the math gets trickier and the numbers get much, much larger. Also keep in mind that the 128 is "ideal", not including the overhead of TCP establishing sessions, etc.

There were a lot of papers published on this about a year ago, i don't have any references to them, though.

Re:I doubt it.

[Deagol]

It has nothing to do w/the advertising, the searches, etc. It has to do SOLELY w/the LARGE downloads that users of P2P networks do.

IMO, you're very wrong.

The university I work for has it's spies watching the border routers, logging streams. Daily, they release a "top talker" list to a select few individuals (not myself) who notify the admins of aberrant hosts. This is to stop blatant abuse, as well as cut off possibly compromised hosts.

Occasionally, I would leave gnut running in a shell when I left for the day. I'd usually end up on "top talkers" with 1-2GB of traffic when no downloads were running! This was solely the chatter of the gnutellanet in action.

Of course, I do configure my client to talk to a rather wide neighborhood, but still...

Re:I doubt it.

[Anonymous Coward]

Every time I visit a web page using my cable modem, I feel a pang of guilt. By visiting a web page, I:

• drive up bandwidth costs for the webmaster that are not covered by advertising
• consume my ISP's bandwidth
• consume shared bandwidth, slowing down my neighbor's computer slightly.

Bandwidth is a finite resource which we should all conserve. One day, eventually, the Internet will run out of bandwidth.

Re:I doubt it.

[sparkyman]

Try running a p2p program on a dialup connection (22.6K, thank you very much). It quite handily eats up most of my connection, leaving very little left for anything else. Nevermind when you are downloading something.

Re:I doubt it.

[garcia]

so you are going to tell me that when I download 5 DiVX movies a day (700+mb each) that I am *NOT* hogging 3.5G of bandwith for that day?

Sure, traffic hogs it, but so do the files. When you have 85% of your cable modem subscribers downloading 700+mb files daily, all day, everyday, you are going to see a significant drop in your overall bandwith.

Again, I will point out that when no one was here in this college town (25000 person difference) that the bandwith was WIDELY available. Once RR started getting everyone back on, the bandwith went to hell.

Re:I doubt it.

[jbarr]

I really wouldn't necessarily characterize it as "chatter". After all, isn't the point of P2P to allow multiple hosts to "share the load"? Though you may not be downloading anything, many might be downloading from you. (This is, of course, assuming you have your client configured to share.) KaZaA, by default, puts its installable executable in your shared directory making it available for anyone to grab.

If you have ever "expanded" the downloding sources, it often shows the download being done from multiple sources. It could just be that your client is uploading part or all of some shared file.

Not that there aren't inefficiencies, though.

Re:I doubt it.

[Anonymous Coward]

you do realize that there were no downloads being performed.

Re:I doubt it.

[SerpentMage]

Here in Europe I have to pay for bandwidth. And so long as I do not run anything like Gnutella I have no bandwidth problems. I can share with Kazza and no problems. But the moment I share with GNUTella, my bandwidth shot through the roof. I kept it running a week and have never started it again.

Re:I doubt it.

[Deagol]

No, I don't allow downloads. I usually hop on to find some obscure file, then hop right off. I don't share files much these days, certainly not with my work machine.

I don't even allow caching -- too gray an area for my tastes, especially on my employer's equipment.

Re:I doubt it.

[Deagol]

Perhaps I responded a bit too harshly. Your original point seemed to imply that nothing other than downloads acounted for a non-trivial use of overall bandwidth. I responded with anecdotal evidence that this is not always the case.

With the "always on" mentality of broadband users, and the fact that most clients simply hide in the system tray when you click the top-right "x" on the window rather than shut down, it wouldn't surprise me if a substantial amount of bandwidth wasn't directly related to a particular client downloading.

Re:I doubt it.

[dattaway]

Bandwidth is a finite resource which we should all conserve. One day, eventually, the Internet will run out of bandwidth.

Consider joining the Bandwidth Conservation Society.

Re:I doubt it.

[gwernol]

Though you may not be downloading anything, many might be downloading from you. (This is, of course, assuming you have your client configured to share.) KaZaA, by default, puts its installable executable in your shared directory making it available for anyone to grab.

You're missing the point - go back and read the article linked in the story. The point is that excluding uploads and downloads these P2P networks are producing a lot of nework traffic. The example quoted is up to 1.6GB a day just for running the client. Again, this is excluding the bandwidth required for uploads and downloads to/from your machine. This is just the overheads of communication, searches and ad pushes.

Not that there aren't inefficiencies, though.

The point being the inefficiencies are so large that just having a few hundred P2P machines running on your network can amount to a significant bandwidth drain, even before they share a single file.

Re:I doubt it.

[machine of god]

Heh, when I am at OSU (soon, oh so soon), I make sure I have stuff queued up at least a day ahead. At off times, I usually get about 1000 k a second. Anyway, all I'm saying is that the talk between clients is peanuts compared to what is being downloaded. Not necessarily not seeing the forest through the trees, just more like purposely ignoring the forest while focusing on the trees.

Re:I doubt it.

[Atlantix]

so you are going to tell me that when I download 5 DiVX movies a day (700+mb each) that I am *NOT* hogging 3.5G of bandwith for that day?

of course you're using 3.5G of bandwidth for the files. but the point of the article is that if it takes you all day to download those 5 movies, you're also using 1G of chatter as your client software responds to searches and forwards other people's searches around the network. so you're using at least 4.5G in that day. in this specific case, yes the files are still the majority of the bandwidth but let's say you're one of those pesky students who only happened to queue up 5 MP3s before you left for class but left the client on all day because you went straight to work and then a party. now the downloads (20M) are a tiny fraction of the total bandwidth (1.02G). BTW, I can certainly understand the college town effect although I was the annoying student in that situation :) Sorry about that!

--Atlantix2000

Re:I doubt it.

[Stephen Samuel]

so you are going to tell me that when I download 5 DiVX movies a day (700+mb each) that I am *NOT* hogging 3.5G of bandwith for that day?

Guh.
His point was that even when you're not downloading your 3G of files a day, you're still using a good bit of bandwidth. The ofiginal article mentions that 10% of his network's bandwidth could be chopped by simply cutting down on network keep-alive packets. If the grand-parent's comments are any indication, Some P2Ps may be well above that.

Or to put it another way: I'm not saying that you always look stupid -- just when you say things like that.

Re:I doubt it.

[putzin]

It's not just p2p either. I run Ethereal every now and then on my DSL router to keep track of those intrepid hackers using my wireless connection (the girl two floors down apparently thinks her computer is actually "on" the internet) and to see if anyone has decided it is time to break in. Just watch a typical email session on hotmail. 80%+ of the traffic is solely advertising and hotmail related extra crap. There were 1000 plus packets before I even saw the first message header (I thought someone was doing something naughty). I couldn't believe it at first, but really, that's just amazing. Now the point.

Which is, we really don't notice all of the extra traffic generated every time we hit a website or fire up Morpheus. Generally, you expect the downloads, but you don't expect the protocol overhead, or the ads, or keepalives, or whatever else might be bundled in. This is where we could save bandwidth if we wanted to. But, we don't want to. I would freak if Ameritech imposed a bandwidth restriction.

Re:I doubt it.

[Magila]

Gnutella is a bad example since it's about the worst possible senerio in terms of wasted bandwidth. Something like kazaa (assuming you're not a supernode) or edonkey use a trivial amount of bandwith while idle since they're not constantly receiving and sending searches (i.e. they keep servers and clients separate). I've said many times that gnutella is a steaming pile because it wastes so much bandwidth on searches, your example is proof just how bad it is.

Re:I doubt it.

[Anonymous Coward]

Run a bandwidth monitor with a Gnutella client up, and you'll see that it's a very inefficient protocol. Everyone knows it, it's been discussed at length, some proprietary networks have endeavored to fix it...

GTK-Gnutella now includes some bandwidth-throttling features that are very convenient, though it means that, occasionally, you trade your best position in the gnutnet for the sake of saving bandwidth. (I've connected to fellow GTK-Gnutella users with more bandwidth than I, and watched the connections drop due to buffer overruns. Apparently Gnut sends some sort of ack back to the originating host when a search is sucessfully passed on?)

Perhaps we need P2P based on a 'king for a day' principle; peered hosts denote a single machine to play server, but only for the next, say, 1 minute, then it rotates. This would reduce some redundancy in control packets; each server would create a temporary in-memory copy of all the users' shares.

Re:I doubt it.

[Anonymous Coward]

Fucking leech. It's dickfucks like you that cause the network to be unstable.

Re:I doubt it.

[patchmaster]

All gnutella clients are not created equal. There have been a number of features implemented in some of the clients that dramatically reduce the query and response traffic. Unfortunately, not all client authors have implemented these features. And some people are still using very old clients that lack these optimizations. If everyone would get on board with query routing and ultrapeers (sic) the situation would improve greatly.

You're right, though, kazaa is still ahead of gnutella in terms of keeping down the unnecessary network traffic.

Re:I doubt it.

[Wyzard]

1-2 gigabytes overnight? Bah.

My university allows one gigabyte of transfer within a 12-hour sliding window, and if you go over that limit, you get severely rate-limited. I used to run LimeWire with 20 connections, just to get my search horizon as big as possible, and I found that simply being connected for about an hour (no downloads, nobody downloading from me) would put me over that gigabyte limit.

I got into the habit of getting some connections up, doing a search, and disconnecting as soon as my results were in. (Unfortunately, this meant I couldn't download from people who were behind firewalls, since doing a "push" reqest requires both parties to be connected to the network.)

Gnutella overhead most definitely takes up a considerable share of bandwidth.

Re:I doubt it.

[DrMaurer]

re: The "always on" mentality:

Well, I don't know if you pay attention to stuff or not, but the advertisements for the cable modems here seem to push that kind of ideal.

"Always on. No busy signals. Fast downloads of multimedia."

Then they complain when we do exactly that. Don't advertise it unless you're willing to deliver it.

But, the question: Is it (bandwidth) a "right?"

I think so. Maybe. Hard to say. It's not a right like life or something that vital, but the limiting of servers and other "bandwidth issues" makes me concerned because it starts to make the internet a one way conduit of information instead of allowing all users to serve their own content, which is really not what the Internet is capable of or, IMO, the ideal usage of the 'net is.

Beyond those ideals, which I admit, probably aren't shared by C*O's around the world, it's important that companies that get into this broadband invest in increasing their own bandwidth. The bandwidth "limit" companies complain about is fixable, and it's only going to get worse if they refuse to make that bandwidth grow with their hopeful customer base.

There is money to be made with the promise of freedom. Are you more likely to go with an (otherwise idential) ISP that allows/encourages (even simply does not dis-allow) servers for your content as opposed to one that disallows & tries to block their use? Freedom is important, and people, especially now, are conginzant (sp) of that.

Yes, I use P2P, not gnutella or kazaa or popular ones but private hubs; take your salt as recommended by your physician.

Re:I doubt it.

[undercanopy]

But, the question: Is it (bandwidth) a "right?"

of course it's not. You're paying a company for an ungaruanteed service (read your TOS, i bet you find alot of "up to"s and "as high as" phrases in there. You're paying a consumer-level price for a consumer-level service.

If you want guaranteed bandwidth and the "right" to run servers, fine, pay for it. Most ISPs would be happy to let you run servers if you pay for the business-grade service.

People running servers off of their own machines are GENERALLY a)more likely to be bandwidth hogs (read: warez) and b) more likely to cause trouble for the ISP (read: warez)

The average person doesn't have a legitimate need to run servers from their home.

Re:I doubt it.

[DrMaurer]

Perhaps I didn't write my post properly, because you seem to have misunderstood it.

Bandwidth as "right". It's not a right like life, as I said before, but rather it's such a simple problem to fix that it shouldn't even be an issue once you pay for access. Yes, I know the cable can be expensive, but the government has grants, I'm sure, to offset the costs.

And your last statement is totally asinine.

Of course the average person DOES have a sue for a server. They should have a platform for their views, and your statement shows what some of those views could be: generalizing and wrong.

I've run a server off my home PC for years, never serving any warez or anything, simply my web site and my own MP3s of my music. Nothing more, nothing less. Limit it to web pages, maybe. Not an ideal solution, I suppose.

I'm talking about speech here, and freedom of the new "digital" press. (God, I feel totally stupid putting it like that, it's almost as bad as ePress or something.) Banning servers could be, possibly, a restriction on that freedom (in the US). Not that would get 10 feet in a tossing contest if attemted to be proved in court. Sure, AT&T can be forced to open up their wires to competitors, but Joe Blow can't put up his web page to exercise 1st amendment freedoms.

Don't ya'll get it? We have this freedom, and we're giving it up because of mythical costs.

But I'm a near-socialist on the issue of bandwidth. Obviously a crackpot. Freedom is not important. Leave the cable laying and bandwidth allocating to the telecom industry, they're obviously upright citizens doing well for humanity. (Insert bullshit about delivering worth to stockholders here. Companies can't be idealistic. Whatever.)

Re:I doubt it.

[Anonymous Coward]

Thank you for modding this +1 Insightful. You just made my day. :)

so lots of applications...

[heyeq]

...are crappily designed and use lots of bandwidth. imagine the bandwidth wastage on every windoze xp bootup when it checks for updates.

i don't understand why this was posted? with so much unlit fibre and the telecoms industry in a slump, it's only a matter of time before bandwidth is so cheap and ubiquitous we'll be scratching our heads at what to do with it.

Re:so lots of applications...

[jethro_troll]

From the article:
All you have to do to blow your monthly download limit with Sympatico or Videotron is to leave your computer running with a P2P file-sharing program turned on

Hey, there should be a law that we can deduct outgoing bytes from the incoming bytes and only be changed for net bytes downloaded. (This should put an end to P2P freeloaders!)

Re:so lots of applications...

[Kintanon]

Maybe when ALL THAT BANDWIDTH I'll finally be able to get real TV streaming to me? I mean, sputnik7.com is great, but all they have is anime and their selection doesn't change very often...
Where can I go to get streaming Comedy Central or SciFi or Cartoon Network or something? I'd pay a few bucks per channel for that, if the quality was excellent.

Kintanon

Re:so lots of applications...

[duren686]

It would be nice from a purely "would be nice" point of view, but in effect you would get charged less for using more bandwidth, which wouldn't fly with anyone who provides bandwidth.

Optimize html

[Winterblink]

I wonder how much bandwidth could be saved annually if people who developed webpages maybe optimized their html a little better? Removing extraneous spacings, simplifying form field namings ("fn" instead of "FirstName"), that kind of thing. Especially sites that get insane amounts of traffic. You know, like Slashdot. :)

Re:Optimize html

[heyeq]

If you're anywhere near being a k0der you'll be squeezing your head in digust. for fifteen+ years we've been battling against asm-style two/three letter variable declarations, and finally have languages that have helped us define naming conventions and the like, and you want us to go BACK to TLA's??? (TLA = two/three letter acronym)

are you insane?

Re:Optimize html

[stefanlasiewski]

Or, as an alternative, how much bandwidth could be saved if webservers used compression, like mod_gzip?

Re:Optimize html

[!splut]

I tend to think that poorly-optomized html is just a drop in the bucket. If there were one thing to optomize, it ought to be images. The average page uses at least an order of magnitude more data for its images than the html. Using smaller images, or just saving things in more efficient formats, such as jpg, or lossless compression formats, would be a big step in the right direction.

Re:Optimize html

[O2n]

This reminds me of the guys trying to ban the nose from smileys
to save, quote, 30% of the bandwidth. :) -- politically correct.

Re:Optimize html

[Zenithal]

> such as jpg, or lossless compression formats, would be a big step in the right direction.

As opposed to what? What sites are you seeing out there that are using an uncompressed image format? bmps & tiffs? I didn't even think you could embed an uncompressed format in a web page.

Re:Optimize html

[error0x100]

The average page uses at least an order of magnitude more data for its images than the html

You'd be surprised. This exact page itself (the slashdot page to the parent comment of your comment): The HTML is 26 KB, and the images on the page ADD UP TO 12.4 KB. Moreover, the images will be cached by the browser locally, or by a proxy, and thus will only be downloaded *once* for each comment I view. The HTML, on the other hand, must be downloaded fresh for *every single comment* I view on slashdot. So after an hour of browsing /., you've probably downloaded at *least* an order of magnitude more html than images.

And do a "view source" .. these pages can be optimized quite a bit.

Most people just think of the HTML as being only a tiny fraction of the download size, and thats why most webpage developers don't even think to optimize their HTML.

Re:Optimize html

[Anonymous Coward]

Exactly! This type of compression is supported in both IE and IIS so it reaches over 98% of the market.

I'm assuming it doesn't work with Netscape 4, but what does?

Re:Optimize html

[fpu]

I was a sysadmin at RADIX, a search engine in Brasil. When we first tried mod_gzip, we got about 40% less output traffic. With the kind of prices we pay for connectivity down here, that meant we saved about R$30,000 (about US$10,000) a month for our 8Mbps link. But then again that was some time ago.

Re:Optimize html

[qwerpoiu]

I wonder how much bandwidth could be saved annually if people who developed webpages maybe optimized their html a little better? Removing extraneous spacings, simplifying form field namings ("fn" instead of "FirstName"), that kind of thing. Especially sites that get insane amounts of traffic. You know, like Slashdot. :)

That's why /.ers abbreviate "First Post!" to "fp".

Re:Optimize html

[J'raxis]

<paragraph> Would you rather write HTML like <emphasis>this</emphasis>? </paragraph>

<paragraph> Sure, &open-quotation-mark; TLAs &close-quotation-mark; may be annoying to read, but they are certainly OK to use if they are understandable enough. </paragraph>

</hypertext-markup-language>

Re:Optimize html

[Sean Clifford]

Good point. However, readability is important. I do a lot of ASP (yeah, I know..., migrating to perl) coding and optimize my code for scalability and speed; however, I won't sacrifice readability. I leave my comments in. Yes, I know I can strip them out for the production server, but I prefer to leave my comments in and use vbcrlf line feeds to break up the html into something readable. Helps with debugging on the development server.

That said, text compresses pretty damned well. So I'm less concerned with text than with images. I much prefer png to jpeg & gif, but gotta make sure the client supports it (dynamically feed whichever).

For sites with massive traffic, you're right - it behooves you to optimize everything you can - but not to the point of making life difficult for the development team:

Monkey 1: "Hey, what the hell is 'lnm'?"
Monkey 2: "Hell if I know."
Monkey 1: "You wrote it!"
Monkey 2: "Yeah, but that was like 4 months ago. What do the comments say?"
Monkey 1: "DOH!"

I don't like my HTML generated in one humumgous unbroken string when I look at the source.

Re:Optimize html

[non-poster]

I'm assuming it doesn't work with Netscape 4, but what does?

Come on! Of course it does!

mod_gzip is for apache; thttpd and other web servers have content-encoding support, too.

Re:Optimize html

[silentbozo]

My question is, what does mod_gzip break? Hell, my crappy HTML/1.0 client supports gzip content (Netscape 4.08), so why aren't more people using it?

Not only does it save bandwidth on the server side, but text pages render instantaneously on the client side when the server supports it (I'm using Netscape on an old Mac, so this kind of stuff is important.) I say everyone should plug it in, and we'll fix all the broken stuff afterwards...

Re:Optimize html

[Winterblink]

Hey, I program in ASP. Flame me if you like, but hey it's a good paying job. :D

Anyway, I'm not advocating tiny variables for the actual dynamic script side of code, merely for the HTML heading out to the client. And I only cited that as an example. :D One bad thing I see all over the place is people using a trillion indents and linefeeds to split up sections of code. Blech!

Re:Optimize html

[rgmoore]

So on Slashdot the images are about half the size of the content. Now consider that Slashdot is pretty light in its use of images compared to many popular web sites. How much of a typical web page on a news site (where a lot of the images can't be recycled for every page) is likely to be used by the images? It's a lot.

Re:Optimize html

[balloonhead]

I wonder if someone could develop an HTML parser then that could compress the code...
Instead of:
<HTML>
<BODY>
blah blah blah <TAG>blah</TAG>
</BODY>
</HTML>

It would just run this through a pre-processor (we could call it the hypertext preprocessor preprocessor, or, GNU style, PHPP) and give us:
<HTML><BODY>blah blah blah <TAG>blah</TAG></BODY></HTML>
Obvioulsy this would be a small saving, but on bigger sites it might provide some benefit (especially database driven sites with heavy HTML output). For debugging it would be no use whatsoever, but as it is a pre-processor you write your code as normal, then via your PHPP app it is put onto your server's doc-root where human eyes might never need to look at it.

I suppose though that it might have to be called a preprocessor postprocessor as the tags would be abbreviated as the last step before sending info to the client. Acronym remains though.

Would it have any actual usefulness? For most, no; but if you were running a large site it means your coders don't need to sacrifice readability as it is all processed behind the scenes; they can still disable it to see the HTML source with line breaks; you could even use much more commenting and longer variable names for readability as it could strip these all out too, just having the absolute minimum file size but producing exactly the same output.

Re:Optimize html

[woogieoogieboogie]

strTheLetterA = "A"
strTheLowerCaseLetterA = "a"
int_counter_for_loop_of_array_name_myArray = 0

Yup, gotta love it when the variable name is descriptive. NOT!!!!!

Re:Optimize html

[KenDaMan]

I don't like my HTML generated in one humumgous unbroken string when I look at the source.

I don't like to look at huge unbroken strings either but if you have ever connected to the internet at 26k you will appreciate any site that optimizes their HTML. Eliminating the whitespace can shave as much as 5% of the file size. Changing variable names to TLN can shave off another 10%-20% of the file size.

I recently compared XML file size on large XML files by using these two methods. I found that some of the files were reduce 25% of the original file size. The network traffic that this eliminated certainly outweighed the readability of the files.

When generating HTML dynaically I create one long string. If I need to debug the client side HTML I use a program that I can copy the HTML into and format for readability by adding whitespace. For comments I use comment tags that I can search for in the server side source.

Re:Optimize html

[Hast]

Does Slashdot do GZIP? In any case I bet that if you added GZIP to Slashdot the transfered size would shrink a lot. (But you wouldn't see that if you looked at the size of the source, as it's decompressed.)

And no optimizations in the world are going to remove the big shunk of data on Slashdot. Since that is in the posts.

Re:Optimize html

[woogieoogieboogie]

Err, this is Slashdot, everyone uses Lynx, wget or telnet to view their webpages. Don't believe me, post a link to slashdot and watch the HEAD requests skyrocket

Re:Optimize html

[SCHecklerX]

Too bad there isn't a 'mod_gzip' for mail. That would be something that could save bandwidth.

Re:Optimize html

[jooniqzb1tch]

/. does use mod_gzip. and I bet it does save a lot of bandwith, considering all the similar content :]

Re:Optimize html

[woogieoogieboogie]

"I do a lot of ASP (yeah, I know..., migrating to perl) "

Why? Do you have a lot of text files which need to be parsed? You could use perl as the default scripting language in ASP and then switch to VBS for it's simplicity where applicable.

I just finished a MSSql/ASP backed Real Estate site for a regional RE company, it includes comprehensive property listing searches, ability to save properties, compare them and add notes the lsitings so users can keep track of the properties they wish to purchase. The entire site is generate from 1 30k asp page with al my functions and the html headers/footers and navigation, and 11 other asp pages, none of which are larger than 7k. The ENTIRE website with all the scripting, style sheets and HTML I am a php guy, but I have to admit that ASP with VBS is extremely easy to work with and with a little effort, you can produce highly compact readable code. Good technology is good regardless of who made it

Re:Optimize html

[anthony_dipierro]

If you're using a preprocessor, just #define long_name ln. If you're not, then create a makefile system to generate your html pages.

Re:Optimize html

[Anonymous Coward]

> are you insane?

No, dude, using long variable names in web development doesn't make a whole lot of sense. In fact, it doesn't make ANY sense.

> If you're anywhere near being a k0der ...you're probably not replacing "o" w/0.

Re:Optimize html

[LegendLength]

u r so rght bro!

Re:Optimize html

[Anonymous Coward]

With the amount of bandwidth that has been wasted on this discussion I could have downloaded Gone With The Wind.

Re:Optimize html

[Sean Clifford]

I didn't think you were suggesting server-side; just that I prefer readability on the client side. But as you and several others pointed out, saving 25%+ bandwidth means that I should optimize client output on the production server.

I can't flame you for ASP development - that's how I make most of my living and what the clients want.

Re:Optimize html

[error0x100]

A "typical web page on a news site"? OK, I'll bite. Here is a breakdown of cnn.com:

HTML+CSS+JS: 95.7 KB

IMAGES: 91.6 KB

The html alone: 47.41 KB

Sorry, but the size of the images are still overshadowed by the HTML+stylesheet and Javascript. There are fewer images than web page code. There are certainly NOT "orders of magnitude" more images than HTML.

And it gets *worse* if you click on the stories. The main story on cnn.com right now has 56 KB of images and 122 KB of web page code!

Go do the math, check out some web pages, and you will see that you are just plain wrong. You are basing your opinion on the conventional wisdom, "common sense", intuitive idea that the HTML is typically only a fraction of the images. Its wrong. And we don't even need to sit and argue about it, you can head straight to web sites like CNN.com and prove it for yourself.

And a non-negligible percentage of the slashdot html source of any /. page is still redundant whitespace.

Doesn't explain anything

[afidel]

Actually P2P work does focus on efficiency because efficiency determines how large the network can scale on a give set of hardware (the users machines and comodity internet connections). ISP's want to cap bandwidth because their current business model demands that they oversubscribe their uplink by around 20-200 times depending on the type and pricing of the comodity connection. Besides caps are based on total bandwidth usage which includes networking overhead (the routers accounting program doesn't care about payload usually)

Re:Doesn't explain anything

[CrackersnSoup]

As a ISP i agree, in part. True every ISP(right up to the Tier 1's) over subscribe bandwidth. The main reasion for bit caps is bandwith is a like water or electricity. You pay for what you use. People have been spoiled on 19.95(and less) all you can eat dial up. Why? Because it was(and still is) a great way to give people a flat rate, wich everyone (majority) likes. I have more people going for my $50/mo accounts then i do my bit billed accounts. 70% or so of the people say they like the idea of the bit bill but absolutly wont got for a service they cant count on what the bill will be. Most if not all that 70% will save money in 18-24 months. Many ISP's have tried the $XX/YY hour dialup accounts, most have given up on them. Few people want them and those that do have been online for a few years and know there online habits well. Belive it or not, dial up still is the first stop for most people new to the internet. Because BW is so new, the stupidity of the ISP's, Its sold as a fat BW pipe thats always on. No one dreamed of bit caps 5 years ago. ISP's a waking up to broadband and its real cost, and user's dont like it. Every user that come's by askign aobut my service ask's why I chave bit caps and people in huston dont. I tell thme i want to stay in business. @Home is a example of what happens when you sell 2000-3000 of BW for $50 a month and NOT expect people to use it. Sorry for getting off topic. I had a point in the beginning, i think. Crackers`n`Soup

Re:Doesn't explain anything

[Anonymous Coward]

The thing is that bandwidth could be far cheaper than it is, but the telcos are keeping it in check to continue to charge huge rates for T1's. The fibre is already in the ground, if the backbone providers wanted to they could light much of it tommorow, but their profitability demands that it stay unlit until demand is sufficient to cause the price of bandwidth to be higher than it otherwise would be, in otherwords they are artifically limiting bandwidth to keep themselves profitable, in a perfect market all the fibre would be lit and prices would plumet.

Re:Doesn't explain anything

[Anonymous Coward]

The point is you didnt oversubscribe enough. Or you may want to try things like proxy servers. Everything on your side of the network is 'free'. Yet you can still use that inside number to cap people. This will help 'some'. You can also 'limit' certian ports with QOS type things. Like port 1425 or what ever kazaa uses only gets 5-20% of the bandwidth. More than likely your TOS have wesaly wording in it that lets you do that. You may even have 'no servers' in it. Well each one of those is a server... Put the math on peoples bills. Or on a web page where they could see what they could be saving. But if you are making extra off em and they are happy with it...

My isp (roadrunner) uses a sloped type bw limiter. It will ramp up to 800kbytes per second very quickly. Then will come back down to 200-250kbytes. So that way small things go fast. Other bigger transfers do not swamp the network. You could also use something like this to 'take out' the more serious people. While keeping the rest of your customers cruising along.

Another thing you can do to help yourself is the DNS cache. Make it HUGE. Watch the number of requests that come out of clients. You will be amazed. I would guess on a average 2-3 hour surf for me. I goto 10-15 pages. Those pages in turn hit 10-15 OTHER links. It gets rather large quickly. A nice way to speed up my connection is to turn on a decent DNS cache. Windows is rather pittfully small.

Find out if any of your customers is going out to other USENET servers on the net. Remember you are going to dl what they want anyways. They will also be paying someone else to use your bw for this. So you might as well make it on your terms. If you put a decent USENET server inside your domain. Then set rules that people can live by. You can remove some of your outside bw, and bring it inside your network. I am not saying grab ALL of alt.* just the ones customers want. Have a nice way for them to request sites. That way you get only the sites your customers want and you can save money by not downloading all of alt.*.

Guide people in how to setup these programs that help you and them. If they must use them, and they must :). Offer versions of gnut that look for local machines before outside ones. Remember inside bw is cheeper. Encourage people to not turn on supernode with kazaa.

Now if you still have bw hogs use your TOS to your advantage. You can write your TOS to flip customers that exceed bw for so many months in a row to a metered type acount. Make sure they are aware of it up front. You set your meter usage to be the same as if they used exactly your max. You will see their usuage rates change dramaticly when they see they are going to PAY for that bw. It seems people do not like metered use it to your advantage.

Another thing you can do to help yourself is to properly configure your routers. Make it so only certian routers will talk to certian ip addresses. Do not assume they will be good little customers and not do this sort of thing. They will. This will clip alot of trojans. Encourage your customers to get ZoneAlarm or the like. Tell your customers you will be port sweeping them. And do it. Trojan sort of trafic is not wanted on your network by you or your customer. Get them discounts on mcafee or trend. Like 5-20% off. Virus companies would probably drool over things like this to sell more copies. Respond to alerts from people telling you that they are seeing crap come out of your network. Take it out. It is unwanted bandwidth and costs you money.

Re:Doesn't explain anything

[CrackersnSoup]

I fully agree. I am working on setting up a IRC server for everyone in town to do there, research of files, on. I explain it very carefully that the BW i charge for is what is used out to the 'net and not in town. They can use all the BW from each other they want. No one(but me) wants usenet access so i have not setup a server for that, yet. When i do it will be a caching server so it will handle wich groups to work with. I will throw a small 80G at it and they will like it or lump it =-)

Currently my bigest problem is 1 user. He is in for a rude awakening. The account he has is 256kbit up and down. 3G transfer/mo. $15 per GB after. In less then 3 days he has hit 1G.He is "only" using 1/2 of it at any one time. If he keeps it up he will pay for my T1 all by his self =-)

The customers i like are the older ones.They generaly do email and surfing with a few stock traders and ebay sellers. Once i get my proxy/dns server online ill be MUCH happier.(just lost a box to now known reasion)

I offer a firewall for $5/mo(bsd ipfw + dummynet)
and the local collage gives norton away free(for those who know) and i sell it for a good deal less then anyone else in town.

Crackers`n`Soup

Re:Doesn't explain anything

[Anonymous Coward]

My ISP doesn't do any capping.

Re:Doesn't explain anything

[Deluge]

Another thing you can do to help yourself is the DNS cache. Make it HUGE. Watch the number of requests that come out of clients. You will be amazed. I would guess on a average 2-3 hour surf for me. I goto 10-15 pages. Those pages in turn hit 10-15 OTHER links. It gets rather large quickly. A nice way to speed up my connection is to turn on a decent DNS cache. Windows is rather pittfully small

You can edit the DNS Client service settings in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Servic es\DNSCache\Parameters. Obviously this applies to NT/2k/XP only, but who uses 9x for anything other than games anymore anyway.

Bandwith...

[A1miras]

Spyware is such a joke. The legality of most p2p networks is questionable, and yet they choose to include more software to make them seem even more dubious than they actually are. Then there are wannabe's like Kazaalite that get rid of the spyware, but keep the ads.

My suggestion for those of you who love your p2p, but hate the added junk?

WinMX: http://www.winmx.com/

No spyare and no ads. If only it were open-source...

Re:Bandwith...

[Mitchell Mebane]

Sounds like you haven't used the KaZaA Lite HOSTS file. It gets rid of the ads.

Re:Bandwith...

[Anonymous Coward]

Where do u find this? :-)

Doubt?

[Justen]

"...I doubt that developers of those free p2p applications have gave much thought to efficiency..."

It is more than doubt. The downloadable P2P programs are made to pull advertising for revenues; that is their priority. Those dozens of pop-up windows weren't coming from Mars; efficiency had nothing to do with getting the biggest, fattest, most obnoxious ad possible on your screen.

jrbd

We need web caches

[Nicopa]

We need web caches... It's stupid to have files crossing the ocean thousands of times. Besides not using web caches causes that those who cannot afford bandwidth costs cannot put content in the web... Caches now!.

Web developers must not be afraid of web caches, since the HTTP/1.1 protocol allows them to precisely define how and when their content will be cached.

Re:We need web caches

[FredMcGriff]

in the mp3 world it was called napster.

Re:We need web caches

[Cloudmark]

While caching does offer a lot of advantages, there are also pitfalls, particularly for those providing it.

Working for an ISP myself and specifically with the bandwidth tracking section, we deal with prety much every type of high bandwidth application out there and in many cases we could save an immense amount by caching. Unfortunately, if we cache and then illegal material is downloaded, we can be held responsible for that material. It's unfortunate that efficiency must be sacrificed but right now it's generally too dangerous for anyone to run a serious caching system.

The rule of thumb for ISPs, at least in North America, is generally that if it's on a client system (subscriber - your PC), then it's not our problem (legally). If a file resides on our cache, then we can be held responsible for it by law enforcement agencies.

As to the general suggestion that a great deal of bandwidth is consumed by overhead, I think there is some merit to it but that it's a fairly small amount compared to what is used by deliberate downloads and transfers. Systems are moving towards greater efficiency in order to improve speed and to work with lower bandwidth platforms (phones, PDAs, etc) but bandwidth is unlikely to be a major motivator. Most broadband subscribers either download too little to cause serious issues (6gb a month or so - limited overhead) or extreme volumes (100gb a month - overhead is dwarfed by content).

Re:We need web caches

[Salamander]

Unfortunately, if we cache and then illegal material is downloaded, we can be held responsible for that material.

That's not really true. Even under the DMCA you can qualify as a safe harbor and avoid that liability. The requirements are too burdensome for an invididual, and that IMO is only one flaw among many in the DMCA, but for an ISP that employs full-time staff it's entirely doable and many ISPs have done it. You'll need to find another excuse.

Re:We need web caches

[Sloppy]

I agree. But I must be wrong about something, because ISPs don't seem to be deploying caches. ISPs would seem to have the most to gain from caches, and they are also at a very natural and sensible point for it.

I know there are ISP slashdotters. Any of you guys want to explain why web caches aren't worthwhile?

Re:We need web caches

[ender81b]

What about specifing a really, really short 'life' for it to be cached.

I know in squid you can specifiy that anything over x minutes is to be discarded. Then again I'm not sure if squid can handle an entire ISP's worth of traffic (probably though).

That would solve your problem with caching illegal content if it was just discarded after, oh, say 90 minutes.

Re:We need web caches

[Sloppy]

Have you considered maybe some sort of whitelist cache? For example, when 100 people are all downloading a RedHat ISO or Microsoft service pack, surely you'd like to move it across your outside pipe only once, and there isn't much chance that RedHat and Microsoft are distributing kiddie porn.

Or is "isn't much chance" not good enough? Argh.

This sucks. If you're protected from liability on how your wires are used, that protection should extend to caches. It's just common sense. Sounds like we've got a fucked up law or something.

Re:We need web caches

[Sloppy]

DMCA deals with copyright-related things. There are some types of content that can be illegal, regardless of copyright issues. Public domain kiddie porn (*), nuke plans, and that web messageboard where you negotiate plutonium-for-heroin deals, fall outside the scope of DMCA.

(*) Heh, I can see it now: kiddie porn from the early 1920s whose copyright has expired. No wait, when I said, "I can see it now," I didn't mean I was looking at it. No, wait, ossifer, this is all a misunderstanding...

Re:We need web caches

[Salamander]

And where is the law or precedent that would be used to prosecute an ISP for caching things that are illegal for other than copyright reasons? Common-carrier laws have been around for a long time, and have been used many times to shield the people who provide a communications medium from culpability for what is transmitted over that medium. The reason I mentioned the DMCA is that, in the specific case of copyrighted material, it attempts to supplant existing law and precedent; in cases other than copyright, that law and precedent is still valid and likely to protect the ISP as a common carrier.

Then again, law moves in mysterious ways. Are you aware of a recent case that would lead to a different conclusion?

Re:We need web caches

[eNonymous Coward]

I agree. But I must be wrong about something, because ISPs don't seem to be deploying caches. ISPs would seem to have the most to gain from caches, and they are also at a very natural and sensible point for it.
I know there are ISP slashdotters. Any of you guys want to explain why web caches aren't worthwhile?

Cloudmark already pointed out one major stumbling block: potential legal issues regarding content which is "stored" on a machine owned by the ISP. Several years ago in New York, the state AG went after a bunch of ISPs because they carried child erotica newsgroups on their news servers. Mind you, customers of those particular ISPs weren't necessarily doing anything illegal, and it wasn't end-users being targeted. The AG was going to file charges against ISP officials simply for providing access to the material. His reasoning was that the ISPs could be held responsible under the law, because the illegal material resided on hard drives under those providers' direct control. I don't know how it all turned out, I just remember that it was big news in alt.privacy back then, much as the RIAA vs ISPs is big news now.

That's the criminal side of things, now for the other perspective. As more people begin to rely on the internet as a publication medium, more people also begin to expect that it's going to be a real-time medium (for some reasonable value of "real"). After all, that's what the commercials promise about this whole digital communications thing. "I can type an email to my grandma and she gets it instantly!" "I can get up to date traffic alerts on my cell phone and avoid traffic jams!"

Not only is it being sold this way, it's being used this way, more so than you might think. For example, a lot of teachers and instructors - all the way down to the elementary level - post things like the class syllabus, assignments, an outline of the next lecture, etc. online. Financial information, like today's current bank rates, is distributed via the web. Sports scores, stats, and betting odds are online. And we can't forget the ultimate example of time-sensitive information, stock quotes.

The information people want is on the web, and much if not all of this information is constantly changing. Users expect to get the most current, most accurate data when they visit a site, local (browser) cache notwithstanding. They expect the ISP to deliver the content, not to alter it or keep it around until it's arbitrarily considered stale. In a perfect world, web developers would specify no-cache if they didn't want their visitors having problems; in the real world, it doesn't happen.

Consider these scenarios:

A student stays home from school due to illness. He knows it's not a big deal, because his teachers post all of the homework online. He visits his teachers' sites to look up the assignments, but gets last week's copy of the pages instead. The following day, none of the teachers are keen on his excuse that "my ISP's cache server ate my homework."

A telecommuting employee has his new projects posted on a web page each morning by his manager. He checks his "New Projects" list, completes all of the tasks, and calls it a day; not knowing that overnight his ISP had implemented a transparent cache to improve performance. The next morning he checks the list again, and - finding no new work posted - he decides to go fishing. When he gets home, there are 4 angry messages from the boss asking why he hasn't completed today's projects.

Joe Sixpack doesn't usually play the lottery, but while he's paying for gas he decides to buy a ticket for the hell of it. He knows that the winning numbers are posted each night at www.superwinninglottomeganumbers.com. That night he checks the winning numbers (having no idea, of course, that he's actually looking at yesterday's). Not surprised that his ticket didn't win, he pitches it. The next morning in the paper he sees that his number was actually a winner...

I guess what I'm getting at, in a long-winded sort of manner, is that caching is more likely to confuse or frustrate users, or even outright harm them, than it is to help them. Caching should be up to the users.

Re:We need web caches

[nolife]

The rule of thumb for ISPs, at least in North America, is generally that if it's on a client system (subscriber - your PC), then it's not our problem (legally). If a file resides on our cache, then we can be held responsible for it by law enforcement agencies.

Not flaming here but where is this information from? I have NEVER seen or read about anything relating to this concept of responsiblility for temporary cache storage, specially since you are considering this a rule of thumb. What about a customers mail that is sitting in your queue? What about those binaries on your news server? What about some files in /home/someuser/myfiles?

Re:We need web caches

[Cato]

There is a lot of support in the HTTP caching headers for controlling the timeliness of cached copies - you can say 'keep this for just 5 minutes then check for a new version'. The trouble is that web developers don't know much about caching.

Of course, your browser caches already, and can easily be more aggressive than proxy caches, so eliminating proxy caches is not going to solve anything. If someone using a web-based application is so clueless they don't know about the Refresh button, AND the application developer managed to do a cacheable dynamic page, that is just lack of training, or using the wrong tools.

Defeating caching is very easy - most CGI-based sites are not cacheable, and many commercial sites turn caching off in order to see all hits.

Re:We need web caches

[greenrd]

Exactly. It doesn't make any logical sense whatsoever. It's completely absurd. It was obviously dreamt up by a lawyer with no clue about how email, Usenet etc. work.

Re:We need web caches

[Cloudmark]

While it does sound illogical and I've yet to see any firm material supporting it, it is nevertheless a concern raised by our corporate lawyers as a consequence of storing copyrighted material on the cache. As I understand it, they're more concerned with the cache as a means of facilitating copyright violations than the actual material in it. The question that was raised at our last meeting basically came down to whether or not a cache could be legally interpreted as 'serving' copyrighted material. Interesting thoughts and they are significant enough to warrant a lot of investigation by ISPs.

As for whitelisting, or only caching certain files, that is being investigated to get as much benefit as possible from caching while protecting themselves.

Right now, there are a lot of new laws out there and they're still being interpreted. Until they are, companies are always going to err on the cautious side.

~Cloudmark

Re:We need web caches

[Sloppy]

I've been using squid for a couple of years, and the stale-page scenarios you described, just don't happen. I'm beginning to suspect that the whole technology has gotten a bad reputation from one bad implementation (AOL?).

Re:We need web caches

[stephung]

I don't think this is true, otherwise Google cache would have been dead already. If google can cache them, why can't ISP cache them as well?

Too much bandwidth?

[Anonymous Coward]

You mean all that extra bandwidth won't go to more ads and porn?

Re:Too much bandwidth?

[heyeq]

well, let's hope it's not ads ;)

AT&T Blocking P2P?

[Valiss]

I have AT&T broadband here in N. Calif and I've notice that I have not been able to connect to Kazaa, iMesh, or any other p2p network for weeks. Is this because they have put a block on it (because of the network usage)? Has anyone else ran into this?

Re:AT&T Blocking P2P?

[Anonymous Coward]

You would think that AT&T would leverage their position somehow (thru tiered pricing if necessary, I suppose) as one of the only players in the game able to to make money off of the Kazaa's of the world by being able to supply this demanded bandwidth.

Re:AT&T Blocking P2P?

[Valiss]

Yeah you would think that, but as I'm sure you know TheMan(tm) isn't always as smart as they would lead us to believe.

And actually I incorrect in my original post, I can connect, but all searches return 0 (zero) results, not matter how broad I make my search terms. Still, I don't know if it's my house (my roomates ran into the same problem) or if it's AT&T.

Re:AT&T Blocking P2P?

[Anonymous Coward]

I just switched to AT&T Broadband in Sacramento. I'm unable to search kazaa's network, but people are able to dl from me.

Re:AT&T Blocking P2P?

[Valiss]

Actaully I am in Sacramento as well although I restricted uploads to zero (to save my bandwidth) for the time being. Maybe it's a Sacramento AT&T thing...

Really bums me out.

Re:AT&T Blocking P2P?

[martissimo]

there is a big thread thats been going about this situation over at dslreports.

if you dig a page or two back into the thread quite a few users seemed to have success by using some http tunnelling software so they were on a non throttled port. might just find an answer to your problem in there if ya look

Re:AT&T Blocking P2P?

[Valiss]

That'sa great thread. Thanks!

Re:AT&T Blocking P2P?

[Anonymous Coward]

http tunnel is a great way to pass the usual restrictions, but use tons of bandwith. www.http-tunnel.com

so what?

[stratjakt]

They sold me 'unlimited monthly internet access', which is wrong on its premise (there's a limited amount of minutes in a month, and I can only connect once, giving me a very real 23328000000KB limit - based on 150k/s in a 30 day month)..

Who are they to place more limits?

The argument of "gawrsh! its awfully expensive to give you what you payed for" doesn't fly with me.

They just cover their asses with a completely arbitrary and vague AUP, which basically says "we can deliver you whatever service we feel like"

The only way to fix this nonsense is some competition.

Where did my bandwidth go?

[jbischof]

I would just be happy if people would realize that 10Mb/s and 10MB/s are two different things. They can never achieve anywhere near 10 Megabytes a second through their network.

Thats because your network only operates at 10Megabits a second!!!.

ah, bits bytes, whatever. Its all just techno babble to them. It isn't our fault that networking equipment reports speed in bits and downloads report speeds in bytes.

Re:Where did my bandwidth go?

[linuxwrangler]

Aha - now I know where it went! Microsoft Word must have autocorrected everyone's 10MBs to 10Mbs.

Re:Where did my bandwidth go?

[Pig Hogger]

"Never let your sense of morals prevent you from doing what is right" - Hari Seldon

Er, it was actually Salvor Hardin who said that.

Not that new

[RollingThunder]

"This 4000% overhead is annoying but tolerable on lightly loaded networks."

(From RFC896)

Of course, that's talking about bytes of overhead vs bytes of real data - there would be much less than 4000 packets per packet containing real data.

Re:Not that new

[Anonymous Coward]

yes, you are corerect about the nimroding.

elementary...

[O2n]

The whole article fails to come with any credible argument for the overhead theory; except, of course, the fact that 1% of a big number is, well, greater than 1% of a smaller one, I don't buy for one second the idea that keepalive and administrative packets can add up to 0.1% of the traffic.

You get more than that just from the TCP/IP headers.

Re:Bugtraq says logout.

[joe52]

This is an optional way of logging into Slashdot. Go to your preferences, then to the password page.

There is a line of text that says
"You can automatically log in by clicking This Link and Bookmarking the resulting page. This is totally insecure, but very convenient."

If you look at the link, it is pretty much the URL that you have noted. It looks like this is not a bug, but rather a very poor feature that the authors know is insecure, but have chosen to retain. At least using it is completely optional.

Efficiency is on some of their mind's...

[Anonymous Coward]

They have to deal with 33.6k and 56k users quite a bit, so there isn't tons of bandwidth always available. As a result, many of the developers (atleast those with a clue) do take this into consideration.

Personally, I'm more bothered by the number of websites that don't use gzip in some form. The bandwidth savings are worth the cpu costs.

Text compresses

[Outland Traveller]

Text compresses really well. It's pretty foolish to sacrifice maintainability of your page code to save bits. There are better solutions.

Re:Text compresses

[Winterblink]

Oh I agree text compresses well, but I'm talking about overall savings on the order of maybe a few k or b a day on a page. For sites like say Yahoo or (gag) MSN that get many thousands of hits a day, perhaps the cumulative savings will add up.

Re:Text compresses

[EnderWiggnz]

no, there arent better solutions.

as i stated earlier, go to google and look at the source - one letter field names, and no line breaks.

why? TO SAVE BANDWIDTH.

every byte is sacred... every byte is great...

and if a byte is wasted, CFO's get quite irate...

Re:Text compresses

[Karamchand]

well, still it wouldn't be of _any_ use if yahoo transfered 100MB less per day.. this is NOTHING. These things are all _relative_ please keep this in mind!

Re:Text compresses

[Outland Traveller]

I can't believe people mod this stuff up..

The whole point of HTML (and XML) is readability. It is nigh-pointless to invent an ascii-based protocol and then cut it down such that even fluent experts need a special tool to work with it. If that was your goal then you should have used a binary protocol from the start. If you could travel back in time would you have implored the w3c to make the html spec as terse as possible to save on overhead? After all, think of all the countless bytes that "" notation wastes!

Why don't we just program everything in assembly while we're at it? Would you say that assembly is the 'best' solution for limited CPU power? Certainly not. There's a place for this kind of penny-wise pound-foolish behavior, but it's certainly not generalizable.

Holding up Google as a proof of your argument that unreadable HTML is a superior solution doesn't mean anything, for obvious reasons. It's interesting to see that they have chosen that tactic the meet their particular set of needs, but that's as far as it goes. I wouldn't be surprised if google's internal HMTL is more readable, but is "compiled" down into a sparse unreadable block for production use. Do you think this added step is worthwhile for most people?

The best compromise is still compression. A fast compression algorithm like LZO (or even gzip) can reduce bandwidth even more than twinking out your variable names and spaces AND still provides readability.

Re:Text compresses

[Proc6]

I agree. All of this kind of talk is just like everyone standing around cheering for a car that gets 50 mpg instead of 40 mpg. In the end, that is not the solution. 100 years from now we won't be in paradise because we all used shorter html form variable names. It was the ease of HTML that made it popular anyway. In addition, its like trying to put out a forest fire with a small blanket. While all the geeks are clapping their hands as they save 1% bandwidth by putting all their HTML on one line with no line breaks, the rest of the world is forging ahead with new ideas like remoting, and XML, etc, etc.

Focus on making the NEXT technologies the best they can, not retarding the ones we have.

Re:Text compresses

[uberdave]

After a certain break-even point Forth becomes better than Assembler.

Re:Text compresses

[Anonymous Coward]

The whole point of HTML (and XML) is readability.

Sorry, but... who cares if it's readable to you? Place I used to work, before publishing a change to the site, we ran static HTML through a compressor that shorted field names, removed extraneous whitespace, etc.

Not a big deal, really. Everything still worked the same, the files just ended up about 20% smaller on average. Net gain was that we were able to handle 20% more clients that we had been able to prior to that. Since we were generating a set of ~15,000 static pages daily, it was a significant savings.

Re:Text compresses

[spectecjr]

The best compromise is still compression. A fast compression algorithm like LZO (or even gzip) can reduce bandwidth even more than twinking out your variable names and spaces AND still provides readability.

No, the best solution is to use ASN.1 encoding, with an editor that understands ASN.1 as its backend format.

This problem was solved years and years and years ago.

Simon

Re:Text compresses

[Outland Traveller]

Good answer, but I don't think that was the question.. Given that http/html already exists, how would asn.1 improve it today? What I meant was that straightforward, stupid, lossless compression is a better solution for today's http/html bandwidth problems than obfuscating html.

Re:Text compresses

[Anonymous Coward]

The whole point of HTML (and XML) is readability. It is nigh-pointless to invent an ascii-based protocol and then cut it down such that even fluent experts need a special tool to work with it. If that was your goal then you should have used a binary protocol from the start.

Yep. We should have used a binary protocol from the start.

Why don't we just program everything in assembly while we're at it?

Because we have compilers. Likewise we should have compilers for html pages.

Re:Text compresses

[EnderWiggnz]

binary stream?

html was developped on unix - we dont do binary ANYTHING....

Re:Text compresses

[quinkin]

It's interesting to note that /. introduced a similar html pruning system (removing line breaks, white space etc.) and later removed it after being flooded by complaints from users.
I guess it is a trade-off between providing a service (even if the service inadvertantly includes html authoring tuition through example) and the costs of the service provision.
As google is the most widely used search engine in the world, and it has little or no html layout of any quality worth emulating, I can't blame them for pruning where possible. It certainly speeds up all of our searches.

Re:Text compresses

[Anonymous Coward]

> The whole point of HTML (and XML) is
> readability.

And that's why web pages are huge, take a long time to download, and XML/RPC is freakin' slow. Here's the deal: binary makes SENSE going over the wire.

> A fast compression algorithm like LZO (or even
> gzip) can reduce bandwidth even more than
> twinking out your variable names and spaces AND
> still provides readability.

So...you want to turn the ASCII into binary and add some performance hits? Sweet!

Re:Text compresses

[Anonymous Coward]

so you're saying if i walk into a bank and ask for $5 for free, they'll give it to me because it's insignificant, and they won't really notice the loss?

Re:Text compresses

[spectecjr]

html was developped on unix - we dont do binary ANYTHING....

Case in point: The original X Windows image format.

It's amazing what people come up with when all they have is a console window.

...but it'll bite them in the end.

[The Fanta Menace]

The more they cap usage, the less people will use (obviously). Then content providers such as streaming radio stations will start to drop off as it becomes more expensive for users to access them.

After that it becomes a vicious circle, with fewer content providers, there's no reason for users to keep their service. Then the ISPs go broke.

Take a look at the Australian example. Almost all broadband providers have a 3Gb monthly cap. The ABC has just started an internet-only radio station, but I really wonder why. It wouldn't take too many days of listening to it for a user to totally max out their cap. I predict the station will be closed due to lack of interest, within a year.

Re:...but it'll bite them in the end.

[AnnaBlack]

I never understood the economics of internet (streaming radio) anyway. Consider the difference between (radio) broadcast and internet in terms of cost per listener. With broadcast, you have a relatively big initial cost for the transmission kit, but your cost per additional listener is then zero. With internet streaming, you have a lower initial cost, but each listener consumes, say, 40Kbps of bandwidth. Each and every listener. So your bandwidth costs rapidly become the dominant factor.

And yes, there's multicast and the MBone... but they're not used, are they?

Anna B

xolox was a huge offender

[teamhasnoi]

Back when xolox (for windows) was in ver 1.2, it would requery *forever*. Althogh this worked great for getting results, I am sure it killed the network.

They have seemingly 'fixed' this in the new release, but it now has banner ads and popups all through it. Ug.

It's pretty good, even though they have some catching up to do. (They went down for awhile for fear of getting sued alá napster.)

Re:xolox was a huge offender

[Anonymous Coward]

Xolox is a Gnutella client. I use a real Gnutella client. I cut you.

hrmm yeah i guess so

[digitalsushi]

as an ISP i can say that we make our money by a gamble that people use X amount of bandwidth. p2p breaks our precious little ratio of what we expect and what we need.

the geek it me though, says "waaa" and that things that dont evolve, die. and the things that dont die. p2p pushes the envelope right now, but all that encourages is more network growth. just think of p2p as those pains you had in your legs when you were 14. sure, it may not be the most efficient thing in the world, but the underlaying infrastructure has to take that into account, or get out of the way for one that can.

Re:hrmm yeah i guess so

[Fastolfe]

p2p breaks our precious little ratio of what we expect and what we need

Uhh, yah, except this is how they're determining how much they can charge you. If the ratio becomes permanently skewed, the way they "evolve" as you put it is to simply skew their prices to compensate. Though your end user connections may be effectively "unlimited", someone upstream pays for the bandwidth by how much data gets transferred. I guarantee the costs will filter down.

So as a business, what would you do? Raise your rates for all "unlimited" customers? Create a new class of DSL customer with a lower bandwidth cap and re-figure the ratio? Block P2P activity entirely? Write into the end user contract some soft usage caps and go after the top 1% of bandwidth consumers? All of the above?

I don't really think P2P is going to drive growth (i.e. more bandwidth for less cost) any more aggressively than the growth we're already seeing. I just think it's going to annoy ISP's and make them re-think some of their "unlimited" bandwidth plans.

Re:hrmm yeah i guess so

[Anonymous Coward]

But you've seen how people react on here, the thought that they should have to pay more for their 'God given bandwidth' just because their raping the service provider makes them bitch like little girls.

It seems like ISP's can't win. :/

A little off topic but I'll ask anyway

[Anonymous Coward]

Here at PSU we're limited to 1.5 Gigs per week. Now I'm not using any P2P sharing software but is there anyway to tell exactly how much bandwidth I'm using? I'm ashamed to say I'm using WinME.

Re:A little off topic but I'll ask anyway

[Anonymous Coward]

You should get a firewall that keeps track of it. Especially considering you're using ME you need a firewall. 1.5 gigs? That's horrible, how can you survive? I'm at CMU, and the computer I'm typing this on (I have two and the other one does all the P2P) has transferred more than 20 gigs this week.

Burned our office once

[RollingThunder]

The office I'm at used to have a contract with a monthly cap - a mere 20GB, with fairly hefty per-GB fees after that.

One Monday morning, I came in, and glanced at the MRTG graphs over the weekend. Keeripes! Somebody had been pushing data at about 250Kbps from Friday night until about 6 PM on Sunday, sustained.

I did a quick calculation, and then informed the bosses that we were going to be paying a lot more than usual this month, and asked if they wanted me to find out why. Of course they did.

Turned out it was one of said managers. He fired up Limewire, grabbed something on Friday, and forgot to shut it off. Seeing our nice low-latency, high capacity link (E10 or thereabouts, just with a really low traffic cap), it went supernode... and we paid about twice the usual for it.

Re:Burned our office once

[afidel]

We had the same problem. Although our multi-T1 connection is not metered we did have it brought to its knees for almost a day when one person set kazaa to be a supernode. I got a very angry call from the wanops people to go tell this user to knock it off and that they would recomend disciplinary action if it happened again.

Re:Burned our office once

[Anonymous Coward]

Wow. I can see a little notice going up at work.
YOU ARE FIRED.

Re:Burned our office once

[mr_exit]

we had someone visit a site with a really badly set up ad system, it kept refreshing the image every 14 mili seconds..
he managed to pull 8 gig overnight... and at 8c a Meg that gets expensive

Re:Burned our office once

[harmonica]

It's erased now that he didn't race Needles.

nope, he's not insane.

[EnderWiggnz]

go look at the html code from google - notice how they abbreviate every object name to ONE letter in the interest of bandwidth.

i'm sorry that you learned how to code sloppily, and are bitching about streamlining code for efficiency, and cost savings.

most of us dont need the damn hungarian notation that MS has spreads like gospel truth. It makes for unreadable names that convey less meaning that a nice clear variable name.

oh - and i know when to use a goto to streamline code, too :-)

Re:nope, he's not insane.

[EvanED]

>>most of us dont need the damn hungarian notation that MS has spreads like gospel truth

Why said anything about that? And besides, MS now discourages its use.

>>It makes for unreadable names that convey less meaning that a nice clear variable name.

Which 'fn' is not but 'FirstName' is.

Now, if you have a dynamically generated page, you could use constants that are set to short stuff like 'fn'. Less code to be transmitted while still keeping most of the readability of the original code. If you discover a bug, temporarily switch to a different set of constants ('FirstName' instead of 'fn') until you sort it out so the resultant HTML is more readable. (Same goes with whitespace: Make a constant ENDL or NEWLINE that is set to '\n' while debugging, then changes to '' for production.)

Re:nope, he's not insane.

[CableModemSniper]

plus I am sure it is quite simple to write a quick little app to convert your Human readable code into small fast code (shortening variable names, removing unesscessary white space, etc.) The best of both worlds, maintainability and efficency.

Re:nope, he's not insane.

[Tony Hoyle]

>>>most of us dont need the damn hungarian notation that MS has spreads like gospel truth

>Who said anything about that? And besides, MS now discourages its use.

Got a reference for that? It's shoved down our throats whether we like it or not simply because 'it's the MS way'. It's damned impossible to debug something called 'lpszglname' especially when it isn't even a string any more because it was changed years ago...

Re:nope, he's not insane.

[EnderWiggnz]

MS is discouraging hungarian notation?

really? do you have a source for that? i'd be really happy if they did, but i fear the legions of VB coders out there wont be able to break old habits...

i assumed thats what you were talking about with teh "readable code" thing...

but yes, in general, you're right, the variable name "FirstName" is MUCH better than "fn" - except when variable names are transmitted with html.

i do like your solution with the production and development version though.

ahh well... clear names are good - hungarian bad, and asm like names are needed to save bandwidth for extremely high-traffic sites...

Re:nope, he's not insane.

[bopal]

the question is not if other people can read your html, only your browser has to understand it. and i don't think that mozilla cares if you write fn instead of FirstName.

Re:nope, he's not insane.

[notsoanonymouscoward]

I wouldn't consider most of the programmers i know to be human. we all have lexers, parsers, and symbol tables built into our brains for various programming languages. problem solved.

Re:nope, he's not insane.

[kaphka]

Got a reference for that?

RTFM.

• "
• Do not prefix parameter names with Hungarian type notation.
• "

Re:nope, he's not insane.

[Anonymous Coward]

If a site changed FirstName to fn and served a million pages a day, that would save 7 megabytes! Say you could save 7 times as much bandwidth per page and served 70 times as much traffic.

That's a savings of 3.5 GB, the same amount of bandwidth sucked by one Gnutella user downloading DIVX movies all day.

Re:nope, he's not insane.

[CyberKnet]

go look at the html code from google - notice how they abbreviate every object name to ONE letter in the interest of bandwidth

That's all very well if you only have a low number of objects on each page, but when the objects move into the hundreds you're likely to run into a little more trouble whilst keeping track of all said objects. After all, not every application is a search engine.

i'm sorry that you learned how to code sloppily, and are bitching about streamlining code for efficiency, and cost savings.

That's highly subjective, at best.

There are benefits other than bandwidth saving to take into consideration. I.E. The time it takes a new coder to become familiar with a new (Web) application, ease of documentation, and debugging.

most of us dont need the damn hungarian notation that MS has spreads like gospel truth. It makes for unreadable names that convey less meaning that a nice clear variable name.

Explain to me exactly how hungarian notation makes for unreadable names that convey less meaning?

Hungarian notation would add scope and type (depending on the strain) information (in lower case) before your "Clear Variable Name" (in title case), and not take anything away from it at all...

I've personally have never seen Microsoft spread hungarian notation as gospel truth. I find this most evident by the fact that Microsoft documentation / code samples in MSDN sways from using it partially, completely or not at all.

However ... despite all this, if you have any factual sources or meaningful (read: backed up) statistics to confirm what is so far just libel and slander, please do post again...

This has been known for a long time...

[fortinbras47]

With gnutella, QueryHit packets can make up as little as 1% of traffic (by numbers of packets, not size) while Ping and Pong packets can be well over 50% of packets. Check out this article to see more detail.

Gnutella is not one of the more advanced protocols, but most of it's problems are present at varying levels in other p2p systems. It's not really surprising that P2P software which spends so much time trying to connect to computers, connect to a computer to start a download etc... and search in a geometric spiderring fashion are quite inefficient.

Re:This has been known for a long time...

[krmt]

That article is from March 2001, which is very out of date. Since then, a lot of development has occurred, most notably the development of supernodes by Limewire, gnucleus, et al. In addition, a new ping and pong scheme has been proposed.

Bandwidth Division

[T-Kir]

Damn dialup crappyness, retype whole message time :-(

Question:

Isn't the excuse of capping broadband connections a moot point, because the general broadband thing is that it is a shared resource? So X ISP saying that 1% of their users are hogging 60-70% of the available bandwidth, then they use that to say 'Right, we're raising prices', in the sense that if there is any load balancing, then the other 99% of users would be able to level up the bandwidth if they needed more, so it divides up (theoretically equally)??

Although I'm spoilt rotten living with my brother in CT, cos the Optimum Online connection (around 5Mbit at it's fastest) has been no trouble at all, damn UK rural 'broadband' (or lack of). Oh well, I suppose I don't have many new Farscape eps to download when they come out :-(.

Just my 2 pence.

Re:Bandwidth Division

[Anonymous Coward]

I suppose I don't have many new Farscape eps to download when they come out

Did ya stop to think that maybe if you watched the show on television instead of downloading it, it might still be on the air?

Re:Bandwidth Division

[Anonymous Coward]

hooray for obscure red dwarf quotes!

Re:Bandwidth Division

[Alan Shutko]

Maybe, if you're a Nielson family. If you aren't, how you watch a show will have no impact

Re:Bandwidth Division

[T-Kir]

Or how about:

Purple Alert... less important than a Red Alert, but more important than a Blue Alert... maybe it ought to be Mauve Alert.

X.25 hax0rz

[drwho]

Back in the heyday of "X.25" networks, there were a lot of illegitimate users. There was inadequate technology to protect and track.

It is rumored that there are accounts on public x.25 networks, belonging to large corporations, that have worked for over 13 years.

No

[Anonymous Coward]

I live in The north bay and have AT&T, no problems with gnucleus. Maybe you pissed someone off ;)

Maintenence required to optimize bandwith

[Anonymous Coward]

This is why Internet Cleanup Day is so important!

So, P2P costes more

[Anonymous Coward]

"have gave much thought to"

P2P costes mo dan it be shuldn ta.

Dat be why my isp be gettin down on me all da time

modern p2p sharing apps have bandwidth limiters

[Foresto]

Peer to peer file sharing apps are still relatively immature, but they're getting better in a lot of ways, including bandwidth usage. In fact, all of the new p2p programs I've looked at have built-in bandwidth limiters, and they seem to be improving with each release. WinMX, Overnet, edonkey2000, and Shareaza (a gnutella app) are some of the best at achieving their goals, and they all include bandwidth limiting options.

The Real issue...

[Quasar1999]

ISPs are putting bandwidth caps on accounts because they see it as a source of revenue. Plain and simple. The crap about how 5% of the users use 95% of the bandwidth is really starting to piss me off... they advertised always on, unlimited bandwidth when I signed up, and now they have enough customers used to the speed, so they essentially upped the price (just like soup companies reduced the size of their cans of soup, but kept the price the same, if you want more, buy a larger can...) if you want more bandwidth, upgrade your package, or better yet, pay $7.95 a GB/Month over our generous 3 GB/month...

Isn't there a law against doing this sorta crap? They said always on, unlimited bandwidth... now they're charging through the nose, claiming crappy stats on usage, and blaiming it on p2p networks... I can't even download my legitimate MSDN ISO images without going over my monthly bandwidth limit, let alone actually doing anything else on the net...

End rant... :P

Re:The Real issue...

[Anonymous Coward]

. . . I can't even download my legitimate MSDN ISO images . . .

MSDN? You're posting to the wrong place buddy.

Re:The Real issue...

[jbolden]

And during the time they advertised unlimited bandwidth they delivered it. They didn't advertise "unlimited bandwidth for life never to be revoked". And I know for a fact that MSDN will send you CDs monthly so there are some gigs you can save right there.

Anyway there is no magic bullet here. A small number of customers are using way more bandwidth then they are paying for. The ISPs can either raise prices for everyone or get those customers to pay more.

As for $8 per gig, that is insane and I'd talk to the PUC about it; that's well over their costs.

There will always be a use for bandwidth

[gtaluvit]

It's like saying 64k will be enough for everyone. As bandwidth and speed increase, people WILL find a use for it. Its how the computer industry survives. No one needs a 3ghz Pentium, unless you want to run Quake 3 with fps in the triple digits, or you want to compress a DIVX even faster. So when bandwidth increases, expect your Mp3's in less than a second, a full length DIVX in a matter of minutes, or your linux iso faster than you can burn it.

Lies! All lies!

[joebp]

The sys admin realized that if he just reduced the frequency of keep-alives, he could shave something like 10% off the monthly bill.

Translated: he cut down on pr0n.

Comcast where I live..

[stratjakt]

..is airing this commercial of goofy testimonials for their broadband cable service. A kid says "Ever been in the belly of a whale? I have", another guy goes "I go to the moon and back twice a day", etc.. etc..

Now, one of them has some guy say "I collected everything Mozart ever did... In 10 minutes!"

To me that's comes through loud and clear as "*wink* *wink* *nudge* *nudge* napster(etc)!"

I would say p2p is the driving force behind non-geeks getting broadband. They don't need it for e-mail, or casual web-surfing. They don't play games, but I know many people eager for an alternative to the bland junk on the radio. (Plus due to geography, radio reception is poor here)

Same thing with the 'work from home' bunk they promote, and yet block VPN connections.

It's like dangling a carrot in front of a mule to get him to move, and he stupidly chases it not realising he'll never reach it. It works fine in cartoons, but eventually the mule becomes frustrated, kicks you, and refuses to move at all.

Someone is smart enough to figure a way to give out the bandwidth and make money at the same time. And, it won't be a monopoly. Maybe 802.11 will be our savior?

Re:Comcast where I live..

[Com2Kid]

To me that's comes through loud and clear as "*wink* *wink* *nudge* *nudge* napster(etc)!"

Oh hell it napster's heyday TCI was running commercials actively PROMOTING their cable modem service as to be used for Napster.

(err, TCI was bought out, heh, kinda explains why I guess. . . .)

Of course the 6 o'clock news was also showing tutorials on how to download files from Napster as well so. . . . heh. (that was kinda when I realized that the entire Napster thing had gotten waaaay out of hand)

Re:Comcast where I live..

[CrackersnSoup]

Maybe 802.11 will be our savior? Not Likely, 802.11/802.11a/802.11b just doesnt have the bandwidth. 11mbit at the radio, 5.5-6mbit read data and less per customer thats on. Canopy on the other hand, can reach good DSL speeds(100-200KB(800-1.6mbit) and thats the first gen hardware, They have some nice PtP 2nd gen coming out and there is talk of allpying that to there PtMP line aswell. 802.11g will be a huge help, but it will stil lsuffr the degedation in performance every time you add a customer.

Re:Comcast where I live..

[Malduin]

Same thing with the 'work from home' bunk they promote, and yet block VPN connections.

Hrmm.. I'm on Comcast and have no problems with using a VPN. All I have is a regular residential plan and I use a VPN from work to home on a near daily basis to sync up the work I do at home with the work I do at work.

Re:Comcast where I live..

[xenoweeno]

Now, one of them has some guy say "I collected everything Mozart ever did... In 10 minutes!"

To me that's comes through loud and clear as "*wink* *wink* *nudge* *nudge* napster(etc)!"

One bit of bad news: Mozart's work is in the public domain, and has been for, ah, quite a long time.

Re:Comcast where I live..

[ChadN]

But specific recordings of Mozart's work may be copyrighted (and most likely are, if it came from a CD originally).

Re:Comcast where I live..

[JMan1]

Mozart's work itself is... too bad there aren't too many recordings of him actually playing.

Re:Comcast where I live..

[kubrick]

Mozart's work is in the public domain, and has been for, ah, quite a long time.

Surely recent performances are still protected by copyright, though?

Re:Comcast where I live..

[Jeppe Salvesen]

I would say p2p is the driving force behind non-geeks getting broadband. They don't need it for e-mail, or casual web-surfing. Hardly. Have you tried suring on a 56k lately? The web sites are becoming ever more image-laden and bloated. Image swapping from their digital cameras. Plus - the fact that broadband doesn't block your phone line like a plain old modem.

When the MPAA gets their heads out of their asses and start providing decent pay-per-view and/or pay-per-download solutions, broadband will become ever more popular.

Quite Sane really

[ACNiel]

I couldn't disagree more.

I just got into an amazingly poorly written program after about a year, and was bewildered by the names, and what they really meant. And it was my code. And yet, I couldn't disagree with you more.

The streamlining that was discussed by the parent isn't for the sake of the coder, it is for the sake of the user. Mostly your argument is founded in long names really don't hurt anything. And if that is true, than long, descriptive names do their job. Here is a prime example of where they do make a difference. Here the variable names (for variables, and even javascript, or vb script embedded in a page) could make a tremendous amount of difference. And that is wht you would be stream lining for.

As for superfluous naming, well that can be just as bad, and unreadable as short names. Addled is addled, and you can use short (maybe more than 3, this isn't RPG afterall) descriptive names, without having to type an entire sentence.

junk1, junk2, junk3 will never be a good idea, but if your form has 4 variables, and you name them
FNm, LNm, MI, and Age, I don't think anyone will be confused.

gif, png

[error0x100]

Agreed, some pages have pretty horrifically unoptimal html. I've seen HTML files that are 40% or more just *redundant whitespace* (e.g. from heavy indentation, spaces, not tabs, that sort of thing).

I suspect a fair amount of bw could also be saved if webpage developers replaced all their non-animated GIFs with 8-bit PNGs. I replaced all the GIFs on my web page (probably about 50 or so) with PNGs, and it reduced the average size of the images by about 30-35%. In some cases files were almost 50% smaller. Not once was the PNG larger than the GIF.

Of the tools I've tried, the ImageMagick (which comes with RedHat and is available on Windows too) "convert" utility seems to consistently produce the smallest PNG files (Photoshop's are always slightly larger because they add "Adobe" and some other crap to the header comments).

Its easy to convert large numbers of files using tools like ImageMagick, so there really is no excuse for web page developers not to use it. Photoshop also has a batch-converter, AFAIK.

"Web browsers don't support" is not an excuse anymore, unless you have alpha (specifically multiple levels of transparency), which some of the older browsers didn't handle so well.

Many web pages have simple, no-transparency, no-animation GIFs. These could all be PNGs. The site http://www.worldofspectrum.org/, as an example, has an archive which includes screenshots of thousands of ZX spectrum loading screens etc.

But on the other hand ...

[snowtigger]

[Extract from a swedish newspaper article last friday]

At least 80% of the traffic from broadband users comes from downloading proprietary files. To stop this would strangle the usage of broadband according to some Tele2 official.

This behavior saves the ISPs (already in trouble) and the music industry should think over their business models. The Music industry has an incorrect pricing and has failed to adapt the new technologies.

I can totally understand.

[Arcaeris]

I can totally understand the limitations of bandwidth in the face of 2p2 software.

I was in my first year of college (living in the dorms) when Napster became popular. That same year, they banned it from all campus computers. The IT guys here said that of the estimated 7200 dorm room computers on campus, a minimum of 6500 were running Napster at any given time. They were forced to ban it because the bandwidth usage was taking away from vital staff/faculty related web-based tools and network services that needed to be maintained. In fact, nothing else could be run on the network.

Now Napster's gone, and I haven't lived on campus since Kazaa and such became popular. I'm pretty sure I know how they're dealing with it.

If one university had to do it, then imagine what the average cable/DSL provider has to deal with. Granted, they don't have as much essential network stuff.

Re:I can totally understand.

[stratjakt]

Yes but, (if your University was like mine), you don't pay for it. Access to the universities private network was a privelege of living on campus. They were bound by a fixed budget that came out of our tuitions/res fees and had to accomodate everyone. Our house, our rules..

I pay for my cable powered internet. I don't see their right to tell me what I can and can't do with it, it was part of no contract I signed, save some ambiguous crap about removing "abusive" users at their discretion.

I made another post in this forum about how they use p2p and VPN as incentives to sell the service. Bait and switch.

The business model in short (and not a lame SP troll):

Split a 10mbit pipe over 1000 users. Most only know how to read e-mail and read dilbert cartoons so they'll never notice we oversold ourselves. Kick the few that will off, cite bandwidth abuse as the reason. (How you 'abuse' something they sold you unlimited access to still escapes me)

The 'stupid sheep' they counted on forking 40-100 bucks a month for something they'd never use, found something to use it for.

Monthly download limits?!?

[idleprocess]

All you have to do to blow your monthly download limit with Sympatico or Videotron is to leave your computer running with a P2P file-sharing program turned on and some files to share.

Do any of you have Sympatico or Videotron and if so what happens when you hit the 5gb limit? This would really suck and if SBC ever adopts this I'm gonna be... well... bending over and asking for more cuz there is no alternative for static IP DSL connected to a remote terminal 1000' from my home. Damnit!

What about all the wasted *unused* bandwidth?

[Anonymous Coward]

If a second ticks by, and a network is underutilized, that unused bandwidth can never be recovered.

Very sad.

Just think of all the terabytes of data that could have been transmitted, on a network that at the time was likely underutilized, but weren't.

Four? Or Millions?

[glubbs]

"Pump "Britney Spears" into Kazaa's search engine, and try to imagine the chatter you generate as you check millions of shared directories."

Aren't you only connected to four other computers (servers?) at a time? Or am I mistaken, and you actually search recursively through the servers that the computers you're connected to are connected to?

Re:Four? Or Millions?

[MxTxL]

The 4 machines you are connected to perform your search then pass it on to the computers they're connected to and so on... i think there is a setting you can change, but generally searches go something like 8 levels deep.

Streaming Real TV

[silas_moeckel]

As an engineer that made a network to do that that tanked (Nothing like lies from sales people) it's possible 512 kbit a sec looks pretty nice but the bandwith costs on the sending end are about 50 bucks a month before servers people etc (thats sending all month) why because it's all unicast because NO ISP wants mcast working outside of itself they dont know how to bill for it. A satalite at 500 an hour is much cheaper than delivering over the internet and inherently multicast. I wonder when somebody will come up with a multicast service that is delivered to a majority of ISP's.

Efficiency is a huge concern

[Anonymous Coward]

I doubt that developers of those free p2p applications have gave much thought to efficiency

...at least for Gnutella developers. Browse the messages in the Gnutella Developers' Forum. Pick random dates, go back 3 months, 6 months etc. Efficiency is practically the #1 issue.

I know where a lot of bandwidth goes...

[NineNine]

I know where at least a few hundred gig a week of bandwidth goes to...

Re:I know where a lot of bandwidth goes...

[sheddd]

I know where about 50 TB/week go :)

Coolmon screenshot:

http://www.yerhost.com/sshot.gif

Re:I know where a lot of bandwidth goes...

[sheddd]

Grr you /.'ers are a curious bunch. www.yerhost.com log excerpt:

09-11 23:54:24 192.168.1.2 GET /scripts/..%5c%5c../winnt/system32/cmd.exe /c+dir 80 - xxx.xxx.xxx.xxx - 404

Exploiter attempter's ip replaced with x's. I'm running .NET Beta 3; hopefully most've you don't know what to do with it yet. And if you do please be nice :)

sounds like

[Gavitron_zero]

the big hog on network bandwidth is TCP/IP...big surprise there.

I can absolutely believe this

[Zenithal]

Gtk-gnutella will explicitly inform you of how much bandwidth it's using at any given time for simple chatter and underlying transaction. It also lets you cap that transaction bandwidth.

By default though, it's set at 5kb/s (as in kilobytes I believe) up and down (each). Couple quick smack's on the calculator shows that's about 350 meg/day. With no transfers.

I can only imagine that other clients aren't being as polite as gtk-gnutella, so I think those figures are 'plausable' if not completely accurate.

This actually goes beyond simple theory. With the traffic at 5 in/out the additional lag is VERY noticable on my machine, and others inside my local net. Probably that's due to my very limited upstream, but still.

HTTP itself isn't byte-optimized

[Media Tracker]

HTTP itself is text-based, and that alone increases the transfers greatly. I guess the main motivation behind such a design was that it makes it much easier to read, and to code HTTP applications; it also eliminates several low-level problems such as byte ordering. HTTP is very easy to pick up, and just any TCP/IP tool (such as telnet) can be used to debug an HTP app. I don't know what the other arguments for such a design were, but that certainly doesn't save bytes. If HTTP was encoded down to be bit level, transfers would be so much smaller!

But on the other hand, as it was mentionned in other postings nearby, clarity should be more important than efficienty. Where is the balance?

I would say that since bandwidth will always be broader and broader, clarity and ease of use should be more important than efficienty... while keeping it sane!

Re:HTTP itself isn't byte-optimized

[Hast]

If HTTP was encoded down to be bit level, transfers would be so much smaller!

Not really actually. A typical HTTP request result in a HTTP header which is 100B. (Well, the local one I just tested did at least.) Both TCP and IP has headers of 20B each. So those layers has the same order of overhead. And If you have a large file which you send you only get one HTTP header over several TCP/IP packets.

And compared to the amount of data transfered in the data part of a HTTP packet the header is very small.

As youself pointed out, it's nice to have a human readable protocol when you're debugging. (Try debugging a TCP/IP stack for a while and you'll be very happy that HTTP is text based.)

My local ISP is doing just the opposite!

[jdgreen7]

I just received an email today that my DSL provider is boosting my speed to 960/960 from 512/512 at no additional charge! They are lowering their prices for most users, too!

It's nice when a company does something good for its customers instead of trying to screw them over.

who cares?

[Anonymous Coward]

with so much fiber glut you'd think people would be happy that all those extra fiber pulls could/are being put to use.

Wow

[Salamander]

The article itself was kind of ho-hum, but the following part of the Slashdot intro caught my attention:

I doubt that developers of those free p2p applications have gave much thought to efficiency.

Again...wow. One would need to search far and wide, even on Slashdot, to find another example of such absolutely astonishing cluelessness. Timothy has obviously never talked to a P2P developer in his life. Sometimes it seems like efficiency is just about the only thing P2P developers think about, unless someone's on a security/anonymity rant. Little things like robustness or usability get short shrift because so much of the focus is on efficiency. Hundreds of papers have been written about the bandwidth-efficiency of various P2P networks - especially Gnutella, which everyone who knows anything knows is "worst of breed" when it comes to broadcasting searches.

It's unfortunate that the most popular P2P networks seem to be the least efficient ones, and doubly unfortunate that so many vendors bundle spyware with their P2P clients, but to say that P2P developers don't give much thought to efficiency is absurd. They give a lot more thought to efficiency than Slashdot editors give to accuracy, that's for damn sure.

Re:Wow

[Have Blue]

Hundreds of papers have been written about the bandwidth-efficiency of various P2P networks

I beg to differ. Hundreds of papers have been written on the bandwidth-inefficiency of P2P networks, and the fact that they are still being written is evidence that they are still inefficient.

Re:Wow

[Anonymous Coward]

OK, so help. There's damn little info out there about 'best p2p client', 'most efficient p2p client', etc.

What's better than Gnutella? I use gtk-gnutella, and it's "satisfactory" from a usability POV, but...things could be better.

Pointers? Opinions?

Re:Wow

[Salamander]

Nonetheless, people have obviously thought about it a lot, and timothy is still full of crap for saying otherwise. :-P

Re:Wow

[Anonymous Coward]

It's astonishingly clueless, no matter who said it, and the Slashdot editor tacitly approved the content. Try to see the real point, OK?

Re:Wow

[Salamander]

I think the infoAnarchy Wiki covers it better than I could.

Re:Wow

[Anonymous Coward]

And why is there tendency to reply as Anonymous Coward to replies to your own messages? Perhaps afraid to admit your own fallacies or mistakes?

Re:Wow

[greenrd]

No, he didn't write any of the italic text. Can you read English? "An anonymous submitter writes" sounds unambiguous enough to me.

Re:Wow

[Anonymous Coward]

Hi, Dan. Illustrating your hypocrisy again, I see. Are you ever going to provide anything *but* fallacies and lies on the original thread, or have you become an AC permanently?

compression

[Twillerror]

Remeber back in the good ol' modem days. I remember getting 10 k a second on some transfer even with a 56.6.

If a P2P network protocol is text based, say like XML, it should compress pretty well and keep some of this extra bandwith down.

If HTTP would actually support compression natively we could save tons of bandwith in those HTML transfers. The page I'm typing this comment on is 11.1 k. zipped it is 3.5, and I think I have fast compression on. I'm sure the main slashdot page would save even more. Slashdot could litterally save megs a day.

It would simply be a matter of Apache and IIS supporting it. And maybe a new GETC command in HTTP that works the same. The browser would ask if the server supports it, and then go from there. Or try it and if it failed, try it normally. Apache or IIS would be smart enough to not try and compress JPEG, GIF, and other pre-compressed files.

Everything from FTP to SMTP could save a little here and there, which adds up quick.

Perhaps the real answer is to write it into the next version of TCP and have it hardware accelerated.

Re:compression

[bedessen]

The page I'm typing this comment on is 11.1 k. zipped it is 3.5, and I think I have fast compression on. I'm sure the main slashdot page would save even more. Slashdot could litterally save megs a day.

It's not a free lunch, you have to consider the resulting increase in server CPU load. It's probably not an issue for low traffic sites but it's definitely a concern for sites like slashdot.

Re:compression

[ShaunC]

It would simply be a matter of Apache and IIS supporting it

Apache does support it, it's called mod_gzip and Slashdot already uses it. The IIS equivalent (sort of) is called PipeBoost.

Shaun

Re:compression

[cduffy]

If a P2P network protocol is text based, say like XML, it should compress pretty well and keep some of this extra bandwith down.

In terms of compressed size, there's no (theoretical) advantage to a compressed XML stream over a compressed binary stream. The reason that XML compresses so well is that there's obvious redundancy that's easy to "squeeze out". A good binary protocol will have less easily removed redundancy -- so it'll be smaller to start with, but won't compress as well. If both represent the same protocol, in theory they should (post-compression) come down only to the same size.

Also, as mentioned, mod_gzip is available and used today.

Re:compression

[Atlantix]

It's not a free lunch, you have to consider the resulting increase in server CPU load. It's probably not an issue for low traffic sites but it's definitely a concern for sites like slashdot.

the obvious conclusion I would draw then is to store the pages in their compressed form. No extra CPU load except when authoring the website plus lower bandwidth usage. Of course, if the site uses pages that are dynamically created (slashdot included) this wouldn't work.

--Atlantix2000

Re:compression

[Anonymous Coward]

Yes, but there is a tradeoff between spatial efficiency and computational efficiency.

Imagine 100 clients (at 5K/s) connecting within 10 seconds of each other to retrieve an 10K document. Simplifying, and assuming that the server has enough bandwidth to sustain ~550K/s outbound traffic, this means it will take the server approximately 12 seconds to service these requests.

Now imagine the same 100 clients requesting a compressed version of the same document, and assume that the server has to perform the compression on-the-fly. Say that it nominally takes 0.2 seconds to compress the 10K document. For the sake of the argument lets assume that the compression is so good that the time it takes to transfer the compressed data drops to almost zero.

So how does this perform? Consider the following simplified version of events: in the first second, the server receives 10 requests and starts 10 compression processes. Each individual compression process then runs at 1/10th of nominal speed, which means it takes 2 full seconds for the server to respond. But in those 2 seconds, an additional 20 requests come in. And now compression runs at 1/30th of nominal speed. But still requests are coming in, and very quickly, the system becomes completely overwhelmed.

It is instructive to take a moment and consider your mention of specialized compression hardware. A number of network cards do indeed already contain special hardware to e.g. offload checksum calculation from the main CPU. Adding compression hardware is not a farfetched idea.

As a rule however, commoditized general purpose CPUs such as x86, through economies of scale and fierce competition, tend to overtake specialized hardware in a relatively short amount of time. In other words, the fancy compression hardware that you buy today for $5000, may be unable to keep up with the amount of data that your $1000 CPU of next year can throw at it. In this contrived example the only way to justify the added cost is when the bandwidth savings amount to at least $4000: i.e. not bloody likely, given that a lot of the data transferred today (images, video, audio) is already compressed/does not compress all that well.

It all comes down to the question whether paying for bandwidth is cheaper than paying for processing power. At some time in the past both processing power and bandwidth were outrageously expensive. Now both are incredibly cheap.

With developments in processor technology still proceeding at the pace of Moore's law and the telecom industry still reeling from the X.25 fiasco and the blow dealt to them by Internet technology, however, I think we can expect the balance to tip over in favor of processing power over the next decade. It already has, to some extent, in the form of increasingly powerful (and increasingly CPU hungry) compression algorithms, and increasingly expensive forms of bandwidth, such as wireless (with correspondingly byzantine service conditions, just the way the telcoms like it). But the real bandwidth revolution (metered access?) is I think still some way off.

Re:compression

[Anonymous Coward]

Actually its called a check box on the "Compression" control panel in IIS. And a copy of Win2k etc Server.

please learn to use its and it's correctly

[Anonymous Coward]

It's is a contraction for "it is." Its is the posessive. You probably don't accept sloppy code; you shouldn't accept sloppy grammar either.

Once again...

[_Knots]

I say P2P mesh networks (ala Gnutella) need to have intelligent meshing algorithms so that the network tries to minimize the number of mesh links crossing a given physical uplink or a given backbone segment.

Such a scheme would return optimized search results because your net neighbors would know of your query before somebody on the other side of an uplink (and, as there is less routing between you, can transfer files faster in theory).

On top of that, with such a router-aware network the wasted bandwith of broadcast packets multiply crossing a given line due to reflection by peers on the other side would be virtually gone once the network became aware of the layout - ideally each node wouldn't have to learn but could get some kind of topological information from a node it connected to ("You are in the same /x block as a.b.c.d - please connect to that node and drop this connection") or maybe even ask the remote node to preform some kind of query for it ("who wants a.b.c.e, because I don't?"). Our current "host caches" like router.limewire.com could gain some intelligence for whom they introduced to whom.

Instead of capping upload and download capacities as much as done now, perhaps those limits should be relaxed but a P2P "introduction" program installed on the ISP's router so that clients behind the firewall mesh with each other before a few of them send meshing links spanning the uplink.

Yes, downloads will still follow the usual TCP/IP pathways - which we presume are most efficient already. But the broadcast discovery packets which now ricochet around the network would, with an intelligent meshing algorithm, span as few uplinks as possible to query hosts as network-close as possible. All in all this would reduce traffic.

Somebody want to blow holes in this for me?

--Knots;

Re:Once again...

[Salamander]

Actually, it's a really good idea and IMO an area where I believe not enough work has been done. Which is not to say no work has been done. There are several projects aimed at finding the best way to determine network distance, and several more that seek to use that information to create more optimal connection topologies. I don't have my links to that stuff handy right now, but if you send me email I should be able to dig them up.

Part of the problem is that many P2P networks have dependencies on some model or other of the higher-level abstract topology through which they route search queries, and it can be difficult to map (for example) a hypercube onto the actual IP-network topology. Lacking a good solution to that problem of mapping one topology onto another, many P2P developers punt; they try to minimize hops through the overlay network, and vaguely hope that by doing so they'll make up for the extra hops through the underlying network. In many cases it even seems to work, because the search algorithms that operate on the higher-level topology can be extremely efficient.

Nonetheless, if someone could figure out a way to reconcile those advanced search algorithms with a more "reality-based" topology, that would be great. If you think you have ideas on how to do that, by all means explore them. The more the merrier.

Re:Once again...

[PureFiction]

An excellent idea in theory. Now try and code such a beast :-)

Trying to build efficient structure into peer networks is like building a house of sand. They are extremely volatile.

You need to take into account numerous factors, like many orders of magnitude in bandwidth capacity between peers, NAT and unNAT'ed hosts, high churn rates, volatile peer groups, etc, etc.

Most people who have tried to overlay fragile yet elegant topologies on top of peer networks have seen them crumble under volatile real world scenarios.

This is not to say it is impossible, but that it is much harder to implement such a network in today's internet environment than it first appears.

Re:Once again...

[_Knots]

Nat vs unNAT: Treat the NAT as an uplink that we should try to limit connections through.

High churn rates / volatile peer groups: yes, there's a lot of changeover in everything, but I'd wager that copying intelligence on connection ("Here's everything I know about the network around me") would endow newcomers with a good base to start off with.

There's nothing fragile about this topology: it's a runtime dynamic mesh topology - exactly like Gnutella's now. The sole difference is that groups of peers would try to actually group themselves by network-proximity (probably IP range, or for things like Road Runner or at a university, reverse DNS mappings might help). Yes, it might take some more effort from users to specify how to identify members of their local group and get it right. But there are surely some decent ways (IP ranges, as stated) of getting it *usually* right.

It shouldn't hurt the network - it should be an option to turn it off, it should turn itself off if it detects its being unhelpful. The incentive towards the users (inside a university or on a local cable loop) would be much faster downloads due to less routing overhead.

Really you could think of this as ultrapeers agreeing amongst themselves as to which of them will actually route outside a group.

--Knots;

Re:Once again...

[PureFiction]

Nat vs unNAT: Treat the NAT as an uplink that we should try to limit connections through.

What do you mean? NAT users only upload? NAT users only perform outgoing connections? There is a lot of documentation on the experiences developers have had when dealing with NAT. The majority of broadband users are behind NAT, and so using their connections effectively is critical to making a functional network.

NAT also comes into play not only for creating connections, but also maintaining connection state in the translation tables. You cannot even assume connections will stay open for arbitrary preiods of time or else you will inevitably start losing connections...

I'd wager that copying intelligence on connection ("Here's everything I know about the network around me") would endow newcomers with a good base to start off with.

That sounds like a good way to eat up bandwidth. One of the characteristics of most mesh routing protocols is that they do not lower overhead, they simply transition it from the overlay broadcast part of a protocol to the mesh maintenance layer. For example, maintaining optimal mesh topologies consumes a lot more bandwidth than the simple connect to a peer and send a query, pass it on style. So in this manner you are using much less bandwidth for queries, but much more bandwidth maintaining the mesh network over which queries are sent.

There's nothing fragile about this topology: it's a runtime dynamic mesh topology - exactly like Gnutella's now.

Gnutella is incredibly fragile. Gnutella sucks. Gnutella is incredibly easy to cripple with DoS attacks that flood queries into the network (among many other possible methods). So I do not think this is a valid comparison.

If you are relying on volatile peers to forward data through a mesh network you also need to provide robust recovery that does not consume extravagent amounts of bandwidth. Again, this is much harder than you think.

The sole difference is that groups of peers would try to actually group themselves by network-proximity

There is a lot of good info about subtle side effects of assuming Really you could think of this as ultrapeers agreeing amongst themselves as to which of them will actually route outside a group.

Then you are talking about a super node architecture, which is used in a number of peer networks and is much more efficient than broadcast & forward protocols like gnutella. But this is also a far cry from a true mesh routing protocol optimized for efficient bandwidth utilization.

Re:Once again...

[_Knots]

What do you mean?

Well, on my DSL line we were given 10.n.n.n addresses and the NATting was done at some hop down the line - we, two nodes behind the NATting router, could talk to each other as 10.n.n.n as well as our true IPs. Clearly doing the former is much more efficient as we don't have to span the router twice for nothing (in effect - yes, we're probably just hitting its routing tables once, but still, the routers before it probably could have handled the 10.n.n.n instead of the other form).

That sounds like a good way to eat up bandwidth.

Yes, I am aware that it's a risky proposition. Though each node knows very little - its position, its neighbors, and whether it is acting as a ultrapeer for its group. Maybe a little more state information (all members of the group?) but not much. And intelligence would only migrate *after* acceptance into the group by at least one member and/or one ultrapeer member (yes, there can be many).

Gnutella is incredibly fragile.

Well, in some ways yes, but in some ways no. It tries its damnest to stay around and with the ultrapeering technology already present, in addition to LimeWire (and presumably others?) ability to drop most DoS-style packets, it's as much an arms race as the internet in general.

Where Gnutella suffers, though, is its "broadcast to the world" idea. With an intelligent meshing protocol, one could concievably implement a much more efficient "local" routing scheme through some clever manipulation of TTLs inside the mesh and/or some more routing information (at the very least "drop this if you've seen it before," which I think Gnutella does already). This is all optional and maybe not even worth persuing. But the intelligent meshing protocol would prevent a DoS on Gnutella from scaling to a DoS on the backbone segments, ISPs, etc. involved because it would try as hard as possible to have as few reflective connections across a given link.

A supernode architecture, sure. And why not? Gnutella v0.6 already is one, yet its still possibly best described as a mesh with supernodes at the fringes in places.

Alright, I'm really tired. So if none of this makes sense, assume it's my fault.

--Knots;

Re:Once again...

[PureFiction]

Well, on my DSL line we were given 10.n.n.n addresses ...

I hope this was all your internal subnet. No broadband connections that I am aware of provision with internal IP's (except for a snafu at att broadband which transposed internal modem IP's with external client IP's)

In the real world everyone gets a public IP from their provider, and the NAT is usually a linksys cable modem / DSL router. In rare cases you might see linux or bsd systems used as NAT gateways.

You can work around NAT problems, but it is not exactly trivial for TCP.

in addition to LimeWire (and presumably others?) ability to drop most DoS-style packets, it's as much an arms race as the internet in general.

It is vastly different because with a small amount of DoS traffic injected into a gnutella network you can waste very large amounts of bandwidth overall. For most internet DoS attacks you need at least as much traffic to DoS as the pain imposed. I.e. it is a linear relationship, whereas DoS attacks in Gnutella are exponential. The gnutella clients have come a long way, but they will never be perfect because the protocol itself is flawed, and prevents robust resistance to DoS attacks (how can you tell if a given query packet is valid or not? You can't if it is done with any degree of stealth)

A supernode architecture, sure. And why not?

No reason at all. In fact, the super peer networks currently appear to have the optimal balance of decetralization and centralization mixed into one. The limited centralization improves query efficiency and reduces bandwidth, while the decentralized nature of ad-hoc super peer delegation allows for much greater scalability with very little dedicated infrastructure.

These will probably be the "sweet spot" topology/architecture for a number of years...

But this is a good bit different than a true mesh topology designed to optimize routes based on routing locality, etc, that you seemed to mention in the very first post. I suppose you could consider a super peer architecture as something akin to a mesh topology, but it is a stretch.

Alright, I'm really tired. So if none of this makes sense, assume it's my fault.

Peer networks are deceivingly simple at first glance, and wickedly complicated in the details. It all makes sense, but that doesn't mean it cant be confusing at times. :-)

Re:Once again...

[_Knots]

How do you engineer a data transport protocol that prevents a DoS?

The only thing I can think of is making the transmitter do some nontrivial computation that can easily be done by the reciever - an overly expensive CRC or something - and have the packet be dropped if the result isn't right. Relaying nodes wouldn't have to compute this because it's already been done.

However, it's still possible to DoS a smaller subset of the network by sending out random packets that will in all liklihood be dropped but nevertheless will require CPU time to be dropped.

Did I miss something? Care to enlighten me?

--Knots;

Re:Once again...

[PureFiction]

How do you engineer a data transport protocol that prevents a DoS?

You can never eliminate it completely (anyone can still flood you off the network given enough bandwidth)

But what you can do is make it difficult. There are a few characteristics that should be implemented.

1. Make any kind of DoS attack require at least as much bandwidth on the DoS source as the bandwidth it interrupts. Meaning, the only usefull DoS is a flood, which no one can protect against anyway. In gnutella, a DoS can have a huge impact on wasted bandwidth by using very little sending bandwidth (the network effect of relaying bogus queries for example).

2. Make communication direct, so that all nodes know whom a given packet is coming from. In gnutella, packets are forwarded, and you never know if it is legitimate or not. Is this person malicious? Or are they relaying data from a peer downstream that is malicious? Having communication direct allowsd you to terminate connections to those you do not want to communicate with.

Relaying nodes wouldn't have to compute this because ...

Any node which relays traffic is a fundamental weakness. I do not like overlay relay networks at all for this reason. You never know (really) where a packet originated from, and this makes it very hard to protect against certain kinds of attacks (from well connected peers injecting bogus traffic)

I don't.

[SHEENmaster]

My download speed goes from 60kbytes/s to 10kbytes/s when gtk-gnutella gets going.

GTK-gnutella has no ads, annd that's without downloads.

By reducing the number of host connections, I can reduce the bandwidth drop to 5kbytes/s.

It's a good product, after being reconfigured.

Re:I don't.

[Uerige]

That's probably because gtk-gnutella, like many other gnutella clients, still does not implement the superpeer (or what was it called again?) system like LimeWire, BearShare or gnucleus. Probably you should try one of those.

Where the all bandwith goes? It is easy

[tandr]

Ready?

Slashdot.org and traffic redirected from its links.

I wonder

[A5un]

How much of current network traffic (data/voice) are really just protocol? I mean all the way down to physical layer (yes, the 1 and 0's). Seems like every layer of abstraction tags a protocol header on to the real payload. Is there any study done in this? I won't be surprised if more than 50% of network traffic are just protocols (IP headers, TCP signals, SONET header or even CRC bits).

Testing bandwidth does it too...

[kawika]

I run a site that has a bandwidth test, and there are people who run big multi-megabyte tests every hour or less to "see if there are any problems" in their connection. Multiply this times lots of people and lots of bandwidth test sites and I'm convinced that a lot of the bandwidth on the Internet is wasted in testing connection speed!

Gnutella?

[Openadvocate]

Just take a look at a "idle" gnutella node, before supernodes or whatever they call them, I could easy spot the times where I had run it, by the graphs in mrtg. Everytime I got a steady 50 kilobytes out every second.

That's "MUTE" point not "moot" point

[Anonymous Coward]

When will people ever get it right. It's a MUTE point. There is no such word as "moot."

Re:That's "MUTE" point not "moot" point

[JordoCrouse]

No - its moot. The point is not without speech, it is without relevance.

Re:That's "MUTE" point not "moot" point

[Rosonowski]

Actually, there is such a word as "moot" if you look into it.

Google tells us there is a venture capital company
And a game by that name; oddly on the "nuances of the english language. (don't even complain about mine, I won't claim to have perfect grammar, since english is an evolving language.)

I also have heard it used in reference to a gathering of were-creatures.

Not to mention, the last name "Moot" is a very real one indeed.

So yes, the word "Moot" does exist, even if people use it wrong.

Re:That's "MUTE" point not "moot" point

[Com2Kid]

Beh, troll though it may be;

Moot at Dictionary.com

Yes, M-W.com has it as well, but their site is nto so easy to link to.

Re:That's "MUTE" point not "moot" point

[Anonymous Coward]

But no "people" were using it wrong... The guy who said that it's "mute" is the one who is wrong. Which honestly is a very stupid error... Has he ever heard the expression spoken aloud? How did he mistake "mute" (myoot) with "moot" (moot)?

If this were legal, it could be far more efficient

[Animats]

If this "file-sharing" stuff were legal, it would be easy to do it efficiently. Each new song would go out on USENET into some binaries group, traverse each link no more than once, and reside in a nearby news server. A modest-sized disk drive per ISP could hold MP3 versions of the entire catalog of popular music. No problem.

So that's the standard with which P2P networks must be compared on an efficiency basis. It's not looking good right now. Current P2P architectures scale badly. This is well known in the networking community, but not widely realized by end users.

A big problem is that it's hard for a program to tell "how far away", in some sense, another IP address is. You can measure latency and bandwidth, but those are only hints. If many programs are doing this, the overall result is suboptimal. There's been considerable work on efficient self-organizing networks, mainly for military use. That's where to look for better architectures.

This sounds familiar...

[Anonymous Coward]

The crap about how 5% of the users use 95% of the bandwidth is really starting to piss me off..

Gee, where have I heard this before?

Oh yeah:
Re:Metered pricing vs. flat rate
by CmdrTaco on Friday March 01, @01:01PM (#3091248)

To put it in perspective, the 3% of readers who read Slashdot the most load 25 times the pages as normal users.

Farscape

[T-Kir]

No I didn't think of that at all, missed it completely... probably because I was sat there watching Sci-Fi Fridays premieres of SG1 and Stargate for 11 weeks, and watched the show in England.

Did ya stop to think that maybe if you watched the show on television instead of downloading it, it might still be on the air?

I wish that we're the case, except that even though SciFi is getting more viewers than usual for the new Farscape eps, and it being their second highest rated show (I think). Them deciding to ditch it might not be related to people just downloading the eps, but to their decision of concentrating on LowFi Low Cost (Non Space) shows to please their board/shareholders etc.

Deagol garcia

[Anonymous Coward]

Weird. You two talking to each other sounds like a conversation between Deagol garcia, which sounds like Diego Garcia where U.S. warplanes tank up.

So your conversation sounds like the island is having a serious talk with itself.

well...

[EnderWiggnz]

the issue is saving bandwidth - and for extremely high traffic sites, i bet you see this type of coding a lot.

and you're also right in saying that for the vast majority of sites, this is pointless.

to further expand on your assembly example -
for the vast majority of applications and processes, a compiler will optimize much better than a human could.

but, if you need an extremely tight loop that will be executed millions and millions of times, assembly is the way to go.

but thats an extremely small percentage of apps that need it.

just like an extremely small percentage of web sites need to have shortened names of objects.

as far as the html spec - the things a friggin hack to begin with :-)

Re:well...

[Hast]

the issue is saving bandwidth - and for extremely high traffic sites, i bet you see this type of coding a lot.

Yes most likely. And most coders also feel that they "optimize" the program when they write fewer characters. It doesn't have much impact (at least none that is good) however.

Humans shouldn't try to optimize stuff that a computer can do better. It's a waste of time, and a source for frustration. (When it breaks the code.)

just like an extremely small percentage of web sites need to have shortened names of objects.

I don't think anyone disagrees with you that compressing pages are good if you have a lot of traffic. But I sincerely doubt that anyone is doing it by hand. (Except stupid people with too much time on their hands.)

The "right" solution is to either preprocess the template HTML files or to add a GZIP module to the web server.

Gnutella IS inefficient

[PotatoError]

If you take a look at the spec for the Gnutella Protocol, you will understand where all this "extra" traffic is coming from.

I've been messing about with gnutella on and off for about 3 months now, I hope to make an open source functional client eventually. It's quite an interesting area because there is so much work to be done on security and efficiency.

The only problem is that P2P networks are never going to be as efficient as centralised server networks and certainly never as fast. I suppose a cynic (like me!) could blame the entertainment industry for forcing out server based file sharing networks.

But I believe the death of server based file sharing is a good thing. The bad side of the server-client model is that it can (and usually is) controlled by an authority and its security is often obscurity based (the obsure bit being hiden on the server). Peer to Peer networks however, offer total anonominity as well as giving users access to the whole component.

Peer to Peer networks are the next step in securing freedom of information on the internet and preventing government control.

It's when Peer to Peer mobile phone networks are produced that things will really get interesting....

Re:Gnutella IS inefficient

[base3]

Please see this reply (inadvertently posted under wrong comment).

Re:Optimize html - automatically

[YetAnotherDave]

I used to work for a company that made an online PIM/email system. We had a tool that went thru the code and stripped comments, linebreaks, and other such noise. Saved quite a bit of bandwidth, and allowed nicely formatted & commented code to be used internally. THis was a C thing, but I think it'd be a pretty straightforward bit of perl...

When you hit the monthly limit...

[Anonymous Coward]

...they just bill you for any additional data you transport. (Sympatico's rate is $0.80/100MB. It's estimated that they pay around $0.08/100MB.)

It works a lot like a cell phone plan with an included number of minutes.

Not all P2P networks are created equal

[0x0d0a]

Freenet is more efficient than, say, the Web would be. Those DiVXes don't need to cross your ISPs downstream connection at all.

Gnutella is noisy, but that's not the fault of the creators. Blame the RIAA -- the first P2P applications were centralized. If you can give up the requirement that there be no single, trusted point of failure, it's much easier to make an efficient network. They attacked Napster, and now people have moved to mostly less efficient approaches.

Re:Not all P2P networks are created equal

[Magila]

Haha, I'm sorry but Freenet and efficient do not belong in the same sentence. Freenet makes huge sacrifices in efficiency in the name of anonimity and privacy.

Wow

[Anonymous Coward]

The parent post was kind of ho-hum, but the following part of the argument caught my attention:

Timothy has obviously never talked to a P2P developer in his life.

Again, wow. One would need to search far and wide, even on Slashdot, to find another example of such absolutely astonishing cluelessness. The parent poster has obviously never figured out that comments in italics are part of the submission from a reader, and editors' comments (if any) are in normal face after the italicized text ;)

Firesale on Bandwidth

[LfifeWithoutMusic]

Large hosting providers are mired in exccess bandwidth and other resources...But, do people buy them this way?

_______
www.sw-soft.com

AOL is what makes us afraid

[sdjunky]

AOL is what makes us afraid since it shows that content providers don't care about refresh dates or content but rather cutting the bandwidth. Even if it means compressing images and caching a page for extended periods of time.

Efficient search in peer networks

[PureFiction]

I doubt that developers of those free p2p applications have gave much thought to efficiency

Some of us have. Search is much of the bandwidth in peer networks is wasted (downloads are downloads, but search can eat up a lot of bandwidth for little return)

There are some efficient, effective peer network search apps currently in development. Hopefully we can eventually leave gnutella and kazaa in the past and move on to more open, efficient networks...

Heart Beat Traffic

[RetroGeek]

When I was working as a network admin, we did a study that showed 50%(!) of the traffic was heartbeat and network system messages.

And we had 56K lines in use.

Not much we could do about it unfortunately...

Where the bandwidth goes, you can go too!!

[McFly69]

Just follow it!

PNG currently works only for still GIF images

[yerricde]

"Web browsers don't support" is not an excuse anymore

As you correctly point out, that's true for non-animated images. However, what do you propose for simple animations? What if animated .gif is the only way I can get advertisers to buy space on my site? Is there a way to hold off Unisys for the last nine months of the life of U.S. Patent 4,558,302? Mozilla (and Netscape 7) is the only popular browser to support Multiple-image Network Graphics, the animated extension to PNG and free alternative to animated GIF images. Excluding IE users is not an option. Or should I try to find (or write) a tool to convert animated .gif to .swf?

unless you have alpha (specifically multiple levels of transparency), which some of the older browsers didn't handle so well.

Even IE 6 doesn't handle alpha very well. (Mozilla does.) However, any PNG image converted from a still GIF image will work fine.

Re:PNG currently works only for still GIF images

[error0x100]

Actually, I wasn't proposing trying to *quit* GIFs altogether. If you have animated GIFs, just stick with the GIF. There really is currently no other reasonable, open standard for animated images that is supported by the major browsers. But I wasn't talking about patent issues at all..

Let me emphasize this: I wasn't advocating PNG because of the patent issues. I was advocating PNG out of entirely pragmatic concerns: a bandwidth saving on images of typically between 10 and 50 percent. This must surely make sense even to all the people out there who neither know nor care about the patent issues.

I was specifically only talking about non-animated GIFs, and primarily also GIFs with no transparency (due to some of the slightly older IE versions struggling with it). This is a totally rough thumbsuck guess, but I'm guessing that this is probably more than 30 % of the GIFs out there.

I do have an opinion on the patent issue, but that is entirely beside the point - I am simply trying to be practical, so I kept that issue completely out of my post. Most people are far more likely to be swayed by solid, practical arguments (such as bandwidth saving, faster downloads etc) than by moral or economic/financial concerns (such as the GIF patent).

Everyone seems to always be trying to convince webpage developers to switch over from GIF "due to patent issues", and seem to often forget that PNGs tend to actually also be smaller (or its added as an afterthought). Its no wonder people aren't interested. Most people don't give a crap about the patent issue.

So I say (to webpage developers), forget about the patent issues: use PNG because it just plain makes sense to do so. Smaller files = more space available on your host and less bandwidth used. Less bandwidth and space used by your host = lower prices on average for you (because your hosts b/w costs will be lower on average, and web-hosting is a competitive market) and faster downloads for your users. The tools to make PNGs are easily available, free, and quick and easy to use. Everybody wins.

Out of interest, related to this: SecuritySpace does web surveys that include what percentage of sites use which image formats etc: http://www.securityspace.com/s_survey/data/man.200 208/techpen.html. PNG is steadily growing, but very slowly.

The good part of the DMCA

[yerricde]

Unfortunately, if we cache and then illegal material is downloaded, we can be held responsible for that material.

Not necessarily. A rider on the DMCA allows service providers in the United States to cache web pages, provided that they meet certain criteria (which are easy with HTTP/1.1) and designate one of their employees as a DMCA agent. Read more on this page.

duh

[cr@ckwhore]

I might be missing the point, but why complain about cost? If the cost of a big private line is a problem, then you should consider VPN.

Put Your Reading Glasses On

[Anonymous Coward]

and re-read the quote:

I doubt that developers of those free p2p applications have gave much thought to efficiency.

The comment was not about all p2p applications, just the free ones. You said it yourself:

Gnutella, which everyone who knows anything knows is "worst of breed" when it comes to broadcasting searches

and the rest of your post seems to confirm the orginal assertion if one accepts the likely scenario that the 'most popular' p2p applications are the free ones.

The comment was intended to be provocative, but not an attack of all p2p developers. I'm sure that professionals, like you, are going to be concerned about efficiency.

Re:Put Your Reading Glasses On

[dmelomed]

And why are YOU replying as an AC?

Everything is better with pr0n!

[Anonymous Coward]

Unlikely. I suspect that slashdot is more that 3% trolls. Much more.

Re:Everything is better with pr0n!

[Anonymous Coward]

Those are just the first post trolls. goatsex trolls, such as myself, don't need to reload heavily, but the first posters will reload constantly.

Two things..

[Restil]

First off all, the P2P networks by design will generate far more traffic than necessary. Necessary, being of course a single set of central servers that collect data from the entire network and serve out that data ONCE to anyone requesting it. However, Napster as we all know died a painful death because there was a single point of failure. Kazaa, gnutella, and others have no head you can cut off. Even if the company that sponsors Kazaa were to be sued/prosecuted into oblivion, the network would remain. The downside, of course, is an excessive amount of unnecessary traffic.

The second big problem is the fact that as far as I can tell, none of the P2P networks take advantage of the teired nature of the internet, attempting to search local networks first, and searching further ONLY when something can't be found closer. Bandwidth is always more scarce (and therefore more expensive) the closer you reach for the backbone. Any effort to keep the traffic within the local network of the ISP costs THEM less, which means they would be far more willing to promote those types of networks, or at the very least not attempt to restrict them.

The network admins for universities were especially outspoken against Napster at the height of that craze, since that single program was consuming all the upstream bandwidth, where there is a DAMN good chance that with a student population in the tens of thousands, there's probably a 99% chance that anything a student was searching for could be found somewhere on the university network, which typically has much larger pipes than the internet upstream.

-Restil

Re:Two things..

[HamNRye]

MY bandwidth just went to turning the plasma ball off... Ho Hum.

~Ham

"always on" bigger problem when sharing

[Trepidity]

At my school at least, the biggest use of bandwidth seems to be people who leave filesharing programs on all the time, which ends up sharing their download directories by default, even if they haven't configured them to share additional things. Having even a few dozen people sharing DivX movies on a high-speed pipe uses up a large percentage of the school's bandwidth, far more than the network chatter does (we're talking on the order of 30-40 GB/day for a single host).

Bandwidth killer

[kaden]

I'd love to test this better, but I think a very big ammount of bandwidth is used by one specific group: tech-ignorant college students. I have some experience, I'm an RA in a dorm at IU Bloomington. People open Kazaa and leave it on 24/7, not even knowing (or caring) that they consume gigabytes of bandwidth daily (especially on male floors where most people have at least a few large porn movies, which are probably in more demand than Britney tunes).

We put up signs notices, strongly requesting people fully close P2P programs when not using them, but it's been my experience that most students can't grasp that it really matters.

Re:If this were legal, it could be far more effici

[base3]

If you thing peer-to-peer networks offer total anonymity, try sharing some pr0n that's illegal in your locale, along with some realistic-appearing stories about killing $HEAD_OF_STATE for a month or two, 24x7, and get back to us about that anonymity thing.

The current apps (other than Freenet/GNUnet) all either connect to or request a TCP connection from the machine sharing the material. When the client retrieving what you're sharing connects to or is connected from your machine, your IP address is known and that's one level of indirection from your identity (barring use of an open proxy).

Although many of the clients, particularly some of the Gnutella ones like Limewire attempt to obfuscate the addresses a little at times, the protocol is open, and $THREE_LETTER_AGENCY or $COPYRIGHT_CARTEL is free to write a client to reap the IP addresses of those sharing certain content (q.v. Ranger).

Re:If this were legal, it could be far more effici

[base3]

Damn it, I replied to the wrong post. Sorry!

Re:Blame the RIAA

[CreamsicleSeventeen]

The RIAA is responsible for the exponentially bandwidth hungry topology of P2P clients? We can no more "blame" the RIAA or Gnutella's creators for this than we can "praise" ancient greek mathematicians for making the inner angles of a triangle add up to 180 degrees. It's in their nature. Shawn Fanning wrote a paper addressing this very topic. You've made such a leap we might as well "blame" online music trading for spawning the RIAA.

The point is...

[Steveftoth]

that it's not the movies as much as the protocol.

I bet that running a ftp server that has the same content will result in less traffic even if the movie is downloaded more often. Why? Because of the crosstalk inherit in the p2p protocols.

I had this problem once....

[Anonymous Coward]

And it turned out to be MY fault. I did this back when there weren't any clients as good as the original gnutella, back when I lived in the dorms.

I had the connections set to allow 32. This put a LOT of strain on the school network. They wouldn't tell me how much I used, but I imagine it was close to that. It probably didn't help that I also had my NAT box accidentally set up to listen to DHCP requests on the outside, not the inside :) I don't know how DHCP reacts to multiple DHCPd's, but that is another mistake.

I haven't checked the gnutella clients recently, but if any P2P clients allow incoming connections to build without weeding any out, you are going to find yourself overrun, and potentially targetted by those in network admin. Yeah, people log off, but depending on conditions, you will get more connections than disconnections. I happened to start mine on a friday or saturday (it was a long time ago), so it was on a weekend. Most of that kind of traffic occurs on a weekend.

mod_gzip for Apache

[jsjacob]

A new HTTP command is not necessary because HTTP 1.1 supports compression as a content encoding (the "Content-Encoding" HTTP header). The mod_gzip module enables compression for Apache. As you suggested, mod_gzip can be configured to compress or not compress certain files matching given criteria.

Re:If this were legal, it could be far more effici

[Anonymous Coward]

Each new song would go out on USENET into some binaries group, traverse each link no more than once, and reside in a nearby news server.

You don't do this already?

I already download most of my... software through USENet. It takes a little bit of effort for larger files but I bet the search success rate is just as good as most P2P services.

Plus there are news sites that track binaries being posted to newsgroups.

You do have to pay a bit a money every month for a good newsserver but given how much you can steal^H^H^H^H^Hdownload, it's well worth the price.

I need Bandwith

[SuperNova2001]

PLZ

Gaming Hogs

[VEGETA_GT]

I find it interesting no one has mentioned gaming in this talk. Personlay I do a lot of gaming, keeps me from killing the marketing drownes where I work. But now I go onto game spy and see 10000 people playing medal of honor. the thouhgt came to me of exactaly how much band width is sucked down in online gaming.

I decided to try it out, ran in a game of medal of honor, and watched the throughput. Neadless to say I saw a constant of about 10 k/sec. I know this sounds like a good 56 k modem speed at best. But start doig math, and you get some big numbers (my math sucks so I an't doing it here).

Now do we stop playing games online, Hell know. Do you realize how many marketing drownes could die from this. Yes soon, my ISP rogers well be caping my line at I beleave 8 gig a month. So Say Myself, my brother game online a bit like normal. My father, broter and me decide to surf the net, and then ontop of that the new linux iso comes out (Generaly A few new ones per per month). Then start doing more math. while my head hurts from thinking about doing this math, I can tell you it blows by 8 GB.

I can understand the ISP's reason for wanting to cap, but in general, most customers only grab a few mp3's, surf, and game. And you would be extreamly suprised what this well sick down in terms of bandwidth. So what ISP's are forcing us to do is take a step back, not forward. It well hurt them in the long run because people well not want to pay extra cash because they decided to play a game of Quake 3.

One last note, Look, a DSL in my area has not caped and is not planing to, while Bell has and Rogers is about to. I know of many people who jumped to rogers when bell caped, so they well all jump to look when rogers caps. In my eyes, for some of the ISP's here in toronto to cap is like beating your head against a brick wall, It can realy hurt you.

My 2 cents plus 2 more

Depends upon your use of "efficient"

[0x0d0a]

Freenet has extremely high latency, yes. Request a file, and it might be a while before you get a response. Try browsing the Web-on-Freenet, and you'll get a less-than-optimal experience.

However, Freenet has efficient file transfers in terms of bandwidth usage, and avoids killing any single point on the network. Freenet is network friendly.

Re:Depends upon your use of "efficient"

[Anonymous Coward]

However, Freenet has efficient file transfers in terms of bandwidth usage, and avoids killing any single point on the network. Freenet is network friendly.

At least thats the idea, current implementations dont seem to work this way. It is only version 0.4 though - so I wouldnt expect it to work 'as planned'. The ideas are there, but..

YHBT. YHL. HAND. :-)

[Anonymous Coward]

YHBT. YHL. HAND. :-)

Hey, in Australia we pay 13.5c...

[Goonie]

per *megabyte* for excess bandwidth usage from most of the broadband providers.

Not many Aussies run gnutella AFAIK :)

We can *always* blame the RIAA :-)

[0x0d0a]

*shrug* I understand that this sounds loony, but I assure you that most P2P coders (well, at least *I* would :-) ) would love nothing better than the ability to use centralized, stable, trusted servers. It'd make their life *much* easier to make solid, easy-to-use (remember the get-a-host game played on Gnutella?), efficient apps.

Gnutella and friends only came about because Napster was under attack by the RIAA. The only way for file sharing to survive was to mutate and scatter, go not just P2P, but fully distributed, with *no* central points of failure.

The RIAA has unhesitatingly attacked any P2P services that have a central server. Given the environment, P2P evolved to be much more distributed.

And in doing so, became less efficient.

Things will probably pick up eventually -- P2P research is under full steam, and is a popular thesis subject now. As P2P and scalable, distributed and untrusted storage becomes a better understood problem, efficiency will improve. At the moment, however, file sharing has been pushed into a raw area of research. And yes, I do blame the RIAA for this.

Incidently, it may turn out to be a good thing in the long term -- distributed, failsafe, untrusted networks have a lot of potential for the future, and it's unlikely that they would have been popularized nearly as soon without the RIAA.

Conflict brings evolution -- World War II brought us atomic power and the programmable computer, and it looks like the RIAA is heading to bring us into the next era of worldwide telecommunications.

Re:We can *always* blame the RIAA :-)

[Dthoma]

"Incidently, it may turn out to be a good thing in the long term -- distributed, failsafe, untrusted networks have a lot of potential for the future, and it's unlikely that they would have been popularized nearly as soon without the RIAA."

You're using such a network right now, which has been popularised with virtually no help from the RIAA whatsoever.

Re:We can *always* blame the RIAA :-)

[lameland]

Do you really think it's that failsafe? What if the master DNS servers were destroyed? Yes, the connections to the rest of the world would still function, but do you know the IP address of every site you go to on a daily basis? How about if a backbone link died? Or an oceanic cable snapped? The Internet's health is still beholden to a relitivly small group of corporations/governments.

The research that he is talking about would eliminate these weaknesses, by making the Internet server-less (or more precisly server-ful); every client would share its information with all other clients, making information truly ubiquitous.

At the root, that is why the RIAA fears P2P clients: they make information (music, etc) freely available and available to everyone.

What they should be doing..

[gylz]

The ISPs should be providing "free local calls". ie. sending data to another user on the same ISP should be free/unlimited. In the long run P2P actually helps ISPs because it increases local bandwidth and can decrease external bandwidth. Sending a packet from one internal router to another is going to be a lot cheaper than sending it across the world, so they should be passing that onto their customers in the same way that most ISPs in Iceland do - they charge more for overseas bandwidth than domestic.

OT

[offpath3]

Dude, you were running a rogue DHCP? That'll get you banned from the network for the whole year at my school! Heck, the guy that lived in my dorm room before me did it and I had to talk with the network support for days just to get them to re-enable my jack after I moved in.

Slow ass downloads...

[LilGuy]

This must explain why I get about a maximum of 2 k/s whilst downloading from Grokster, Kazaa, Bearshare, Imesh etc etc... I actually miss my cable modem back home... I used to get a good 100k/s normally on these p2p apps.. now that I'm in college its not even worth trying to use any of these... thank god for friends with a bit of knowledge.. In the next couple weeks I'll be beta testing a new p2p sharing which my college won't know about and limit my bandwith.. this will run through IRC so it should be a change of pace..

anyways just thought I'd rant about how cable kicks my college's ethernet's ass... :)

Song reference

[doublem]

Where has all the band width gone?

Do, da do, da do.

Re:Song reference

[oojah]

Where has all the band width gone?
Where's it all gone to?
Don't browse.

Can't think of any more for the moment.

Inefficient? Yes!

[Anonymous Coward]

If you have to do HTTP Tunneling, you're converting your stream to some kind of ASCII that will make it through a proxy, chopping it into little pieces that all have headers, adding TCP/IP wrappers and sending it over the network.

You can use XML - just realize that a control message that could be three bytes in binary might become 300 bytes or even bigger (you might have to wrap that XML in HTTP). That's a hundred times larger.

Of course, all of this is to get around that pesky corporate firewall to download Pink. The efficiencies aren't only in bandwidth - we're spending a lot of time on one end creating the firewall, and a lot of time on the other end circumventing it. It's pretty ridiculous.

indeed

[twitter]

Quoth the article

And if people are using the advertising-supported versions of those programs, there is even more traffic generated as the ads are "pushed" at the user.

It's amazing but true that the same thing can be said of advertising-supported web pages. It seems that an amazing amount of bandwith is taken up by advertisments and images I did not request. Indeed, as much as 90% of all bandwith used by comercial sites is composed of such "network chatter" as X-10 suggesting I check out the girl next door who just lost 90 pounds on the ginsing diet. And it blinks. All that just to get about four kilobytes of text and four kilobytes of image that I actually want to see. The ratio over conventional media is reversed: on the internet, most content is crap, whereas in conventional media all the content is crap.

Bandwith must be conserved on the internet so more crap we don't want can be pushed on more of us.

Re:Blame the RIAA

[Anonymous Coward]

By applying this logic, I blame you for that tomato in my fridge going bad last week! you heartless bastard, how could you do such a thing. I will never forgive you.

10% savings from reducing keep-alives???

[twoslice]

WTF?? who would send a keep-alive signal every 10 packets?? Since they are being charged by the packet this is what it would work out to. Even RIP is more efficient than that!

Re:10% savings from reducing keep-alives???

[Anonymous Coward]

who would send a keep-alive signal every 10 packets

The network was not saturated 24x7, just on 24x7. Many hours (even days on some weekends) could pass with no traffic other than keep-alives. For strategic reasons, it had to be on 24x7 but I'm not going to defend that approach - I thought it was dumb at the time.

Drastic Steps I Had To Take

[Anonymous Coward]

I manage serveral small cable modem systems across the country. My latest installation was in Kansas City. Withing a week, my users started complaining about the system being slow. Looked at my reports and found 2 users with 10GB and 17GB UPLOADS.
That totally kills the service for all other people. Even after I throttled their upstream, a week later they still showed up in th GB range (compared to 1 - 5 MB for other users). This system runs on a T1 for 48 users. You tell me if it's right to let 2 people make the system miserable for the other 46.
I hope the RIAA sees the light and starts selling MP3's directly off their web site and we can get rid of the [unnecessary] bandwidth hogs. Preserve B/W for Linux ISO downloads.

Yes, p2p is a bandwidth hog

[Anonymous Coward]

That's why you get ISP's blocking p2p ports. And when they do, they just piss off their customers. That's why there needs to be more things like this. This may not be the solution but at least it's a step in the right direction.

A New Low

[whereiswaldo]

Obviously the person who submitted the story doesn't know what he is talking about.
Efficiency is a major focus of large P2P apps. When you are making hundreds of connections, you need to be efficient, or it won't work worth a damn. Coming up with an efficient enough algorithm is probably the hardest part!

As far as using up international bandwidth, the reason it is so expensive in the first place is because not enough of it is used. Telling people not to use more of it is saying that it should always remain expensive.

The story submitter is totally off-base.

60% wasted howz that?

[inquisitive]

My office folks were eating up 60% of the BW with Kazaa, I had no option but block at the FW, Now things are back to normal...

P2P ports? Anyone?

[Anonymous Coward]

OK. We all agreed that P2P makes lot noise, can someone list what we can block on our routers.

Complete list of ports for all P2P sw so we can block/cap them.

It's up to the ISP

[karji]

ISP's could help those applications become more efficient by hosting the application servers on their own servers.

What simpler method than that, to make your ISP more efficient?

Let the ISP run a supernode....

[anonymous cupboard]

It seems to me that perhaps the best answer would be for the ISP to run a super-node. It sounds strange, but the decrease in off-network bandwidth could more than compensate for the cost of running the node. Use of port filters can ensure that the only supernode seen by people on that network would be that of the ISP.

I guess the ISP would then be potentially in trouble from the RIAA even if they didn't store anything themselves.

Use QoS?

[Cato]

Putting all P2P traffic into a 'low priority' queue on all routers, and HTTP traffic and everything else into a 'normal priority' queue, would help this. Actually some sort of bandwidth allocation (WFQ, CBQ, etc) could be used rather than priority queuing. P2P apps would get the whole pipe if no higher priority traffic is around, but just X% if there is other traffic.

Of course, this is wildly impractical given the complete lack of uptake of QoS in the Internet - but since bandwidth hogs such as Pointcast and P2P drove earlier adoption of single-point QoS boxes such as Packeteer, it is not beyond the bounds of possibility. ISPs could deploy this without cooperation from other ISPs, just as a way of giving better service to non-P2P traffic within their network.

Of course, some would say that P2P should not be segregated - in which case, perhaps they could buy a premium service that puts P2P into 'normal priority'...

Re:If this were legal, it could be far more effici

[AnnaBlack]

Binary encoding of files usually increases their size by around 30% (typical figure for uuencoding). Formats like mp3 don't compress anymore worth a damn, so you're stuck with that increase. So you gain some efficiency, but you also lose.

Anna B

Apologies - slow day at work

[oojah]

Pinging our hosts like people in the Unis do
All suprised, if we hear some pongs
And I'm sitting down here just watching you, and I'm thinking

Where has the band width gone?
Where's it all gone to
Don't browse
My modem's hurting
Don't browse

Re:

[Anonymous Coward]

The internet, its not just for porn anymore............

Hey, they did it right the first time

[Magius_AR]

Central server systems cut down on alot of the needless broadcast-type communication P2P needs to find stuff.

As legal systems have attacked and all-but-destroyed such systems, we take what we can get.

Magius_AR

Hang on....

[PotatoError]

I realise that Peer to Peer networks don't offer total anonominity yet but they can do and therefore will do one day.

Using a third party as a proxy and hiding the data from that third party using public key encryption, you can prevent the recipient from knowing the sender's IP address and vice-versa.

The only weakness in this system would be if $COPYRIGHT_CARTEL owned the recipient machine AND the third party machine. The chances of this are extremely low - and if we were really bothered about it we could add more proxies to lower the probability further.

It wouldn't slow the transfer down too much - the client would choose a third party which was willing to proxy the data and had optimal bandwidth to do it.

Maybe such a client could have an option in the preference menu for how many proxies you wanted to use giving the user an option between anonomity and speed.

Re:Hang on....

[base3]

You just described the ZKS Freedom network, which, curiously enough, was taken down for "business reasons" right after 9/11.

IIRC, they did open source the router code.

Re:Where is Jon Katz?

[Anonymous Coward]

Hopefully you'll notice this reply!

You must return to IRC some day!

-Your best boyfriend.

Run out of bandwith

[rossy]

I think you are right. The Internet is already
out of bandwidth. All I seem to see is all this
noise.