Gnutella's Challenge

Gnutella News sent in an excerpt from a clip2 DSS report about gnutella's evolution and condition. "the network has neither smoothly scaled nor catastrophically collapsed since average traffic grew to regularly exceed dial-up modem bandwidth in August 2000. Instead, the network persists in a fragmented state comprised of numerous continuously evolving responsive segments, the largest of which typically contains hundreds of hosts. We estimate at present that unique Gnutella users per day number no less than 10,000 and may range as high as 30,000. We suggest that further technical innovation and wide adoption of this innovation are necessary for the Gnutella network to scale beyond its present state."" Read this if you're interested in p2p ^[?] .

Comments

Swarm delivery is the way around bandwidth limits

[Jim McCoy]

One of the best features of Mojo Nation is that it breaks files up into smaller pieces so that when you want to download a huge file you are not blocked on the limited upstream capacity of the peer at the other end; each agent sends a small chunk of the file, allowing the peer retrieving a file to request multiple pieces in parallel and moving the download speed restriction back to the downstream capacity of the local connection. This sort of distribution system turns a pool of peers into a swarm of ants carrying small pieces of the content. RAID-like error correction protects against peers disappearing and allow for flexible choices about where to go for the pieces to the file. The new 0.920 release of the client starts to demonstrate the advantages this has over conventional peer delivery systems.

One downside to swarm delivery systems is that data is "published", simple sharing of a common filebase (a la Napster and Gnutella) is not possible. Someone has to upload the pieces to the system in the first place for them to be available because the system does not do the "let me take a look through your hard disk for things to give to others" kind of file sharing found in other P2P systems. jim

True P2P applications have this limitation

[jht]

Gnutella, being a real P2P applicaation, will suffer from scalability problems that a server-based system like Napster can work around. If Napster gets too popular, they can always add fatter pipes and bigger servers. But Gnutella is bandwidth-constrained since there is no central server farm tracking all the users.

The exchanges in Napster themselves may in fact be peer-to-peer, but we need to remember that they have big honking servers arbitrating the connections.

Gnutella's design is terrific (and a great hack), but unless they can re-jigger things to knock the slow connections down in priority (or some comparable solution), they're doomed to be a victim of their own success. I guess the other possibility would be for a minimum bandwidth requirement for the software to enforce. Perhaps some enterprising person will write a Gnutella that only allows, say, 144 Kbps and up connections on the network.

It would be interesting, though cruel, to relegate all the dialup people to second-class citizen status, but it would allow Gnutella to scale a lot past the existing limits.

- -Josh Turiel

More evidence of P2P's weakness

[Salamander]

This is a bandwagon that just won't roll very far, and the reason - as usual - is obvious to people who've studied the field for a while. Naively implemented, a P2P protocol tends to generate O(n^2) messages for a given workload, where N is the number of nodes. This can often be brought down to O(n) but only with absolutely top-notch developers and a lot of effort. Better than O(n) is usually impossible.

By contrast, hierarchical systems tend to hover between O(n) and O(log(n)) depending on the particular problem. This does not necessarily apply only to single-rooted hierarchies, either. A multi-rooted hierarchy tends to exhibit the same scaling behavior, though of course the more roots you have the more you start to look like P2P and share its scaling characteristics.

The long and the short of it is that P2P just doesn't scale well. Even the best-implemented P2P protocol can merely approach the message efficiency of a naively implemented hierarchical protocol. For large numbers of nodes this results in the P2P implementation simply getting swamped. The only question is how large and how swamped it has to be before it becomes unusable.

just have an option that says your bandwith type

[CiXeL]

all you do is have a option under preferences ala napster that says your bandwidth type ie. 56k/cable/dsl/t1/t3/etc then whenever you connect to another T1 have it make a strong connection between the two of you. Its network matchmaking. To prevent people from specifying a lower bandwidth than you have, have the program limit your download bandwith to that specific speed which should keep people honest while helping to better organize the network.

Slur on Christmas Island

[StrawberryFrog]

> sleazy (gnet2.ath.cx has the exact same TLD as another website whose URL contains the word "goat";

cx is a country TLD. Why should you call the whole of Christmas Island sleasy because of one goat who lives there?

Re:Idea: (scalable and distributed)

[Axemaster]

Actually, MojoNation does something very similar to what you propose.. its still a beta product, and it's still growing, but it looks good so far:

* Automatic mirroring nodes
Mojonation block-servers remember what blocks seem to be popular (most requested), and if they dont have them, they may go grab a copy to mirror locally.

Nodes would automatically mirror data from local (fast) mirrors, so that it's more accessible.

See above. Data that is popular is automatically mirrored. When data is published to the network, dual-redundancy is used to avoid losing the data if some blocks turn up missing. Think RAID. Well, no, not exactly, but it is somewhat redundant.

... 56k clients could connect and ask the "net" of super nodes for the queries on content..

It's called a content tracker, and anyone can run one on Mojonation. There are two central "master publication trackers" (MPT's) that keep lists of all publication servers, and the clients retrieve this list initially from them. There are possible plans to distribute the MPT's as well.

Content Security

All of the content posted to the network would have meta-moderation on it; anyone can classify data, and mark it as such.

There is currently no 'rating system' in mojonation, but it is something being looked at, barring the technical hurdles in doing so.

Privacy

If possible, I'd like to see users IP addresses hidden; only have a unique login name/password setup for security; but this may make hackers/spammers hard to track and ban, but hopefully the meta-moderation would filter out most of it.

I'm not sure if Mojonation is going to go this route eventually, but if ya use TCP/ip, you can be traced eventually anyway. UDP is unreliable.. As for data privacy, Mojonation actually chops a file up into small blocks, then encrypts those blocks, and distributes them randomly. Then it send the description and block locations to the master server. In essence, nobody knows whats in each individual block on their server (if they run a storage server); everything is encrypted. I am breezing past all the details here, feel free to read more about it if you wish.

Volunteers
Anybody?

http://sourceforge.net/projects/mojo nat ion/

Reputation Tracking

[pjrc]

I've been toying with an idea for P2P filesharing, which involves a truely decentralized reputation tracking system. The idea is similar to PGP's "web of trust", which you "know" others based on their public key, and people you know give certifications of others by signing their public keys.

What good is all that... well, a host could make decisions about which queries to route and which to discard based on any information about the reputation of the originator. Hosts would allow faster sends to downloaders with good reputation. Abusive hosts (Spammers, DoS attacks, etc) would ruin their reputation quickly (or keep recreating new keys all with no reputation).

Reputation in such a system would be very valuable. Somewhat like slashdot karma, it would appeal to many individuals, who would likely go out of their way to gain reputation signatures, perhaps by providing or mirroring lots of high quality files, attaching good meta-data descriptions to files, etc. The client software would need to have ways for everyone to do moderation on files and users... but unlike slashdot, there would be no universal score, only lots of keys/reputation scores, signed by other users. The software could also automatically detect certain behaviors (files available for download, on-line for long times) of other hosts, and issue reputation points. The idea is that a reputation score is to have a way to allocate the available resources (mainly bandwidth), to establish an incentive for users to share files and act in ways that benefit the network, and of course to make the network resiliant to abuse.

Now, for a system like this to scale, each host will need a LOT of disk space, to store a giant database of keys and signatures on them, and it would ultimately act like a giant cache. Each host would obviously collect the most positive signatures... the initial communication would be similar to boasting, the requester would send several of the best moderation signatures, hoping that the remote host already knows those people who signed and will therefore offer faster transfers, propagate a query farther, etc.

Maybe this ultimately works out to be the same as digital cash in MojoNation. I believe it is a different idea, in that it's based on an assumption of abundance.... everybody can win. You can get a great reputation without someone else giving up anything. In a cash system, when you get cash (mojo), someone else gives it up, and the overall philosophy is of scarcity.

If you have any ideas or thoughts to add to this, please post. Am I totally out in left field here, or does this seem like a reasonable idea?

oh, *please*

[vsync64]

I'm sick of this. "Gnutella's going to collapse! We need new innovation! It doesn't scale!"

It doesn't have to cover the entire Internet. The fact that you can simply specify a server to contact makes the solution so obvious that I can't believe people are still whining.

Let Gnutella split into multiple networks. It worked for IRC, it will work here, and it will work for similar problems in the future. Any problem that doesn't lend itself well to subdivision is probably badly specified. Don't forget that the Internet is a network of networks, and it works well for a reason.

Idea:

[kwj8fty1]

I've been reading tidbits around the net, and I'd like to ask what people think about this:

Automatic mirroring nodes

Nodes would automatically mirror data from local (fast) mirrors, so that it's more accessible. It would need to "learn" what files are requested, and then mirror them. What would stop the script kiddies from "rating" the content they want up, so it would be mirrored more often?

Structure

If all of the clients are required to keep a copy of the "whole database", it is not feasible without everyone on the network having a T3+, or later OC3+ connection. But as with the data, the nodes keep track of other nodes, but only if the bandwidth permits. 56k clients could connect and ask the "net" of super nodes for the queries on content. No one node should be in control; but many based on the same rule set. You would have to have a setting on the client for "perm super-node", or just "56k browser". Even the 56k browser could contribute to the network however; two 56k modems that are on the same segment of 'net can transmit with very low latency; they can buffer queries from the super nodes, and allow for faster access.

Content Security

All of the content posted to the network would have meta-moderation on it; anyone can classify data, and mark it as such. People can also rate classifcations; so to prevent some spam. If a file with the same name shows up on the 'net, it could end up with the same rating. (my_garage_band_called_nirvana_that_nobody_has_hea rd_of.mp3)

I'm sure that folks have a complex yet effect methods of rating. (flame wars may ensue) but I'd be really interested in hearing ideas.


Privacy

If possible, I'd like to see users IP addresses hidden; only have a unique login name/password setup for security; but this may make hackers/spammers hard to track and ban, but hopefully the meta-moderation would filter out most of it.

Volunteers

Anybody?

-Eric Johanson - ericj.spambad@cubesearch.com

This sig for rent

needs: Better GUI and User Friendliness

[nirvana_am_i]

One of the reasons that Gnutella flounders is the Interface. The creators had a novell idea of creating a node-based network for file sharing, but their interface needs some work.

Napster, and then Scour, both simplified their application so any nitwit (even some mac users using macster) could gain access to the resources.

A better interface as well as some way to have the top hosts from gnutellahosts.com automatically be used everytime the application is loaded up is definately a must.

What the developers need to do is try a rapid prototyping model . As much as I hated it while doing that damn internship, it really does work. People need to be surveyed on how the application should work. The only way to come up with a good product which cathes the broadest audience is to get feedback from that audience.

Thats enough bs from me.