Gnutella2 Specs - Part 1

Mr Fodder writes "The first part of the Gnutella2 specs are finally up." Our previous Gnutella2 story has a little more information.

Comments

QoS based on material type?

[Anonymous Coward]

I think that packets related to pr0n should get a higher priority on the network. Has that been addressed in this version?

I may be wrong

[Henry V .009]

Correct me if I'm wrong, but isn't Shareza just making their own protocol and stealing Gnutella's name for it?

Re:I may be wrong

[fault0]

great.. let me create a fork of another open protocol such as freenet, openft, or even gnutella, and call it gnu3. i can have the first gnutellav3 client done by tommarow. it'll be on slashdot in a week.

there is quite a bit of danger to your reasoning. an official gnutella2 standard should be adopted, instead of one client calling it's new protocol the next gnutella protocol.

Re:I may be wrong

[Anenga]

Well, here's the story.

Micheal Stokes (Shareaza developer>) thought that the GDF (Gnutella Developers Forum) was a too slow at fixing Gnutella's problems (unscalable, too much unused bandwidth, unorgnaized for future additions) so he went ahead himself (by himself) and wrote Gnutella2. He has done this before, when he wrote the spec for "Remote Queueing" (kind of like IRC). He wrote his spec first, developed it in his client, released it then proposed the idea to the GDF. The GDF likes it and now Limewire, Bearshare and Gnucleus all support it.

The GDF is pissed that Mike went ahead and "updated Gnutella" without asking them first. Granted, they have a right to. The GDF is meant to be a consensus, a forum for all developers. And a "assumed" condition of that is to let the other developers know *ahead of time* before going ahead and doing something this massive. And the entire idea that he called it "Gnutella2" (using the Gnutella brand) and advertised it as the "next revolution in P2P" (which it actually, IMO, is) pisses them off even more.

However, if you notice, it seems only the developers with corperate ties are pissed. Other clients such as GTK (Linux), Gnucleus, etc. all seem interested in the protocol, I believe GTK already said they'd implament it. Limewire and BearShare still seem upset. (It's like owning a oil company, then someone comes out with electricty - sucks).

Anyways, Mike likes the Gnutella ideals - that it is open and free. So he called it "Gnutella2". Partly to "refresh" Gnutella and revive Gnutella's bad image it has with the general user (which it has achieved IMO) and to show users it's the "second generation" of Gnutella.

The Protocol is being released now. This is part one, the next one will go over the new packet encapsulation and what not.

Re:I may be wrong

[darkov]

there is quite a bit of danger to your reasoning

I dunno - if it maintains backward compatibility and doesn't break the network then what's the harm? If it's a better protocol and people migrate to it then we have progress. Design by committee is all well and good except that it's really slow and can get political, especially when commercial interests get involved. Sometimes the best decisions are not made. Survival of the fittest is the best approach

Re:I may be wrong

[0x0d0a]

Huh?

All of them are different, but let's take a look:

FastTrack -- the protocol is barely an improvement over the original gnutella, and with some additions from the LimeWire people, there are no improvements. It's also closed.

DC -- totally different, and from a technical perspective, much less impressive. Little more than IRC+DCC with a non-idiotic interface.

Re:I may be wrong

[pod]

To followup further...

FastTrack -- the protocol is barely an improvement over the original gnutella, and with some additions from the LimeWire people, there are no improvements. It's also closed.

The FastTrack protocol is vastly superior to Gnutella, especially the original Gnutella. It is, quite simply, one of the best, if not THE best, P2P protocol out there.

DC -- totally different, and from a technical perspective, much less impressive. Little more than IRC+DCC with a non-idiotic interface.

DirectConnect interface non-idiotic?!? DC has the stupidest interface of any P2P app I've ever seen. People keep bitching about how hard eDonkey or WinMX are to use (for example), but if anything, those are WAY simpler than DC. Just point and click your way to downloads. I have yet to download a single file via DC. The thing flies in the face of everything users expect of P2P apps, and even Windows apps in general. It's a disgrace.

Just my opinion though, move on, nothing to see here.

Re:I may be wrong

[Tomble]

I dunno - If it maintains backward compatibility and doesn't break the network then what's the harm?

Huh? As far as I'm concerned, the network is broken. The initial design came about from some guy (one of the Winamp creators, IIRC, but don't quote me on that) knocking something together as a proof of concept as a response to his new employer's (AOL/ Time-Warner) attitudes to file sharing, etc.

The original program was not as many people mistakenly assumed (due to the name) free software, it was closed source windows software, that other people had to reverse engineer, the design was fairly shoddy (because, as aforementioned, it was more or less a proof of concept), leaving far too much bandwidth spent on catering to people who use firewalls (and much of the time, those push requests simply get lost). Then, the countless different clone implementations tend to not-quite fit the same specifications (as each other, let alone the original), causing no end of problems.

I did use 2 or 3 different versions of Gnutella, for quite some time, probably over a year. I got lots of stuff. Of course, most of that is only half there, because of not always finding the same files again, or not being able to connect to a servent (most of the time), or stupid screw-ups where you resume a download, and find that the remote server is actually sending you something else, so you have to stop it and muck about with dd or similar to carefully cut the file back to what it was originally. Then, after a while, I noticed not only people who were putting up lots of copies of the exact same file (eg an advert for some site) under numerous different (and totally misleading) filenames (and also files that appeared to be proper files, even having pretty large file sizes, which turned out to be just windoze URL files, padded with vast amounts of space), but even, hacked up servents, that would return numerous different responses to any query you could come up with. EG, search for FOO BAR and they would return exactly FOO_BAR.mpg, FOO_BAR.htm, FOO_BAR.jpg, FOO_BAR.mp3, FOO_BAR.exe, and eventually, more cunning variations. I'm sure there were other similar things done by crackers and spammers, et al, but I can't remember them all.

BUT, ultimately, the thing that makes the Gnutella network broken most of all, to my mind, is the sheer LEGIONS of such utter CRETINS who have not the slightest idea what they are doing, flooding the network with queries that are almost GUARANTEED to return absolutely f**k all, because they simply do not get how the queries are matched. If they did, they would probably still not be able to get it right.

A few months ago, I pretty much stopped using Gnutella, as the network seemed to be getting progressively worse, and I seemed to be able to get less and less (and yes, I did used to share some files- ones that people seemed to want, too). I tried looking into various other P2P type networks, like GiFT and Freenet, and GNUnet, but felt badly let down by what I found. After a while, I tried having another look at Gnutella, and it was so screwed up it made me feel sick. The flood of bad queries (now including torrents of empty queries, about 80%, I'd say) was even worse, and trying to search for anything yielded almost exclusively the spam responses. I kind of got the feeling that maybe groups like the RIAA/MPAA could have been deliberately creating the noise and spam themselves, to try to make the network worthless. After about ten or fifteen minutes of searching, I gave up in dispair. As far as I'm concerned, Gnutella is dead.

Well, if someone proposes a new version, and it addresses most of these problems, IMO it would really be best if it broke compatibility. Nice clean break. Well, there's my 42 pence worth, flame away, all.

I am curious..

[mumblestheclown]

I am curious to hear stories of anybody who has at any point used gnutella to do anything but transmit copyrighted material in any substantial way.

By "monitoring" requests in limewire or by putting in ambiguous search terms, I estimate that well, well, well over 99% of files available through gnutella-based p2p services are copyrighted.

Oh yes, we all have heard the usual arguments. Technology doesn't break the law, people do. Aka, the Pontius Pilate / Eichmann defense.

Re:I am curious..

[Per Wigren]

I've used it to download Grim Fandango again, because CD2 of my original Grim Fandango broke when I accidentially dropped it on the floor and rolled my chair over it. :)

Re:I am curious..

[ceejayoz]

They are, however, free to distribute as long as the GPL license is included. It's perfectly legal and moral to download Linux distros over a P2P program.

Re:I am curious..

[Foresto]

"I am curious to hear stories of anybody who has at any point used gnutella to do anything but transmit copyrighted material in any substantial way."

I can't help noticing a similarity between copyrighted material on p2p, and porn on home video. Just as porn drove home video technology into becoming an industry and commodity, copyrighted stuff seems to be driving file sharing network technology toward becoming a viable distribution method. Right now, p2p seems to be approaching an adolescent stage of development, as it begins to address scalability issues and alternative applications like efficient radio broadcasts. This technology is becoming more useful, and as it does, I expect it will used to solve more problems than just swapping MP3s.

In other words, don't assume just because you see copyright infringement now, that the tech won't be something we all rely on for legal activity in the future.

Re:I am curious..

[akb]

I know I spent a lot of time looking for mirrors of RH8, all of the published ones were overloaded. You could actually look at the mirroring network for free software distros as an inefficient P2P network.

P2P networks like Edonkey and Freenet have the property that it becomes easier to download a piece of information and its more likely to be closer to you the more it is downloaded, rather than the reverse with a centralized server.

Paying for bandwidth to host large digital content is not always feasible for some information distributors. A group that I work with that produces freely redistrutable media is considering how to make full resolution video available. Sometimes even for the low res video we now make available we have peaks over 40mbps when a piece of info is popular. If we can't find a donor for a substantial amount of bandwidth then we'll probably use a P2P network.

It would be more efficient bandwdith-wise if ISPs implemented P2P nodes for their customers, rather than the customers doing it themselves. They recognized that this was the case a long time ago with newsgroups and more recently with Akamai. Maybe when there's more freely redistributable content available they will do so.

Digital signatures take care of the security concerns you raised. You can download them from authoritative website and check the file after you've downloaded it. Freenet and Edonkey use digital signatures natively.

Re:I am curious..

[LostCluster]

Image if the same anti-Linux organization posted trojan-containing distributions and started sending them out over ftp or http... all it takes is a few people too lazy to check their hashes and it will become impossible to audit back who released all the exploits into the wild.
Nope. It'd come out that the University of Middle-of-Nowhere's FTP mirror got hacked, so anybody who downloaded from them should check to see what they got. If you decide to download your Linux from an FTP server that's known to be owned by Microsoft, you need professional help. In being able to trace back who you downloaded from in real-identity rather than username form, it gives you a much better trail for reporting untrustworthy servers. Especially if you work in the ad business and instead of having an ftp server you have a p2p client where people can transfer clips etc etc using the existing network. Just, there are so many uses for P2P besides violating copyright.
Nah, FTP is still better for that use. An FTP server isn't that hard to set up, there are plenty of open source packages to pick from. Almost every web browser is capable of downloading over FTP, so your client likely already has the software they need, rather than asking them to download a special (offen spyware-laced... boy is that unprofessional) client.

Moreover, wouldn't this kind of communication be something only your client should see, and not something left out for other people to grab. Sure, you could sit and cancel every other user who tries to grab that file while you wait for the client to take the file, but that means you have to be there the whole time the file is up. As compared to FTP, where you can set it up so only the somebody who knows the right username/pw can get at the file, none of the P2P programs let you do that.

I'm not disputing that there are uses for P2P that don't violate copyright... I'm saying that P2P sucks compared to the mainstream protocols such as FTP, HTTP, SMTP/POP3 e-mail, etc.

Shareaza's gnutella?

[Superfarstucker]

Shareaza is the only gnutella client that even uses this '2.0' protocol so im more inclined to say that its not really a new gnutella protocol but more or less an extension of the current gnutella client (apparently they didnt like the rate @ which gnutella was progressing)..

Awesome

[baywulf]

Yet again I will be able to get "backup" copies of all the music and videos I soon plan to buy!

Re:blah

[GeckoFood]

...gnutella is a failure...

[snip]

Actually, what I personally find more frustrating is that when you actually do find what you want, the download fails because either the host drops offline or refuses to accept the connection. Another little irritant is the large number of files out there that are deliberately misnamed so that when you download and open them, you find yourself dropped into someone's personal porn site, regardless of what you're looking for. I used to look for cool stuff like the blooper videos and whatnot, but I got one to many that was deceptively named. Not worth the effort, really...I uninstalled the damn thing and quit trying.

Re:Official?

[fault0]

> Is this official, by the original Gnutella developers? (who were winamp related, right?)

Yes, the original Gnutella developers worked for Nullsoft, then a division of netscape, a division of AOL, now a division of AOL/tw.

No, it's not official in that sense. It's not even official in the sense that other gnutella-client (such as limewire, bearshare, gtk-gnutella, qtella, gnucleus, etc..) developers have adopted, or agreed anything of shareaza's new protocol.

Of course, I hope this new protocol works good, but it's wrong to attach the gnutella name to it. It doesn't have much to do with it at all. Next thing that'll happen is someone else will come up with a totally different protocol and call it "gnutellav3" or something. Bad precident.

Feedback rating?

[jonr]

Is there a feedback rating in the Gnutella protocol? Wouldn't it be nice to mark a file+sharer -1, Fake. The more "-" the file gets, the lower it is in the result list. And the sharer should be punished, although I don't know how.
J.

Yep, here's an Example

[Anenga]

OpenOffice recently asked the P2P community to help out in distributing it's massive install file over their networks.

It's now on Gnutella2.

magnet:?xt=urn:bitprint:S5Q756FJ7326XXDGA7KZBF25 PC RWCT7Y.XKR2LGSL2K3DR4CTE5H5PDZGQCZOKN2NJWOOQHQ&dn= OOo_1.0.1_Win32Intel_install.zip

I get 15 sources in seconds. (G2 required - good luck on G1)

Not Gnutella2! Shareaza1!!!

[smd4985]

As a developer of open-source software for the OPEN Gnutella protocol, I find publishing of THE Gnutella2 spec by ONE development team laughable. The spec may be revealing about how the latest Shareaza client works, but to say it describes Gnutella2 is NOT true. The Shareaza guys simply use Gnutella to boostrap their new 'Shareaza1' network, they haven't redefined the Gnutella protocol.

The truth is that Gnutella2 is on the way, but not from Shareaza. Gnutella2 is a loose connection of various enhancements to the Gnutella network that have been implemented over the last year or so by SEVERAL COOPERATING Gnutella vendors. The latest enhancement is GUESS, which was introduced before Shareaza's new searching methodology and seems to be Shareaza's inspiration.

The Shareaza people continue to attempt to preempt Gnutella as THEIR protocol, when in fact they are pretty much branching off from the network. Shareaza should feel free to leave the OPEN Gnutella network, but please don't try to steal a name that belongs to the users and developers of Gnutella.

You used the wrong phrasing...

[Cerlyn]

In the United States (at least), everything made since 1923 was, has been, and still are copyrighted, even if they were never registered with the copyright office. So everything you see on a peer-to-peer network is indeed copyrighted.

A more approriate question (as some of the responders have answered) is if anyone has used a peer to peer network for a legitamite purpose. The problem here is that the issues are quite grey. If I have Game X, or Game System version 1.1 can I download copies of the games/BIOS/etc. online for use with emulators/replacements for broken discs/etc.? If an online broadcaster, paying royalty fees, uses ABAcast or Peercast to distribute their works, do I in turn have to pay royalty fees since I am rebroadcasting them?

Unfortunately, there is a major gap between what people think they can do under copyright law and what they actually can do. While I have not extensively researched the above (IANAL), technically, all the above commonly considered legitamite things are *illegal* unless you have worked some deal out to repay the copyright holders.

The problem you really should be asking is if anyone uses P2P networks to delibrately distribute their copyrighted works, either as a primary or secondary channel. A few minor bands likely do. The next question is if you'll ever find them on Slashdot. And I do not know the answer for that.

Note I personally have *never* used Napster, Gnutella, Kazza, or any of the other networks, mainly because being caught doing so may jepordize my ability to be hired in certain areas. I used to be one of those nasty college network administrators trying to keep your P2P usage down because it overloading our bandwidth, and we could not order a significantly bigger pipe because our local phone switch could not handle it. Feel free to flame me for my ignorance as you will.

Net Impact -- Doesn't go far enough

[Anonymous Coward]

There are several steps in the right direction, but the new design still doesn't do enough to address impact on the network. I work at a public sector ISP and we try like heck not to "censor" traffic based on port numbers, but lately we have had to kill off several PtP applications because they were hosing links and firewalls. Since much of the next-to-last-mile is using flow-based methods to make the Internet fair, not to mention asymetrical NAT, any PtP structure needs to put a higher priority on limiting the number of "flows" or "conversations" (hostA:portA hostB:portB).
This *does* include UDP as many routers/firewalls/packet shapers do perform flow-based rules on UDP conversations as well.
We've seen a relatively small link full of Bubster traffic bring a medium-end firewall to it's knees
by causing far too many conversation setup/teardowns. GnuTella should try to construct a network of long-lived inter-hub connections such that a query is never sent over a *new* connection more than a a few hundred times. Fortunately the new design is at least progress.

the real gnutella

[asv108]

As mentioned in previous posts, the specification posted has nothing to do with Gnutella, Sharazea is just stealing a widely recognized name, this specification has nothing to do with Gnutella. If your interested in real gnutella development go to the Gnutella Developers Forum. There are quite a few open source clients available, the most popular being Limewire and Gnucleus.

I've been playing around with the limewire source for ahwile, it is well documented and there is no spyware in the open source version. I love how people complain about Limewire and spyware, when it is open source. Anyone can take the gpled limewire source and package it without spyware without having to reverse engineer it like closed source KaZaa.

Re:Not Gnutella2! Shareaza1!!!

[Anenga]

The client he works on is "Limewire".

I find publishing of THE Gnutella2 spec by ONE development team laughable

Then lets hear you? I haven't seen any replies in the GDF from Limewire on the spec yet. You find that specs being released by Shareaza are laughable, but what about when Limewire proposes their GUESS proposal? Or "CHORD" proposal? Didn't "one development team" work on those? Sure, you released it before actually implamenting it, but still... the rest of the GDF just questioned about it, you were really the only development team. People don't say "Gnutella's GUESS proposal" they say "Limewire's GUESS proposal".

The latest enhancement is GUESS, which was introduced before Shareaza's new searching methodology and seems to be Shareaza's inspiration.

Can Limewire stop saying that? Which is totally and utterly untrue?!! Mike was working on G2 long before you sent him your spec on GUESS. He told you in a private e-mail that he was working on it before hand, and that he would probably release his with GUESS.

And the specs released today are **VERY DIFFERENT** from your damn GUESS proposal.

implemented over the last year or so by SEVERAL COOPERATING Gnutella vendors

Oh, and what about your Remote Queueing feature? Shareaza founded that, and it's included in Limewire. Mike wants to cooperate, but your not giving him a chance.

The Shareaza people continue to attempt to preempt Gnutella as THEIR protocol, when in fact they are pretty much branching off from the network.

"Shareaza People"? There is only one developer for Shareaza, Mike. Shareaza supports G1 and G2, it supports "Gnutella".

I would like a copyright protected P2P network!

[miffo.swe]

Dont get me wrong its not to help the RIAA or anything. I just would like a peer network where you could exchange only free uncopyrighed stuff. Today such things drown in britney Spears and porno and goes unoticed. A browsable peer network filled with only free stuff would lift it up and give legitimacy to peer networks. Today people trying to distribute free or new music etc. dissapears totally from the surface under a big pile of copyrighted crap. Pirating is actually in essence hurting unsigned and struggling musicians.

Maybe im the only one in the whole world who doesnt like to pirate, i dunno.

Pr0n QoS issues

[0x0d0a]

Pr0n priority levels have been a concern of many P2P network users for some time. While the collection of protocol updates in gnutella2 has many general performance enhancements, it does (rather glaringly) lack any improvements specifically addressing the pr0n issue. A good deal of user upset has been produced by this -- however, fortunately much of the furor is undeserved.

Improvements to improve your pr0n viewing experience are well underway in many *clients*, rather than in the protocol itself -- protocol changes would produce compatibility issues. A number of proposed improvements include changes to the routing system to use dictionary-based priority changing. Query and result packets containing entries with phrases such as "tits", "ass", or "CowboyNeal nude" will be given an elevated priority in sending, improving latency for those users who really need pr0n. There has been some debate over whether this is entirely appropriate -- it reduces fairness -- but when it comes right now to it, pr0n-obtaining is a task with hard realtime constraints. The Gnutella developers and GDF members recognize that the goal of P2P software should be to best serve the community as a whole -- and so some unfairness will be allowed.

Preliminary dictionaries for the new routing prioritization may be downloaded from various of the GDF developers -- links have been posted in relevant discussion on the GDF board. The proposed dictionary format allows granting of variable priority -- "tits" matching a packet might increase the routing class by 1, but "teen lesbian slut" would increase it by 3, giving it priority over merely "tits" packets.

There is some concern over abuse -- that users hungry for low latency may simply include terms like these in their filenames. Indeed, a few users have already begun doing so. A second solution, perhaps blacklisting, may have to be used later on if this becomes a severe issue.

Because of the real-time nature of Gnutella, there are limits that can be placed on how much latency alteration that can be made. Queues are never massive at a single node, since most clients allow only relatively small send buffers. Early tests show 10% to 40% improvement on high-priority pr0n-containing packets. This is somewhat variable depending upon the network traffic -- if background traffic is composed mostly of smaller "ping" packets (instead of result packets), latency improvements tend toward the latter number.

There are a few other improvements on the table. Those of you that follow my work know that I'm interested in distributed trust used to rank the users. This trust network can be adapted to rank users based on the quality of the pr0n they serve, and give higher priority to users that give really top-notch pr0n -- unfortunately, this requires a client UI (and effort on the part of the user) to do manual ranking.

One more controversial proposal includes a Freenet-like network-wide caching system. Pr0n that is being frequently downloaded will be mirrored to as many systems as possible. This will improve download speed (and end-user experience, as people will be able to view locally-cached pr0n, and thus be introduced to new and interesting forms of pr0n). The p-cache (as it's already being called) even goes Freenet one better -- it is being designed to support speculative caching based on past searches. If your client catches even a hint, based on the searches that you've done, that you might be remotely interested in "cuties in French maid outfits on the beach", say, it will search and download all the related files it can find on the Gnutellanet. Aside from the massive cache of pr0n that builds up (if the user chooses to browse it), this is mostly user-transparent, yielding only low latency, high-availability pr0n searches tailored to your tastes.

Also notable is the support (for pr0n only, I'm afraid -- scarce network resources must be conserved) for multicast, introduced with the new UDP support. When you request a download of a large file, a remote server can give you a time offset until the file will be sent -- usually, an hour or two -- and will establish you as a "subscriber" of this file. When the time expires, the server will multicast-stream the pr0n file to all people that have subscribed to the broadcast. Now, an "hour or two" may seem like a long time, but it's far better than simply waiting in a unicast queue, possibly for days. You will need to be on the MBONE for this -- some college users or business users with videoconferencing may already have this handled, but the rest will need to request "MBONE support" from their ISP.

Try Furthur

[nestler]

Check out http://www.furthurnet.org.

Legitimate P2P sharing of live music.

Legitimate P2P

[mcrbids]

P2P most certainly DOES have legitimate applications... Ever hear of Bit Torrent?

Essentially, the idea is this: When you are downloading a file, when you receive a packet of data, you now have that packet of data, and there's no reason you can't immediately share that packet of data.

So, people downloading something from a Bit-Torrent capable site are themselves distributing the content... as it is being downloaded!

The end result is that a huge number of clients can download content (iso images, etc) from a site without increasing the total bandwidth usage of the site by much at all.

Check it out - it's pretty amazing!

A revolution in P2P? I don't think so

[Sanity]

"next revolution in P2P" (which it actually, IMO, is)

Hardly. Directed searches for information (rather than the Gnutella/Gnutella2)-style broadcast search, has been around for a while now. For example, Freenet has employed a directed search from day one (albeit with a slightly different application), and FASD is a good example of how this can be generalized to fuzzy searching.

Calling Gnutella2 the "next revolution in P2P" would be like calling the latest model in horse-pulled carriages the "next revolution in transportation" years after the advent of the motor car.

Re:A revolution in P2P? I don't think so

[Sanity]

You obviously have no idea what your talking about.

Have you ever heard the phrase "It is better to keep quiet and be thought a fool, than to open your mouth and remove all doubt"? Remember it, you will find it useful in the future.

Oh, and if you want to be taken seriously, perhaps you should have the courage to put your name to your utterances.

Gnutella2 isn't just a new searching mechanism.

It isn't a new searching mechanism at all, it is still using a brain-dead broadcast search. Link compression, partial file sharing, and the other features you mention are just putting lipstick on a pig.

More people will be using Gnutella2 than Freenet anyways.

Oh really, so you must know how many people are using Freenet?

Freenet's search methods dont' really work.

What? Have you ever even used it? I have, and it works fine for me.

So you couldn't really get out that "secret information" unless there were a kabillion sources. Not so on Gnutella2.

Utter bullcrap. You have obviously never tried to use Freenet.

Perhaps you could first (A) Read the spec

What spec?

(B) Download and use Shareaza

I don't run Windows.

(C) Learn more about the technology and methods it uses.

Sounds like you need to take your own advice.