Slashdot Mirror

Technically speaking, It's not impossible; The problem is that it's spammable/DoSable and will need an authority to either allow/deny nodes from inserting to index or someone like our good old friend 'hosts guy' to maintain a list of known good source nodes that people can download and only share the indexes from those.

No authority is needed, because there isn't one already. In the centralized index situation, no human validates torrents uploaded to the centralized indices. Instead, the users do. If you go search for any blockbuster movie you care to name on Pirate Bay, you'll get 50 pages worth of hits. The first 10 to 15 hits might be useful, with various bitrate encodings and various subtitles and audio tracks in them, and then it very very quickly tails off into utter trash. It doesn't seem to hurt Pirate Bay. Nobody ever selects the torrents with zero seeds unless they're looking for something so niche that there's no other option, and no one seeds bogus torrents. Even their pathetic originators give up extremely quickly.

And/Or other simple restrictions like limiting the number of torrents any node can add to the index.

In a decentralized index, that limit is only in the local node, where it is easily removed. Not worth bothering to write the code in the first place.

And/Or a voting system that allows all nodes to vote on others to help the client applications with prioritizing/filtering the index.

The seed count effectively serves as a voting system today. It's by far the most useful metric. About the only other useful metric is a user-defined list of strings. Quality video encodings tend to have some release group tag in the torrent name. Easy enough to push priority up a bit if the user's preferred string is present.

What's missing is implementing support for search within Mainline DHT. Kademlia DHT on which it is based has a scheme already designed:

Filename searches are implemented using keywords. The filename is divided into its constituent words. Each of these keywords is hashed and stored in the network, together with the corresponding filename and file hash. A search involves choosing one of the keywords, contacting the node with an ID closest to that keyword hash, and retrieving the list of filenames that contain the keyword. Since every filename in the list has its hash attached, the chosen file can then be obtained in the normal way.

Mainline DHT has omitted that functionality. If it were implemented, index sites would no longer be required.

Obviously Mainline DHT traffic would increase substantially, but it would still be quite small compared to torrent traffic. Also, if it were implemented exactly as described, clients would be responsible for filtering results coming in from the DHT. Most users want the logical AND of their search terms, but Kademlia specifies a logical OR. Performing that processing is simple enough though, and of course the client could present results much like web search engines do, with results that contain as many of the keywords as possible presented first, followed by results with fewer and fewer matches. You don't get the fuzzy matching most of the web search engines employ doing that, but as it happens, you also don't get fuzzy matching from Pirate Bay search anymore, so that's no loss. Client authors then have the option of preemptively fetching .torrent files in order to get tracker lists to be able to rank the results by how active they are, or of waiting to let users do some manual culling first. That whole process is substantially slower than a centralized index site. Mainline DHT is anything but fast, most of the time. It is, however, bulletproof. As long as the DHT exists, files could be found.

BEP 0005 specifies KRPC methods of ping, find_node, get_peers, and announce_peer. What's needed is a new BEP to extend the protocol, adding search_peers.

First of all, I apologise for the tone of my post... lkcl's posts frequently attract trolls and, posting at a late hour again, my hasty judgement got the better of me. :-(

When you give your bittorrent client a magnet link with a "btih" (BitTorrent Info-Hash) component, it starts looking up the DHT (asking other peers it's already made contact with) for peers sharing that infohash's torrent. As soon as it finds at least one, it connects to that peer ("joins the swarm") using the BT protocol and asks for the torrent's metadata (the.torrent file's info section). It is at this point where it will pop up a window asking you where to save the torrent and/or which files to download from it.

Given the above, one way that "gittorrent" could work would be the following:
1. Each Git object (file/file tree/commit) would be a separate "torrent", identified by its hash. Information about which peers have which hashes (i.e. objects) would be stored in the DHT. Nothing new for DHT so far. With this in place, you could checkout any individual git file/tree/commit and its history (each git commit references its parent).
2. Here we introduce a DHT protocol extension (let's call it "get_hash_addenda") using which you could get information stored in the DHT (this is the major difference with .torrent files) about newer commits. With this in place you would also find other users' "forks". This would also be useful for ordinary torrents by the way (get subtitles/new episodes for this torrent).
3. With these in place, you now have a (very slow) "gittorrent" implementation. Additional extensions/local DHT data caching/assumptions/whatever) would be used to speed the whole thing up. Existing BT trackers (fast peer lookup for a given infohash) would work as is. Existing git daemons would also work, and could also be extended to speak DHT and the BT protocol. All these things map well to existing BT concepts.

The protocol(s) used for transferring objects between peers could be the BT protocol (provided we treat each git object as a separate torrent, as lkcl recommended), and/or any protocol already supported by Git (as an analogue of "web seeds").

Example bare bones magnet links:

magnet:?xt=urn:git:<object id>
magnet:?xt=urn:git:<long hex number representing a user>/<repo>[/<branch|tag name>]

or even magnet:?xt=urn:git:<user>/<repo>/<branch|tag name>
if a way of having non-spoofable and globally unique nicknames is found (probably in an FCFS fashion).

A nice research project on extending BT (which already somewhat implements the "get_hash_addenda" functionality) is Tribler.

The earlier "gittorrent" effort that lkcl criticised can be found here. A cursory glance reveals an emphasis on trackers and almost no mention of DHT, probably due to it being written in 2008.

Disclaimer: All this is from a layman's point of view and horribly inaccurate. The DHT protocol extension especially could probably be avoided using some convention. I've been reading up on how BitTorrent and DHT works lately and, honestly, I can't blame you: apart from the BEPs it's basically UTSL.

There are 100 or so really good proxies out there(maybe tor end nodes?)

Tor would be possible, but tor means that every X minutes you would get a random endnode (it might not be every X minutes, the way tor works in detail is not my area of expertise), which is suboptimal at best for bittorrent. Tor is also more often than not slow, and unreliable for connections that take a long time. Typical use such as HTTP consists of relatively short connections (a few text files, a few binary files for images), but I do have to admit that I don't know if HTTP Keep Alive is respected out of the box by the tor bundles (and I doubt HTTP Keep Alive is such a good idea on an "anonymous" network anyway).

In general, Tor is as reliable as the weakest node in your circuit of nodes you're using as a proxy, which in my experience is not all that reliable at all.

I have no idea what that would do to the torrenting protocol, but it might make it have issues recognizing available resources.

A peer in the bittorrent protocol put simply is a pair of an IP and a port, and it has no trouble distinguishing between other clients on the same IP. It's a little bit more complicated (there's a randomly generated 20 byte ID involved, but as far as I can remember trackers aren't required to send that to other peers in compact mode). Even when you're not using a tracker (by such extensions to the protocol as DHT or PEX), it's still not much of a problem. I've recently spent a bit of time reading up on the protocol (do check out the BEPs if interested, there's some really cool stuff in there) for a little hobby project.

The thing is, without joining the swarm there's really no way to keep track of the peers in the swarm other than by IP and port, and even then. Say for instance you're a member of the swarm and you've got a provider that uses DHCP. The researchers script then does the announce request to the tracker to get a list of peers and among the randomly returned peers is your IP and port number. After about an hour or so, you leave the swarm, shutdown your computer and go to bed. You release your IP back into the ISPs DHCP pool, and 10 minutes later the guy who lives next door turns on his computer and gets that IP. His bittorrent client by default chooses the same port number as you. Suppose that the tracker only returns IP-port pairs in compact mode, instead of the 20 byte id, ip, port triplet, then there is no way for those researchers to see that the peer has become someone else. The chances of this happening are small though, unless it's a really really popular torrent, and even then the handful or random peers returned by the tracker would require quite a bit of luck to have the same peer in it twice on sufficiently large torrent.

Next, even IF the tracker returns the 20 byte id, IP and port triplet, the 20 byteid is supposed to be randomly generated every time the bittorrent client starts. There is an optional field that you can pass as a unique id to the tracker for keeping track of you, but peers in the swarm will never see that. So basically, you can't really tell who is who from IP-port pair. It is a peer, but two different peers over a period of time could be the same person.

Everyone here has probably heard this, but an IP address does not represent a person.

From the press-release from the univesity about the paper:

"In order to remain anonymous, - explains Professor Rubén Cuevas - many of them rent servers from companies that perform this service and then publish contents from those servers".

Are you sure they are renting it, and didn't just script-kiddie their way into the server? Explain to me the logic in paying for something, with mon

There have been proposals to make use of partial matches (and papers about how to do so effectively), but to my knowledge, nothing has come of them.

Bootstrap is the interesting issue.

It is indeed. The bittorrent DHT solution is based on Kademlia (or the BEP for Bittorrent specifically). If you google a bit you'll find a few papers and some interesting things, including attack vectors. I'm implementing a version of Kademlia at the moment to have nodes in a network find other nodes for accepting work in a distributed environment, and bootstrapping the thing is "the weakest link". You could set up multiple bootstrapping nodes, but suppose that a network failure takes out your access to the bootstrapping nodes you're basically humped when you want to join the network.

Sure, once a node is online and given enough other nodes stay online enough of the time, it would be possible to have a persistent network.

What you're referring to is known as "churn" and provided the network is large enough it becomes less of an issue. There's actually an interesting paper on churn in Kademlia, but I'm sorry to say I can't seem to find it anymore. As it turns out, Kademlia is quite resistant to churn provided there are enough nodes in the network. Not really surprising, but it's a nice read if you want to know how resilient the network is without having to test it yourself.

I suppose you could do something like search google for random torrents, join in, test the folks you connect to for being part of the decentralised network, grab network info from there etc. It still uses google as a central reference point but it would be more robust than having some sort of hard-coded 'peer tracker' server, or using any sort of brute-force port scan of the internet.

My current solution involves trying multicast and if that fails a broadcast to find other nodes if the bootstrap servers are down. This usually allows you to find other nodes on the LAN (depending on the network configuration) and once you have a single node in the network you can start doing lookups. Few admins like the idea of you "scanning" the network to find other nodes, understandably. Of course, this method of discovering nodes has disadvantages as well, for instance you could have a segment blissfully unaware of another segment in the network until the bootstrap servers come back online.

However for typical bittorrent use I doubt that this particular strategy will work. The work I'm doing has little to do with bittorrent, or how P2P is traditionally seen by the bulk of its users. I doubt that the implementation I'm working on is directly usable for another problem than the one I'm working on.

I'm using a private tracker with DHT disabled?

The "private torrent" extension to BitTorrent is completely insecure, and fairly easy to subvert. See this discussion amongst others.

Additionally, unlike public trackers, private trackers perform extensive logging. If the tracker gets busted, the police have detailed information on your activities for months in the past.

As far as I know, nobody has come up with a de facto distributed, anonymous replacement for trackers.

*ahem* BitTorrent uses the Kademila DHT for distributed tracking, and it works really, really well so far. I haven't used a centralized tracker for a year or so now. There's also peer exchange (PEX). The trackers aren't the weak points of BitTorrent, and haven't been for years. The weak point is in .torrent distribution, with commenting/forum, seed/leach counts, search, etc. Magnet links partially resolve this by moving the .torrent files out to the DHT as well.

I'm in Pittsburgh (Oakland neighborhood, near Pitt/CMU), and I can now pick up all major local stations via digital OTA (CBS, ABC, NBC, Fox), as well as PBS. I'm in the southeastern corner of my building, so ABC (broadcasting from the east) comes in the strongest (almost don't even need an antennae). PBS is broadcasted right from my own neighborhood (Mr. Roger's neighborhood? Yeah, seriously ;-), so I really don't need an antennae there, either. Took me awhile to figure out how to get Fox, CBS, and NBC. Fox and NBC took a little fiddling with the antennae direction and gain, and CBS takes a little bit more. But now, I get more or less all the important channels (plus a few shopping and religious bullshit channels, too). I only get one digital channel from CBS & Fox, but I get all three of the broadcasts from ABC & NBC (their main digital HD, local weather, and the 'extra' channel, which appears to be mostly reruns at this point). And for the little bit of actual worthwhile stuff that comes in off of cable (e.g. Daily Show, Colbert Report, Stargate Atlantis, Eureka), I just go to either hulu or bittorrent,... ;-)

It's been many years since I've followed Freenet, but I'm pretty sure you're wrong on the "security through obscurity" bit. The security is derived from his graduate research on the properties of routing in small world and scale-free networks. Plenty of papers and presentations on the searching that made its way into the 0.7 release can be found at http://freenetproject.org/papers.html. And as for lack of documentation, I won't argue with that, but don't forget that early in BitTorrent's development lifetime, Bram Cohen only put up only a vague specification of the protocol (but now vastly updated, see http://bittorrent.org/beps/bep_0003.html), but a separate Wiki was established to document all its nuances after pouring over the source code (see http://wiki.theory.org/BitTorrentSpecification). This Wiki begat the hundreds of alternative BitTorrent clients you find today. The same thing could happen with FreeNet, by other people with "years of prior Java experience." (And please, it's not like Java was written for the programming elite and takes that much time to master. My first encounter with Python was reading over the piece-picker module for the original BitTorrent code, and the language barrier was easy to overcome.)

there is also a UDP Tracker Protocol for BitTorrent, UDP doesn't even hear the RST packet. Comcast will have to figure out a way to turn off something that doesn't have an off switch.

Not necessarily.

Since nearly every router can be presumed to have a wide-open (and likely quite fast) pipe to the Internet, there are plenty of ways to get around the need to have a central server. Some others are rather unknown, or even a bit old, but those reasons by themselves don't make them inapplicable to the role.

Storage for all of this can be a problem, but that's an easy one to solve: The small size that such a worm must be combined with the relatively large amount of bandwidth available on each infected host means that only a very small percentage of them need to be able to store a quantity of files for the rest of the network to consume. As luck will have it, a substantial portion of these routers will be connected by fast Ethernet to Windows share, which these days means that there's a good chance of having multiple gigabytes of storage available without anyone ever noticing, let alone anything being logged.

(And, of course, the routers will be able to share and relay different versions of the worm amongst themselves locally over WiFi -- just try tracking that.)

It doesn't take a rocket scientist to connect the rest of the dots, so I won't bother.

An ambitious programmer of the caliber needed to devise such a beast to begin with wouldn't see much of an impediment with these vast resources. With careful and diverse seeding of the first round of infection, such a worm would be very hard to stop, let alone trace back to its originator.

Plus, it really only deals with the Gnutella network, whereas most of the traffic nowadays would probably be using Bittorrent.

And throwing up your mp3s on the band web site is trivial when you're a local hit and expecting a couple thousand downloads; it's not quite the same when you're hoping for millions of downloads.

I guess you haven't heard of Bit Torrent which does a remarkably effective job of scaling to massive sizes without linear increases in bandwidth usage? Yes, that big, oppressive black thing is the rock you've been hiding under - get out!

At one point, suprnova.org related traffic accounted for something like 40% of all the traffic on the Internet. Yes, 40%, internationally. You can be quite certain that they had nowhere near that kind of hosting capacity directly...

Up to the connection/process limits of the server, BitTorrent allows you to serve 1000 clients with virtually no increase in total bandwidth usage over serving maybe 2-5 clients with HTTP.

No, it's not quite the same as Deezer's "on demand" music system but it still allows for massive distribution of material at minimal cost.

(Bittorrent is an ISP's friend because it can keep large amount of bulk data transfers within their network).

This is just retardedspeek. Who comes up with this kind of WTFery?

BitTorrent communicates with peers ALL OVER THE WORLD. That does NOT constitute keeping those data transfers "within their network" unless you actually mean "outside their network". See the spec: http://bittorrent.org/protocol.html

For a more efficient protocol that actually DOES cache transfers and cut external bandwidth usage, see NNTP. And don't pretend you can speak for your local ISP until you ARE your local ISP, and have ingested a healthy dose of reality.

Have you ever tried setting your upload rate to zero and looked at what it does to your download rate?

That's because ACKs are counted as upload, not because fairness is built into the protocol. Other peers can't even tell how much you're uploading, because nothing is stopping you from lying about it.

The protocol is very simple and documented online. You will notice that it includes information to allow clients to implement, and I quote, "tit-for-tat-ish" algorithims, but those are not a part of the protocol, and since the protocol is open there is nothing stoping clients from gaming the system. Yes, your client can choose to decide if it should throttle uploads to a particular peer that it thinks is abusing the system, but your client would be guessing, and that guess would be based largely on information provided by the downloader.

Seed-heavy torrents (like, say, most major Linux ISOs a couple of weeks after a release), end up choosing a very low upload rate, because they get the data fast anyway. More "normal" torrents, with a mix of seeds and leeches, download faster with an upload rate set close to the maximum upstream bandwidth of your connection.

The optimum upload rate for maximal download speed is purely a function of the chunk size. There is a reason that tracker sites employ ratio quota systems. Those are exploitable too though.

Go on thinking you're right though. It keeps the downloads fast for us leaches.