PDTP - The Best of Both FTP and BitTorrent?

ikewillis writes "For awhile I've been following the development of PDTP (Peer Distributed Transfer Protocol), which is trying to merge the concepts of FTP and BitTorrent. This sounds like it could be useful for apt-get repositories or other high demand FTP sites. It's designed to be used as part of scalable networks which could replace manual selection of FTP mirrors. It also supports a number of other nifty features like cryptographic file signatures. Isn't it about time we ditched FTP for something better?"

Comments

BitTorrent Mod That's Neophyte Friendly

[chatooya]

BannedMusic.org made a BitTorrent wrapper that installs the application and then automatically launches the download, they call it an "easy downloader" and have instructions and a script for sites that want to make their own. Makes it a *lot* easier for sites to give out big files to non-techy audiences.

Re:Sigh....

[Smitty825]

Heh...a few years ago, /. made an April fools joke about Python and Perl merging into a new language called "Parrot" Apparently, some people liked the idea, and started the project. I have no idea of its status, though :-(

Re:Sigh....

[Martin Blank]

Currently at v0.1.0, awaiting Something Big in Perl 6, it would seem.

P2P Research

[Demandred]

There are several P2P research projects that are looking at building reliabale and scalable P2P systems.

Take a look at Tapestry, and Chord (and read some of the papers) to understand the issues involved in providing scalable and high performance P2P services. Not only is scalable search and overlay graph connectivity an issue, but also node failure and short session times of P2P nodes.

Additionally, when you actually handle the issue of downloading data, building application-lvel multicast trees to distribute the data efficiently on a large scale is not easy. Two papers from SOSP '03 SplitStream, and Bullet address that issue.

need lightweight clients, not installers

[SuperBanana]

BannedMusic.org made a BitTorrent wrapper that installs the application and then automatically launches the download, they call it an "easy downloader" and have instructions and a script for sites that want to make their own.

And unfortunately, it's windows only, and still requires installing the software, which is 3MB+.

What is needed is something along the lines of a very small, very simple java client or a browser plugin. Azureus is java, but is huge and has massive feature-bloat for the purposes of just downloading(and sharing back) one file. However, Bram and others don't seem terribly interested in expanding possibilities; a mac developer offered up numerous improvements to the BitTorrent team for the mac client(which among other things is based on 3.3a, not 3.4.1, weeks after 3.4.1 released) and was rewarded with deafening silence.

The bittorrent protocol is http based. It's extensively documented on the bitconjurer website. Cmon folks, let's at least see a mozilla plugin or something! :-)

something better == sftp, or HTTP!

[sPaKr]

I thought something better was sftp. As for distributions.. why not HTTP? Setup one reflector that dynamically kicks outs redirects as new mirrors come online. This is mutch better as we have a ton of clients already installed (curl,wget,..etc) We also have load balancing, dns round robin, authorzation, security(read: SSL) well defined in the protocol. All we need is a cgi script to kick out the redirects, and another that will make signature files based on the publically available SSL cert. Whamo all the same features.. and we didnt have to reinvent the wheel.

Re:about time

[PhiberOptix]

actually it is called sftp. scp is a substitute for rcp.

QPL is OSI approved

[tarcieri]

Hello. I'm the project manager for PDTP, and author of Skyfire. There's nothing wrong with the QPL whatsoever, unless you mind the fact that it's GPL incompatible (but then again, so is the Apache license). The QPL is an OSI Approved license, so there's nothing to worry about.

Re:The concept is great, but...

[tarcieri]

Hello. I'm the designer of PDTP.

...mirrors would need to be in sync at all times for this to work. Otherwise your PDTP client is only able to download from the mirrors that are in sync, or worse, will get some chunks from files that aren't up-to-date, causing problems. Unfortunately, it's (almost) impossible to mirror new files instantaneously, so mirrors are never all in sync, all the time.

I suggest you look at this page with graphic illustrations of PDTP networks for a better idea of how PDTP works. There is no concept of a "mirror" in a PDTP network. The Source Server is the central authority on all files being distributed over the network, and notifies all servers/piece proxies on the network whenever files become available or unavailable. Like BitTorrent, the network is largely self-sustaining, with clients uploading pieces to each other and verifying their integrity with MD5 or SHA1 checksums. Files are tracked on the network with integer keys, so if a file were altered its name would simply be mapped to a new key, and the entire network would be notified that the previous version is no longer available.