PDTP - The Best of Both FTP and BitTorrent?

ikewillis writes "For awhile I've been following the development of PDTP (Peer Distributed Transfer Protocol), which is trying to merge the concepts of FTP and BitTorrent. This sounds like it could be useful for apt-get repositories or other high demand FTP sites. It's designed to be used as part of scalable networks which could replace manual selection of FTP mirrors. It also supports a number of other nifty features like cryptographic file signatures. Isn't it about time we ditched FTP for something better?"

Comments

BitTorrent Mod That's Neophyte Friendly

[chatooya]

BannedMusic.org made a BitTorrent wrapper that installs the application and then automatically launches the download, they call it an "easy downloader" and have instructions and a script for sites that want to make their own. Makes it a *lot* easier for sites to give out big files to non-techy audiences.

Re:Sigh....

[Smitty825]

Heh...a few years ago, /. made an April fools joke about Python and Perl merging into a new language called "Parrot" Apparently, some people liked the idea, and started the project. I have no idea of its status, though :-(

Re:This isn't fair...

[LostCluster]

At least you're giving me some credit for my quotes...The Reuters Wire used my words without mentioning my name earlier today.

Re:Sigh....

[Martin Blank]

Currently at v0.1.0, awaiting Something Big in Perl 6, it would seem.

P2P Research

[Demandred]

There are several P2P research projects that are looking at building reliabale and scalable P2P systems.

Take a look at Tapestry, and Chord (and read some of the papers) to understand the issues involved in providing scalable and high performance P2P services. Not only is scalable search and overlay graph connectivity an issue, but also node failure and short session times of P2P nodes.

Additionally, when you actually handle the issue of downloading data, building application-lvel multicast trees to distribute the data efficiently on a large scale is not easy. Two papers from SOSP '03 SplitStream, and Bullet address that issue.

Re:P2P Research

[Demandred]

Actually, it does have major scalability problems...please refer to some of the numerous papers published on exactly this topic.

The supernode/ultrapeer addition to the protocol was meant to address some of the scalability issues of the previous version of the protocol. However, the fragile nature of the original overlay network is only made worse since ultrapeers are explicitly meant to be highly connected nodes.

The problem with Gnutella is that it can't do search efficiently due to the the broadcast nature of forwarding query messages. Research such as Random Walkers(SIGCOMM 02), and Attenuated Bloom Filters (IEEE INFOCOM 02) have tried to tackle the problem of scalable and fast search in unstructured P2P networks like Gnutella.

Re:Sigh....

[Anonymous Coward]

Parrot appears to be alive and well here. Version 0.1.0 was released on Feb 29th.

need lightweight clients, not installers

[SuperBanana]

BannedMusic.org made a BitTorrent wrapper that installs the application and then automatically launches the download, they call it an "easy downloader" and have instructions and a script for sites that want to make their own.

And unfortunately, it's windows only, and still requires installing the software, which is 3MB+.

What is needed is something along the lines of a very small, very simple java client or a browser plugin. Azureus is java, but is huge and has massive feature-bloat for the purposes of just downloading(and sharing back) one file. However, Bram and others don't seem terribly interested in expanding possibilities; a mac developer offered up numerous improvements to the BitTorrent team for the mac client(which among other things is based on 3.3a, not 3.4.1, weeks after 3.4.1 released) and was rewarded with deafening silence.

The bittorrent protocol is http based. It's extensively documented on the bitconjurer website. Cmon folks, let's at least see a mozilla plugin or something! :-)

something better == sftp, or HTTP!

[sPaKr]

I thought something better was sftp. As for distributions.. why not HTTP? Setup one reflector that dynamically kicks outs redirects as new mirrors come online. This is mutch better as we have a ton of clients already installed (curl,wget,..etc) We also have load balancing, dns round robin, authorzation, security(read: SSL) well defined in the protocol. All we need is a cgi script to kick out the redirects, and another that will make signature files based on the publically available SSL cert. Whamo all the same features.. and we didnt have to reinvent the wheel.

Re:about time

[dresgarcia]

"Isn't it about time we ditched SCP for something http://samba.anu.edu.au/rsync/better?"

Rsync. You can even have it checksum your files. . .

Re:about time

[PhiberOptix]

actually it is called sftp. scp is a substitute for rcp.

Re:Nitpick(s)

[Rick and Roll]

Python is case-sensitive, BASIC is not. Not being case-sensitive is a big advantage for newbies.

You could try teaching them LOGO or lisp, but lisp is boring and LOGO is no better than BASIC.

Hey genius...

[Anonymous Coward]

QPL is OSI approved

QPL is OSI approved

[tarcieri]

Hello. I'm the project manager for PDTP, and author of Skyfire. There's nothing wrong with the QPL whatsoever, unless you mind the fact that it's GPL incompatible (but then again, so is the Apache license). The QPL is an OSI Approved license, so there's nothing to worry about.

Re:The concept is great, but...

[tarcieri]

Hello. I'm the designer of PDTP.

...mirrors would need to be in sync at all times for this to work. Otherwise your PDTP client is only able to download from the mirrors that are in sync, or worse, will get some chunks from files that aren't up-to-date, causing problems. Unfortunately, it's (almost) impossible to mirror new files instantaneously, so mirrors are never all in sync, all the time.

I suggest you look at this page with graphic illustrations of PDTP networks for a better idea of how PDTP works. There is no concept of a "mirror" in a PDTP network. The Source Server is the central authority on all files being distributed over the network, and notifies all servers/piece proxies on the network whenever files become available or unavailable. Like BitTorrent, the network is largely self-sustaining, with clients uploading pieces to each other and verifying their integrity with MD5 or SHA1 checksums. Files are tracked on the network with integer keys, so if a file were altered its name would simply be mapped to a new key, and the entire network would be notified that the previous version is no longer available.

Re:BT Bandwidth-saved?

[colonslashslash]

Myself and a friend run tlm-project.org where we use BitTorrent to distribute Linux ISO's. We don't have any comparitive statistics vs standard FTP transfer, but obviously, as soon as we have seeded out an ISO to one or two people, our bandwidth constraints are halved. Then, when those seeders dish out the files to a few others, our server pretty much gets to kick back and relax, or devote bandwidth to our other torrents.

All in all, I love the BitTorrent protocol, I think its a great implementation for something like we are doing, although, as another user said, it is a shame about the "illegal" reputation it seems to have gained.

FTP vs TELNET/SSH

[rjch]

"Isn't it about time we ditched FTP for something better?"

Yes, it is. However, SSH has been around for a significant time and still hasn't replaced telnet, even given the horrific security holes in telnet.