Slashdot Mirror
PDTP - The Best of Both FTP and BitTorrent?
ikewillis writes "For awhile I've been following the development of PDTP (Peer Distributed Transfer Protocol), which is trying to merge the concepts of FTP and BitTorrent. This sounds like it could be useful for apt-get repositories or other high demand FTP sites. It's designed to be used as part of scalable networks which could replace manual selection of FTP mirrors. It also supports a number of other nifty features like cryptographic file signatures. Isn't it about time we ditched FTP for something better?"
Interesting... this could bring piracy back to the ftp world, rather than the emule appz or bittorrent world where it's easier to get caught.
Files (or file segments) could be matched up using hashes that ensure that the proper files are sought and grabbed. MD5 could provide the primary file hash, and then a faster hash could be used for individual segments. The hash could be calculated at the beginning of the segment transfer as part of the handshake process, then stored on the client box for comparison once the segment transfer is complete. If the hash matches, then it's saved and the system continues. If not, the segment is dropped and a new source is found, with an option to simply ignore anything from that host, either for that specific file or globally, at the user's option.
PDTP networks could have synchronized user accounts (assuming the networks aren't too large) for priviledged file access, with periodic synchronization with other members upon account changes. Content might be another matter, but if a file hasn't completed transfer, then perhaps it would simply not be marked as available for open transfer.
You can never go home again... but I guess you can shop there.
That's just great! Now the media will consider FTP a movie-stealing method. Then the MPAA will call a ban to all FTP servers!
HTTP does all that. There are well-defined
and well-implemented (Squid) cache-tree protocols
for HTTP. This is very old stuff. FTP is just
plain obsolete. It ads *zero* value over HTTP,
and it's harder to use. Trying to bring FTP up
to the standards of HTTP is a futile effort too,
since HTTP is mature on many more dimensions,
and does not suffer from the gross defects of
the more primitive FTP such as transmission of
port numbers as stream data.
-I like my women like I like my tea: green-
Public: In which case http/bitorrent are good choices currently. I can see that pdtp could be better than bit torrent for this.
Private: This includes transfering stuff to and from work, or for a small number of family to access photo collections. For this I currently use ssh/scp/sftp, rsync and scponly. These tools give me reliable, and efficient methods for secure personal file transfer. "scponly" provides a limited chrooted shell to allow specific users only access to a set directory using scp/sftp/rsync.
So yes I can see a place for a pdtp on my box, where it would complement other file transfer systems I have.
Elivs
the guy at autopackage.org was attempting something simmilar to this but for package distrobution...it looks like with this protocol, youjust need to set up all the OSS servers with packages on them and boom...you have one huge honkin FTP site with all packages nessisary for all things...then you just ned to download a discription file and then the package manager can grab all the packages from a few PDPT gets and your done...good bye RPM hell.
I am the Alpha and the Omega-3
No kidding, today has been ridiculous.
There's a point where it's funny, but then there's a point where it's really just overdone. I haven't even read half the stories posted, but it seems like they're all fake... and if they aren't, Slashdot is really ruining the credibility of some people. Not sure if the BSD on Gameboy is real or not, but if it is, who's gonna believe it?
You don't see CNN taking the day off. "OSAMA BIN LADEN CAPTURED!!", you click the link and read a long drawn out story that COULD be true, but at the bottom: "...April Fools!"
Karma to burn. *shrugs*
We have secretly replaced these Slashdot mods' sense of humor with a rusty nail. Let's see if they notice!!
I checked the drive to find it was filled to the max with over 100 GB of porn! Lucky I always travel with a spare hard drive =)
From excellent karma to terible karma with a single +5 funny post...
What in God's name is wrong with x86? I hear this all the time. It's just like Slashdot's bizarre fascination with bashing X because it's old. But interestingly enough, X continues to thrive for the same reason x86 continues to thrive: it works, works today, and works with your old applications. The fastest desktop and workstation processors on the planet are x86.
Ok, sure, CISC is dead, x86 is a convoluted mess, yada yada yada, but engineers have gotten around many of these problems with the instruction set, and compiler writers have gotten around the rest. There is absolutely no reason to drop x86 for something new at this point.
And don't get me started about the political crap at the end. I find it pathetic that you have to inject your political propaganda into a post on a technical site about a new file transfer protocol. I don't care what candidate you support, that's just ridiculous. You're just an attention starved karma whore. It makes me sick.
BitTorrent suffers another problem in that the only usable implementations are currently only available in Python. The primary problem with Python is its excessive resource usage
Really? I'm currently running four throttled BT downloads on a PII-350 w/64MB. Max CPU usage is 8%, load average 0.25. If you're really that bothered see here for an alternative.
but other problems arise such as integration of the Python implementation into a native GUI frontend for a given platform
Ever heard of WxGtk? RPMs for most distros, if it wasn't part of your default install.
as well as the need to bundle the Python runtime with the BitTorrent client on most platforms as few deployed systems have a Python runtime available
Now this is just silly. I dont think there is a linux distro which doesn't include Python libraries and even for Windows it's a single small executable. Besides (correct me if I'm wrong) but isn't one of the reasons for using Python that it has bounds-checking on arrays and is therefore proof against the cause of most exploits - the buffer overrun?
The main problem with the "BitTorrent" idea is that it gets associated with "illegal" actions.
I was on the "Desert Combat" Testers team and we had to download 600-700mb patches once a week... off one ftp server.
When i mentioned the idea of using a modified BitTorrent client/server to ease the strain on the server i was told we could not use "illegal" tools.
First educate the public and then start to think about upgrading things to help the internet not crash and burn.
Phil
A psychopath can't tell the difference between right and wrong. A sociopath knows the difference - he just doesn't care.
Hmm, I was thinking about this earlier.. does anyone actually have any statistics for how much server transfer badwidth was saved by distributing a popular file (latest anime release or something) over BitTorrent? How much does it actually help?
Others have stated that the bogusness of nearly all of /.'s content today may harm the reputation of some people. BSDs on Gameboy? PDTP? Gateway shutting down all stores? Which of these is/are true, if any at all?
But what concerns me even more is this: Some people do not understand the /. community, and might not understand that much of the content here is bogus today. Therefore, there is the possibility of bogus information being propogated as correct information, and that has the potential to make a lot of people look bad.
The one about UML Dating Patterns or whatever was good. That could have been a good April Fool's joke. In fact, I think /. could have been more sophisticated by preparing, over the year since last April Fool's, a really killer story, something that will totally blow peoples' minds... but only ONE story. Make it simple enough that it seems legitimate, but just humorous enough that someone might question their sanity. Now, most of the stories are true, one of them is false (or partly based on truth)... Which one is it? This might drive people into reading the linked sites, in an effort to figure out if they're legitimate or not, and in the process, they might learn about some products, like PDTP, that could be useful in the future. April Fools could, therefore, have constructive results, in addition to being funny.
Or even funnier... make a story that looks TOTALLY BOGUS, but turns out to be 100% true.
But overdoing it the way /. did today was just ridiculous. I even got a little too excited this morning and inserted a lot of explitives in my garbage posts under the garbage stories... But after a few minutes, it got old and I regretted it.
Erasure codes have the property that for a file N packets long, you calculate some number K of coded packets, and the receiver needs only to receive N of any combination of coded or source packets to be able to recreate the original file.
For instance, I could have a file 100 packets long, and calculate 25 coded packets, then I could receive, for example, 80 packets of the original file and 20 coded packets (80+20=100) and recalculate the entire original file. Or 90 original and 10 coded packets. Or 75 original and 25 coded packets...etc.
Before Tornado codes, this was computationally difficult to do in practice for large files. Typical use of Tornado files is to just send all coded packets, and receivers can "fill their cup from the fountain, dipping it into the stream whenever they want" to get any N number of packets to recreate the original file. Obviously, this makes a lot of sense in a multicast domain.
But for distributed file transmission, I'm not sure this makes sense. You would need to collect N different packets. If you got the same coded packet more than once, it would not help you.
Keeping track of which unduplicated coded packets you have and need is just as difficult as keeping track of which unduplicated original packets you have and need.
Packet loss is not really much of a problem if you are using TCP. On the other hand, it is a problem when you are doing multicast UDP over the Net or satellite.
So overall I'd say file-level FEC using erasure codes is pretty much useless for distributed file transmission.
Anyone care to differ?