Bridging Torrent and RSS

lerhaupt writes "PEP, the Prodigem Enclosure Puller, is a small php script which find all the enclosures in an RSS 2.0 feed URL, and utilizing Prodigem's new bittorrent API will have a torrent created and seeded for each. As an example of just what this exactly means, Prodigem is now using PEP to automatically torrent the top items found in the del.icio.us popular video feed. In general this now means distribution via bittorrent can be had with almost zero work or duplication of effort."

Comments

PEP source code

[NoInfo]

PEP is less than 400 lines of PHP. Here's the source code for the curious:
http://prodigem.com/code/pep/pep.txt

(from the PEP home page)

Re:PEP source code

[tolan-b]

Slight correction:

# : For Bash, PHP and many others

// : For most languages with C like syntaxes (Including PHP as that's the example we have here)

/* something */ : Ditto

<!-- something --> : HTML

Re:PEP source code

[pmjordan]

Sorry to be pedantic, but the comment syntax for HTML is actually , as it is in XML, and also SGML, since that is the 'big daddy' of both markup languages. SGML and HTML actually have more than just that, but:

The Wikipedia Article on comments gives you a list of the comment syntax of quite a few programming (and markup) languages, so I won't make an inferior attempt at doing so here.

~phil

anybody...

[Anonymous Coward]

have a torrent for this?

seeds please.

So, for us who can't code

[RealisticCanadian]

Am I correct in believing we will soon have an even greater wealth of torrents readily available to the users of the World?

If so, three words for the *AA's: Yer Fucked. Cope.

:^D

Hmm...

[FlyByPC]

Automatically-downloaded RSS, distributed over the Net;

Automatically-initiated Bittorrent downloads;

...Is it just me, or with the addition of a few local scripts, does this sound like a virus-writer's favorite fantasy? They need to make sure there are some very good safeguards on this!

Then again, with Genetic Algorithms, perhaps this is the next missing ingredient for cybersentient life...

Re:Hmm...

[glesga_kiss]

This was done like a year ago and there have been no issues so far. Remember, most folk will be downloading non-execuatable media, and any binaries will not be automatically executed on download. Downloading pirate binaries has always been "at your own risk". Really, this is no more dangerous than the same number of users manually clicking the link. Neither way is less vunerable WRT viruses.

Azureus and other BT clients already have RSS readers, using Regex's to match media in the RSS xml. This rules for TV series that are currently airing, it's great to return home to two or three new episodes each day. Many BT sites have RSS feeds, however the only one that got it right was btefnet, who had the inteligence not to post the same media twice. Most other sites have moron users who will post a 200 meg version of the file, followed by a 50 real media rip that no one wants but wastes our bandwidth because we get it anyway!! Throw in ratio sites, users dropping in 4 gig "season 1 dvd rips" and you can see why I stopped using it in the end. We just need a BT site that "gets it". Answers on a postcard please...

Just another pollutant

[rahlquist]

Much like breathing outside, having scripted automatic downloads of 'anything' in a feed isnt a great idea IMNSHO. Sounds like walking out the front door and having the exhaust from a diesel blown in my fact just because I said I liked trucks.....

Re:Just another pollutant

[Skye16]

Sure, for everyone who isn't behind a strict firewall.

:(

Re:Just another pollutant

[Bogtha]

having scripted automatic downloads of 'anything' in a feed isnt a great idea IMNSHO.

Did it ever occur to you that feeds themselves are scripted automatic downloads?

Re:Just another pollutant

[Bogtha]

A RSS feed is chosen by a conscious decision to download a little bit of data at a regular interval. Joe Schmo sets something like this up and we could have 1000's of people downloading gay p0rn when a web site gets hacked, etc.

Firstly, if somebody hacks a website, they can put gay porn on it anyway. This script doesn't change that.

Secondly, if you are objecting to the fact that people can be tricked into downloading unwanted videos instead of simply unwanted pictures, then a) you can do that with normal web pages anyway, and b) it's enclosures that do this, not this script.

But most importantly, you seem to be viewing this as some kind of tool to automatically download things via BitTorrent. That isn't the case. This is a script that sets up a normal web page, with normal BitTorrent links, that just happens to get the original data from an RSS feed.

As far as end-users are concerned, they don't go anywhere near the RSS feed. This just looks like an average SuprNova style web page to them. There's no automatic downloading by end-users.

Terrible writeup

[Bogtha]

Okay, to clarify a few things:

Enclosures are basically the RSS way of providing a link to an external resource instead of having a normal entry. Think podcasting - basically an RSS file with links to MP3s instead of textual entries.

What this tool seems to be intended to do is take an RSS feed, download all the external resources from it, then generate and seed torrents for each external resource.

For those of you thinking that this is a way of distributing RSS feeds via BitTorrent, think again - the feeds are distributed normally, and this doesn't let existing feed readers do anything new with BitTorrent, they'll still be downloading both the feeds and the external resources though HTTP.

So basically, this would take a podcast, download the MP3s, and generate/seed torrents for each of the MP3s. The torrents would then appear on this PHP page for people to download, but feed readers wouldn't know anything about it and carry on operating exactly the same as normal.

EPIC 2014

[USSJoin]

This, IMHO, totally rocks. Seriously: the computer now handles its own slashdot effect, intelligently. Truly, this allows for media-centric societies (like EPIC 2014) to be propogated; when we don't get that "Too many connections, MySQL fails miserably" message with popular content (even on normally-unpopular webpages), we are able to much more quickly diseminate information of interest to all of us.

Kudos to the developers; I, for one, am impressed.

Re:not-so Terrible writeup

[lerhaupt]

Just to clarify, Prodigem comes with a torrent feed for each user. So once all your enclosures have been converted to .torrents, you can provide people with your new feed. For example, http://www.prodigem.com/torrents/rss/pep_delicious .xml

So it nearly happened...

[samjam]

Someones job has been replaced by a small PHP script.

Shell script to follow :-)

Sam

Re:Video?

[Thuktun]

Superb hosting 4800MB Storage, 120GB bandwidth, $7,95.
Kunowalls!!! Random sexy wallpapers (NSFW!).

Would it be asking too much for you to keep your sig to your sig?

Oh, How The Litigators Are Gonna Love THIS

[ausoleil]

Automation of Bittorrent was inevitable. More or less, torrents become the functional equivilant of the automated podcast gathering programs.

Things is, most podcasts are original content. Much of the 'torrents, and let's be honest, are not, and they are not exactly sanctioned (meaning they're pirated works.)

Gee, do you think that the MPAA legal goons will be among the early adopters? Think that they will have the RIAA folks for company?

Bet the farm on it.

Come to think of it, one must wonder when or if the adult industry will resort to infringement lawsuits to protect their unique content...sure there has been a scant amount of it, but eventually, someone is going to pay big for those Jenna Jameson clips. (LOL.)

I would like to envision a future internet...

[Anonymous Coward]

... where virtually _all_ content that is viewable on the web is distributed in the same manner as bittorrent content - where your web browser shares all content (including images, and any other embedded objects that may display, such as video or audio [with the exception of realtime live streams]) that is on the web page(s) that the browser is open to (usually uploading the whole time the reader is spending viewing the page). Browsers would contact the server to initially request the page, and then contact other peers to download the actual content.

It would completely eliminate the slashdot effect, would likely have the general positive effect of improving people's download speeds, and would have the upshot of making it impossible for any leglislating body to make software like bittorrent illegal without making the internet itself illegal.

SwarmStream?

[Doc Ruby]

Does anyone know how to make a swarm network like Torrent deliver chunks in a more sequential order? Even if locally they're unpredicted, maybe a "superchunk" order, where, say, each megabyte arrives in order, but the kilobytes within it fill in in an arbitrary order. That's what's necessary for Torrents to stream media like video on demand. If there are enough servers in the swarm (say millions), enough will likely be online at any time (say thousands) with the chunks that enough sources can respond (say hundreds) to ensure that every superchunk can be delivered "on cue" (pun intended), delivering a stream with minimum accumulation latency that can be compensated by a few-megabyte (milliseconds) buffer.

Re:SwarmStream?

[shadowmatter]

Try CoolStreaming. Although I haven't seen it in action, some of my friends have (mostly while visiting Europe or Asia), and they say the quality is near-perfect. And this isn't a stupid academic exercise -- it's a real implementation, with up to 10,000 simultaneous users recently. The academic paper, providing the general algorithm, can be found here. Google for more on the implementation.

- shadowmatter

Re:SwarmStream?

[SlayerofGods]

You can do this in bittorent, you'd just need to tell your client to grab the pieces in order (I don't know if there are any that do this... but it would be possible to program it to do so)
Problem is however it will seriously degrade the ability of the network to provide the last pieces if everyone does this since that's when it's most likely someone will D/C.
Using random pieces ensures that the network will have a good amount of all the pieces and not a lopsided amount of the pieces towards the start.

Steaming over a swarm is problematic anyway because of varying connections speeds. You'd need to buffer so much ahead in case you hit someone with a slow DL speed or someone D/Cs in the middle of you downloading that chunk that it would basicly be just like downloading it anyway ;)

Re:SwarmStream?

[Doc Ruby]

Does the BT protocol already have a "mode" or config that already tells server peers to deliver packets in order? Or is the reprogramming you're talking about something that would work only with all reprogrammed torrent agents at opposite ends of the connections?

FWIW, the "superchunk" protocol architecture I described is designed to address exactly that latency problem, which is the entire shortcoming of existing torrent apps.