RSS & BT Together?

AntiPasto writes "According to this Yahoo! News article, RSS and BitTorrent could be set to join in a best-of-both-worlds content management system for the net. Possible?" Update: 03/17 21:39 GMT by T : Thanks to Steve Gillmor, here's the original story on eWeek to replace the now-dead Yahoo! link.

Comments

RSS polling intervals

[tcopeland]

"Now, should an aggregator be polling every 30 minutes? The convention early
on was no more than once an hour. But newer aggregators either never heard of
the convention or chose to ignore it. Some aggregators let the users scan
whenever they want. Please don't do that. Once an hour is enough. Otherwise
bandwidth bills won't scale."

Hm. That's interesting. The RubyForge RSS feeds get polled every
half hour by a couple folks, i.e.:

[tom@rubyforge httpd]$ tail -10000 access_log | grep "16/Dec" | grep export |
grep 66.68 | wc -l
19
[tom@rubyforge httpd]$

Hasn't caused problems yet, but maybe that's because RubyForge only gets about
30K-40K hits per day, and the feeds get just a fraction of that.

Re:RSS polling intervals

[blowdart]

Consider also that, like Kazza before it, people are now running "hacked" bittorrent clients which throttle upload speeds to a stupidly low level. Even if an RSS driven bittorrent was well behaved, it wouldn't be long before an unfriendly one arrived

Neat idea.

[grub]

This could be carried further into a whole indymedia via BT. It would be even harder for governments and industry to silent dissident voices.

Good concept

[SargeZT]

It is a good concept, by all means. But, the bittorrent development community isn't that impressive. The program is great, but implementing RSS into BitTorrent would require an overhaul of the entire engine. I would love it if this got put into future versions, but I'm not that hopeful.

BitTorrent is no-go for small files..

[dk.r*nger]

BitTorrent doesn't scale for very small downloads (less than a few MB, I'd say), due to the tracker.

The tracker keeps, well, uhm, track, of the available pieces of the file, and every client reports in every time has got, or failed to get, a piece. So, using BitTorrent to distribute RSS feeds won't work, because the tracker will take up as much bandwidth, if not more, as a HTTP request, resulting in the "Not changed since your version" request.

Apart from that, well, yes, BitTorrent is great to distribute large files :)

If he wants to save bandwidth.

[junkymailbox]

stop putting up "graphics, and even multimedia files"! .. or use akamai or some other servers.

but i guess bumming off bittorrent/p2p bandwidth is not a bad idea either.

why not nntp for syndication?

[ph00dz]

I always thought that syndicators should take advantage of the current distributed architecture of the newsgroups to syndicate their content... but hey, maybe that's just me. The only real problem is one of authentication -- since you're downloading content from a publicly accessible source one would have to come up with some clever way of making sure you're grabbing content from the author you choose.

IRC

[Bluelive]

Using rss polling seems to me just a way to fake a subscribe push technology. Why not just use a push technology like irc. A channel per tracker, just join a channel to get the updates when they are send. Youd probably still want to use rss for events that youd miss while not online for longer periods.

fidonet

[mabu]

A good analogy would be comparing the setup to Fidonet and their "echo" messageboards. It's a very efficient method to distribute news.

The key to usefulness however, is enabling technology to prioritize and authenticate the RSS feeds in some way.

Re:fidonet

[MS_leases_my_soul]

As a former FidoNet node SysOp, I have had a similar idea for a couple of years. I have messed around with the code but never been happy with it to a point of putting it on SourceForge.

The idea goes like this:

If you want to host a RSS feed, you run a program that is basically a peer cache. People hit your IP and "subscribe" to the feed. You give them a list of other subscribers' IPs and the public key for the feed. The client then hits these peers and checks to see who has faster bandwidth. If the peer is faster than you, you ask to become a leaf under it. It will either accept you as a leaf or pass you on to any leaves it thinks are still faster than you.

When you have an update to your RSS, you sign it with a digital signature to prove the
authenticity of the RSS file. The fastest peers actually poll the RSS publisher. Whenever
they get a new RSS file, they push it to the leaves under them. The RSS file continues to flow downstream until every node has the RSS feed.

Files under a certain size are just automatically grabbed by the top nodes whenever they become aware of them. Leaf nodes ask their parent node for the file, so again, the small files flow down the tree.

For larger files, everyone uses BT pretty much as it exists today.

Using a system like this, you could even go beyond digital signatures and include public key encryption so that you had to have the public key for the feed to even be able to read the messages. The feed owner could choose who would be allowed to have the private key, thus controlling who could post while at the same time keeping the traffic unreadable to any sniffing the wire.

Integrate this into an encrypted peer-to-peer app like WASTE and you might have something worth using. So who wants to start developing code?

Setting up sharing for Bittorrent

[Anonymous Coward]

Try out the Hunting of the Snark project client.

It has a simple option --share that automatically shares a file or directory through bittorrent by creating the metainfo file on the fly, launching a mini webserver to serve the .torrent file and acts as the bittorent clients that acts as the initial seed.

And it is a nice and simple commandline tool.

Content management system ?

[mybecq]

Can somebody explain how RSS and BitTorrent equal a content management system ?

Sounds more like a (possibly improved) content delivery system.

Too bad the article didn't indicate anything about content management.

Next logical step...

[Bugmaster]

The next logical step would be to augment HTTP itself to piggypack on top of BT (as suggested by multiple people earlier on this site); this will make slashdotting go away for good. I can see three major problems to both the RSS+BT and HTTP+BT integration schemes: leeching, cracking and discovery. If everyone starts to leech, then BT's advantages are nullified. If someone cracks the client, they can corrupt portions of the feed/website that is being served (checksums solve this problem, but AFAIK they rely on the majority of users being honest). Then there's also the chicken-and-egg problem of discovering the .torrent file (or its equivalent) in the first place: someone still has to serve it so that you can jump-start your torrent madness, and that someone can get slashdotted easily.

These problems are not insurmountable, but they are not insignificant, either. Thus, I don't think that RSS+BT is the instant-gratification, no-risk paradise that the Yahoo article makes it out to be.

Re:Stupid article

[ForteTuba]

Ideas can be inspiring to others. Churning ideas often leads to better ideas. Sharing ideas can get your systems built when you don't have time yourself. Not everything good is a built tool: To every thing, code, code, code... There is a season, code, code, code... And a time for every system to develop. A time to think, a time to build, A time to talk, a time to choose A time to share all our thoughts, A time to learn; not to do this is to lose...

A notification server protocol? cf. NTP/NNTP/SIP

[mattr]

Maybe a kind of event notification service would be useful (I get to it after a few comments...)

A) Sounds nice, but even without a torrent, using an open source hash algorithm (client and server agree on how to calculate the hash) would provide a way for the client to only download the hash value itself to check for freshness.

This way,
1 the author knows how many people have consumed the data and their general geographic distribution.
2 the author can make a decision to stop publication, which problematic but at any rate easier to enforce than if he or she starts out authorizing a torrent.
3 the author is free to pay for bandwidth if he or she will happily serve one per user just not a zillion per poller.

B) To be sure, it is easy to see who publishes an RSS feed / incites a Torrent download over somebody's infrastructure, whereas it is not so easy to discover the identity of an anonymous coward. You could also publish a pseudo-RSS feed itself exclusively over the torrent network using sequential filenames for more anonymity maybe..um.

C) Personally I have a current need for frequently updated RSS for a certain application and I'd set up a server that my internal network clients would poll frequently. But I'd still need for one machine to know the instant things change on the web too.

D) I'm wondering if a hierarchical network of servers might be useful here to publish event notifications. UDP is lossy, and we don't want to lose any events so that's out I guess. In NTP, various strata of time servers are used and clients try to sync to Greenwich time (light data) by the best route available. In NNTP, a client usually uses only one news server to get a fat feed, and different server owners often choose to handle only a subset of what's available in the whole world, which might also be the case (try serving every event of importance to someone in the world.. what is the bandwidth needed for that? How many bits to describe it in ip-like dot format?)

Probably there is another service that does what I'd like and it just flew out my left ear, but it just seemed to me that the best thing would be to combine the lightweight NTP network which lets clients synchronize their understanding of time despite general flakiness, and the NNTP network which allows different servers to decide to serve only certain segments of the worldwide aggregated feed.

And SIP does a lot of things that might be useful. And there is MDS (metacomputing directory service for the "semantic grid" - pdf / google's html). And there's Jini ..

Anyway we do want to know some things with at least one minute resolution. (A storm watch? A news headline so we can turn on the TV or video stream?) At TV stations I know people constantly are watching the TV out of the corner of their eye to see if something earth-shattering comes up. How about a chime to tell you to look instead? How else to people get First Post? ;) I'd just like to beat normal notification systems for current events and website updates, for starters, based on a relatively robust and timely mechanism.

Maybe a low bandwidth network with some of these characteristics would be useful to distribute update event notifications that filter down to everyone's devces. A big company could have one or two machines consuming a global event notification thread, add events only it knows about, and serve this information on a push or pull basis to all its employees. Hmmm, tasty. Come to think of it I want something like that for another project too, Does anything already do this?

One interesting paper (2001) I found is on an emergency notification network based on subscribe/notify messages over SIP, a widespread voice over ip prot

Why not use pgp-style digital signatures?

[Anonymous Coward]

The main downside comment I have seen on this thread is the issue of trust: either content suppliers don't trust the network (e.g. the NYT comment,) or readers don't trust the network (CIA, Evil Bloggers, whatever) to not send them a bogus feed.

(Note I don't know details of how BT works, just general idea - fell free to take this idea and run with it however it makes more sense.)

I like the notion of this happening at the web-server level, which allows it to be generalized to other forms of content distribution than just RSS. When a client first connects to a server, it downloads (and caches) that server's public-key. When the server gets a request from a client whose HTTP header says it is BT-enabled, it can return a redirect to the torrent (presumably servers would only do this when it is a net win for them - e.g. for large static files,) which would be somehow wrapped in a digitally signed envelope.

The interting thing is, after the first connect, the client can get 'official' content from an aggregator/distribted proxy, and still be assured t is authentic. (At least, as sure as if they had gone directly to the main site - obviously, their DNS could have been hijacked, etc. You could either choose to live with the status quo and accept this level of 'security,' or use key-signing authority to verify the public key belongs to the claimed owner, as we do now with SSL certs.)

Re:modtorrent

[elFarto the 2nd]

Hi,

They've already started a project for this at http://mod-torrent.sourceforge.net/

Regards
elFarto