RSS & BT Together?

AntiPasto writes "According to this Yahoo! News article, RSS and BitTorrent could be set to join in a best-of-both-worlds content management system for the net. Possible?" Update: 03/17 21:39 GMT by T : Thanks to Steve Gillmor, here's the original story on eWeek to replace the now-dead Yahoo! link.

Comments

RSS polling intervals

[tcopeland]

"Now, should an aggregator be polling every 30 minutes? The convention early
on was no more than once an hour. But newer aggregators either never heard of
the convention or chose to ignore it. Some aggregators let the users scan
whenever they want. Please don't do that. Once an hour is enough. Otherwise
bandwidth bills won't scale."

Hm. That's interesting. The RubyForge RSS feeds get polled every
half hour by a couple folks, i.e.:

[tom@rubyforge httpd]$ tail -10000 access_log | grep "16/Dec" | grep export |
grep 66.68 | wc -l
19
[tom@rubyforge httpd]$

Hasn't caused problems yet, but maybe that's because RubyForge only gets about
30K-40K hits per day, and the feeds get just a fraction of that.

Neat idea.

[grub]

This could be carried further into a whole indymedia via BT. It would be even harder for governments and industry to silent dissident voices.

BitTorrent is no-go for small files..

[dk.r*nger]

BitTorrent doesn't scale for very small downloads (less than a few MB, I'd say), due to the tracker.

The tracker keeps, well, uhm, track, of the available pieces of the file, and every client reports in every time has got, or failed to get, a piece. So, using BitTorrent to distribute RSS feeds won't work, because the tracker will take up as much bandwidth, if not more, as a HTTP request, resulting in the "Not changed since your version" request.

Apart from that, well, yes, BitTorrent is great to distribute large files :)

why not nntp for syndication?

[ph00dz]

I always thought that syndicators should take advantage of the current distributed architecture of the newsgroups to syndicate their content... but hey, maybe that's just me. The only real problem is one of authentication -- since you're downloading content from a publicly accessible source one would have to come up with some clever way of making sure you're grabbing content from the author you choose.

IRC

[Bluelive]

Using rss polling seems to me just a way to fake a subscribe push technology. Why not just use a push technology like irc. A channel per tracker, just join a channel to get the updates when they are send. Youd probably still want to use rss for events that youd miss while not online for longer periods.

fidonet

[mabu]

A good analogy would be comparing the setup to Fidonet and their "echo" messageboards. It's a very efficient method to distribute news.

The key to usefulness however, is enabling technology to prioritize and authenticate the RSS feeds in some way.

Re:fidonet

[MS_leases_my_soul]

As a former FidoNet node SysOp, I have had a similar idea for a couple of years. I have messed around with the code but never been happy with it to a point of putting it on SourceForge.

The idea goes like this:

If you want to host a RSS feed, you run a program that is basically a peer cache. People hit your IP and "subscribe" to the feed. You give them a list of other subscribers' IPs and the public key for the feed. The client then hits these peers and checks to see who has faster bandwidth. If the peer is faster than you, you ask to become a leaf under it. It will either accept you as a leaf or pass you on to any leaves it thinks are still faster than you.

When you have an update to your RSS, you sign it with a digital signature to prove the
authenticity of the RSS file. The fastest peers actually poll the RSS publisher. Whenever
they get a new RSS file, they push it to the leaves under them. The RSS file continues to flow downstream until every node has the RSS feed.

Files under a certain size are just automatically grabbed by the top nodes whenever they become aware of them. Leaf nodes ask their parent node for the file, so again, the small files flow down the tree.

For larger files, everyone uses BT pretty much as it exists today.

Using a system like this, you could even go beyond digital signatures and include public key encryption so that you had to have the public key for the feed to even be able to read the messages. The feed owner could choose who would be allowed to have the private key, thus controlling who could post while at the same time keeping the traffic unreadable to any sniffing the wire.

Integrate this into an encrypted peer-to-peer app like WASTE and you might have something worth using. So who wants to start developing code?

Content management system ?

[mybecq]

Can somebody explain how RSS and BitTorrent equal a content management system ?

Sounds more like a (possibly improved) content delivery system.

Too bad the article didn't indicate anything about content management.