Slashdot Mirror
RSS & BT Together?
AntiPasto writes "According to this Yahoo! News article, RSS and BitTorrent could be set to join in a best-of-both-worlds content management system for the net. Possible?" Update: 03/17 21:39 GMT by T : Thanks to Steve Gillmor, here's the original story on eWeek to replace the now-dead Yahoo! link.
Also, if you make your feeds static files, rather than dynamic, a modern server is going to have no problems serving it hundreds (or thousands) of times a minute if necessary.
The article's idea is simply to make the web (at least the rss) distributed and then query the distributed server to change from 30 minutes refresh to a faster refresh. But the distributed server needs to be updated also. It may simply be cheaper / more efficient to simply run more servers.
...practical ways. It's a nice program, I've used it on occasssion but it does have its share of bugs.
And setting up a server isn't quite easy.
It really could be a lot better with some work.
clifgriffin > blog
Of course it hasn't caused any problems. It's a couple folks every half hour. Try a few thousand folks every minute (imagine it's a metaserver for some online game, or a blog during a major news event).
Still, I'm not seeing anything beyond the "duh" factor here. All that needs to happen is for browsers to handle torrent links. Not some souped up napster app, a browser, so that I can type in a torrent link and get any web page (or other mime doc) for the browser to handle. Change the RSS to use the new URL scheme, and there you go. You could also do it as a proxy, but you run into worse cache coherency issues than with direct support of the protocol; who's to say who has the correct mapping of the content url to the torrent url?
Good luck, mind you, on getting anything but blogs, download sites, and perhaps hobby news sites to jump on board. This issue has been beaten to death in the IETF and many other circles, and it all boils down to content control -- the NY Times simply doesn't want its content mirrored like that.
I've finally had it: until slashdot gets article moderation, I am not coming back.
Sorry, guys, but you are basically reinventing USENET over TCP/IP.
Bandwidth bills on a static page are also trivial.
A well behaved program won't go GETs on every RSS page, but will do HEADS, compare them to what it already has, and decide from there to get or not get the new page.
A HEAD request is very small, and unless you're doing millions of them, this shouldn't be an issue.
- Serge
I blogged about the possibilities of using BitTorrent to deliver web content back in April, but I didn't consider RSS. The idea worked out between myself and some friends was a network of transparent proxies as a way of dealing with Slashdot-style "flash crowds". When you request content, your proxy requests the content from you, and simultaneously broadcasts the request to nearby machines. If any of those machines have already downloaded the content (some form of timestamp and hash is necessary to ensure it's the correct and authentic version of that URL) then they will send that content to you, allowing servers already under or expecting heavy load to push out a new HTTP status message "use torrent", supplying a (much smaller) torrent file. This allows web servers to scale much better under flash crowd conditions.
The drawback of the WebTorrent idea is that you need some way to group all the images, text and stylesheets together, otherwise you have to make a n inefficient P2P request for each one. RSS is a great way of doing that.
There aren't many details online at the moment of the work we did on the WebTorrent idea; it was mainly an e-mail thread -- get in touch if you'd like details. The project page is available, but I stopped updating it so it doesn't have all the work that was eventually done.
> A well behaved program won't go GETs on every RSS page, but will do HEADS,
> compare them to what it already has, and decide from there
> to get or not get the new page.
An even more behaved program will issue a GET with the "If-Modified-Since: " header, which will mean the server will return a "304 - not modified" if the file hasn't changed, or the actual file if it has.. Thus doing in one operation what a combined HEAD and followup GET would take 2 to do.
Sig out of date
What I would like to see is modtorrent for apache. Where you could specify that files larger than 20MB would get sent as a .torrent instead. And it wouldn't require you to make a .torrent manually instead it would create it when a file was requested. And put it in a directory so it was ready to serve it the next time someone wanted it. Would work great if you want to have large files such as movies and demoes on your site.
I fought the corporate America, and the corporate America bought the law.
A better solution would be eliminating the need for syndicators to constnantly poll waiting for RSS updates by using IP multicasting to notify syndicators of when the content of a particular RSS feed has changed. Multicast protocols which provide such announcements already exist, such as the Session Announcement Protocol which would notify those curious of updated RSS feeds. A URL to the updated feed would be provided, and afterwards whatever file transfer protocol you wish could be used to fetch the RSS feed itself, even BitTorrent.