Slashdot Mirror
RSS & BT Together?
AntiPasto writes "According to this Yahoo! News article, RSS and BitTorrent could be set to join in a best-of-both-worlds content management system for the net. Possible?" Update: 03/17 21:39 GMT by T : Thanks to Steve Gillmor, here's the original story on eWeek to replace the now-dead Yahoo! link.
Hm. That's interesting. The RubyForge RSS feeds get polled every
half hour by a couple folks, i.e.:Hasn't caused problems yet, but maybe that's because RubyForge only gets about
30K-40K hits per day, and the feeds get just a fraction of that.
The Army reading list
This could be carried further into a whole indymedia via BT. It would be even harder for governments and industry to silent dissident voices.
Trolling is a art,
It is a good concept, by all means. But, the bittorrent development community isn't that impressive. The program is great, but implementing RSS into BitTorrent would require an overhaul of the entire engine. I would love it if this got put into future versions, but I'm not that hopeful.
And why did you staple the trout to the RAM?
The article's idea is simply to make the web (at least the rss) distributed and then query the distributed server to change from 30 minutes refresh to a faster refresh. But the distributed server needs to be updated also. It may simply be cheaper / more efficient to simply run more servers.
...practical ways. It's a nice program, I've used it on occasssion but it does have its share of bugs.
And setting up a server isn't quite easy.
It really could be a lot better with some work.
clifgriffin > blog
I'll believe it when I see it. This idea has been circulating the last few days through the blog world, the same people who think they're going to crush traditional media with the sheer power of their blogs. I say whatever.
"Windows Me offers tremendous reliability and stability improvements..." -- Paul Thurott
mmm, I love a fruit salad!
BitTorrent doesn't scale for very small downloads (less than a few MB, I'd say), due to the tracker.
:)
The tracker keeps, well, uhm, track, of the available pieces of the file, and every client reports in every time has got, or failed to get, a piece. So, using BitTorrent to distribute RSS feeds won't work, because the tracker will take up as much bandwidth, if not more, as a HTTP request, resulting in the "Not changed since your version" request.
Apart from that, well, yes, BitTorrent is great to distribute large files
but i guess bumming off bittorrent/p2p bandwidth is not a bad idea either.
Konspire2b looks like a better option than BitTorrent for distributing news. You could have a channel mapping to an RSS feed and just wait for the news to come to you. No polling intervals and low bandwidth requirements for the operator. With BitTorrent you still have to poll for updates and this removes that requirement.
I always thought that syndicators should take advantage of the current distributed architecture of the newsgroups to syndicate their content... but hey, maybe that's just me. The only real problem is one of authentication -- since you're downloading content from a publicly accessible source one would have to come up with some clever way of making sure you're grabbing content from the author you choose.
Using rss polling seems to me just a way to fake a subscribe push technology. Why not just use a push technology like irc. A channel per tracker, just join a channel to get the updates when they are send. Youd probably still want to use rss for events that youd miss while not online for longer periods.
A good analogy would be comparing the setup to Fidonet and their "echo" messageboards. It's a very efficient method to distribute news.
The key to usefulness however, is enabling technology to prioritize and authenticate the RSS feeds in some way.
What morron modded parent as insightful?
Does your usenet reader serve news articles to other users?
No, you need a costly usenet servers architecture. Not only machines, but also huuuge bandwith. Today's usenet servers that want to serve large portion of world hierarchies can only get it via dedicated satellite usenet-only feeds.
RSS+BT on the other hand is poor server and rich clients that exchange articles between themselves via p2p network only supervised by a BT tracker.
Robert
Bastard Operator From 193.219.28.162
Can somebody explain how RSS and BitTorrent equal a content management system ?
Sounds more like a (possibly improved) content delivery system.
Too bad the article didn't indicate anything about content management.
I blogged about the possibilities of using BitTorrent to deliver web content back in April, but I didn't consider RSS. The idea worked out between myself and some friends was a network of transparent proxies as a way of dealing with Slashdot-style "flash crowds". When you request content, your proxy requests the content from you, and simultaneously broadcasts the request to nearby machines. If any of those machines have already downloaded the content (some form of timestamp and hash is necessary to ensure it's the correct and authentic version of that URL) then they will send that content to you, allowing servers already under or expecting heavy load to push out a new HTTP status message "use torrent", supplying a (much smaller) torrent file. This allows web servers to scale much better under flash crowd conditions.
The drawback of the WebTorrent idea is that you need some way to group all the images, text and stylesheets together, otherwise you have to make a n inefficient P2P request for each one. RSS is a great way of doing that.
There aren't many details online at the moment of the work we did on the WebTorrent idea; it was mainly an e-mail thread -- get in touch if you'd like details. The project page is available, but I stopped updating it so it doesn't have all the work that was eventually done.
These problems are not insurmountable, but they are not insignificant, either. Thus, I don't think that RSS+BT is the instant-gratification, no-risk paradise that the Yahoo article makes it out to be.
>|<*:=
This is the first time I've heard FidoNet mentioned in... must be almost a decade. It's like the huge amateur network (which for a brief period outnumbered the Internet in raw node count, mind you) never existed.
:-/
:-)
Anyway, FidoNet was not without its share of problems. The killing bullet, I'd say today, was the social factor - there were too conservative forces clinging to backwards compatibility at the cost of anything. Anything had to work with the most basic piece of software; this effectively shot progress and evolution dead.
Not that there weren't attempts. There were. They just weren't successful.
Anyway, setting up echoes would have the same problems as FidoNet echoes. The number one problem was typical for Slashdot: DUPES!
Echoes were set up so that one node relayed a message in an echomail forum to its other connected nodes for a particular echo, effectively creating a star topology, different for each forum. However, since each sysop just wanted the echo linked, he would just hook up to somewhere, and forget about it. Then, others would hook up from him, and all of a sudden somebody had hooked up to two different valid uplinks.
The result? The star topology all of a sudden had a loop in it. Messages would keep circling (since FidoNet used dedicated dialup lines, latency between nodes was typically in the hours range) and dupe filters were created.
All of those filters and filter-enabling tags were optional, of course. After all, you couldn't mandate an operational node to change its behavior, you could just ask nicely.
Political play to no ends.
Anyway, there were many other funny effects with EchoMail. Crosslinking was another - when one echo got linked to another at a node, so that all messages in echo X would enter echo Y at that node and vice versa. The most exotic of these was when a religious echo got crosslinked with a fantasy humor one -- through crosslinked physical directories at a node (the FAT pointers for the different directories hosting the two echoes pointed to the same location on the disk). Anyway, much hilarious discussion ensued, and not many understood much what people were trying to say in the crosslinked echo.
/ former sysop and NEC in FidoNet
Ideas can be inspiring to others. Churning ideas often leads to better ideas. Sharing ideas can get your systems built when you don't have time yourself. Not everything good is a built tool: To every thing, code, code, code... There is a season, code, code, code... And a time for every system to develop. A time to think, a time to build, A time to talk, a time to choose A time to share all our thoughts, A time to learn; not to do this is to lose...
A) Sounds nice, but even without a torrent, using an open source hash algorithm (client and server agree on how to calculate the hash) would provide a way for the client to only download the hash value itself to check for freshness.
This way,
1 the author knows how many people have consumed the data and their general geographic distribution.
2 the author can make a decision to stop publication, which problematic but at any rate easier to enforce than if he or she starts out authorizing a torrent.
3 the author is free to pay for bandwidth if he or she will happily serve one per user just not a zillion per poller.
B) To be sure, it is easy to see who publishes an RSS feed / incites a Torrent download over somebody's infrastructure, whereas it is not so easy to discover the identity of an anonymous coward. You could also publish a pseudo-RSS feed itself exclusively over the torrent network using sequential filenames for more anonymity maybe..um.
C) Personally I have a current need for frequently updated RSS for a certain application and I'd set up a server that my internal network clients would poll frequently. But I'd still need for one machine to know the instant things change on the web too.
D) I'm wondering if a hierarchical network of servers might be useful here to publish event notifications. UDP is lossy, and we don't want to lose any events so that's out I guess. In NTP, various strata of time servers are used and clients try to sync to Greenwich time (light data) by the best route available. In NNTP, a client usually uses only one news server to get a fat feed, and different server owners often choose to handle only a subset of what's available in the whole world, which might also be the case (try serving every event of importance to someone in the world.. what is the bandwidth needed for that? How many bits to describe it in ip-like dot format?)
Probably there is another service that does what I'd like and it just flew out my left ear, but it just seemed to me that the best thing would be to combine the lightweight NTP network which lets clients synchronize their understanding of time despite general flakiness, and the NNTP network which allows different servers to decide to serve only certain segments of the worldwide aggregated feed.
And SIP does a lot of things that might be useful. And there is MDS (metacomputing directory service for the "semantic grid" - pdf / google's html). And there's Jini ..
Anyway we do want to know some things with at least one minute resolution. (A storm watch? A news headline so we can turn on the TV or video stream?) At TV stations I know people constantly are watching the TV out of the corner of their eye to see if something earth-shattering comes up. How about a chime to tell you to look instead? How else to people get First Post? ;) I'd just like to beat normal notification systems for current events and website updates, for starters, based on a relatively robust and timely mechanism.
Maybe a low bandwidth network with some of these characteristics would be useful to distribute update event notifications that filter down to everyone's devces. A big company could have one or two machines consuming a global event notification thread, add events only it knows about, and serve this information on a push or pull basis to all its employees. Hmmm, tasty. Come to think of it I want something like that for another project too, Does anything already do this?
One interesting paper (2001) I found is on an emergency notification network based on subscribe/notify messages over SIP, a widespread voice over ip prot
The main downside comment I have seen on this thread is the issue of trust: either content suppliers don't trust the network (e.g. the NYT comment,) or readers don't trust the network (CIA, Evil Bloggers, whatever) to not send them a bogus feed.
(Note I don't know details of how BT works, just general idea - fell free to take this idea and run with it however it makes more sense.)
I like the notion of this happening at the web-server level, which allows it to be generalized to other forms of content distribution than just RSS. When a client first connects to a server, it downloads (and caches) that server's public-key. When the server gets a request from a client whose HTTP header says it is BT-enabled, it can return a redirect to the torrent (presumably servers would only do this when it is a net win for them - e.g. for large static files,) which would be somehow wrapped in a digitally signed envelope.
The interting thing is, after the first connect, the client can get 'official' content from an aggregator/distribted proxy, and still be assured t is authentic. (At least, as sure as if they had gone directly to the main site - obviously, their DNS could have been hijacked, etc. You could either choose to live with the status quo and accept this level of 'security,' or use key-signing authority to verify the public key belongs to the claimed owner, as we do now with SSL certs.)
What I would like to see is modtorrent for apache. Where you could specify that files larger than 20MB would get sent as a .torrent instead. And it wouldn't require you to make a .torrent manually instead it would create it when a file was requested. And put it in a directory so it was ready to serve it the next time someone wanted it. Would work great if you want to have large files such as movies and demoes on your site.
I fought the corporate America, and the corporate America bought the law.
Does your usenet reader serve news articles to other users?
Yes: the way people traditionally read USENET news is by becoming a USENET node, downloading articles to the directory hierarchy of the local machine, and then redistributing them to neighboring sites. Reading news by connecting to centralized news servers via a network client happened many years later.
No, you need a costly usenet servers architecture.
There is nothing intrinsically "costly" about it: it's something a PDP-11 used to handle and that regularly ran over dial-up.
Not only machines, but also huuuge bandwith. Today's usenet servers that want to serve large portion of world hierarchies can only get it via dedicated satellite usenet-only feeds.
Just like a BT solution, you only redistribute those articles that you yourself are interested in.
The reason why we got a USENET infrastructure with a small number of backbone sites (compared to the readership) that carried everything is simply because a bunch of sites took on that role and carry everything. There is nothing in the protocol or design of USENET that requires it.
RSS+BT on the other hand is poor server and rich clients that exchange articles between themselves via p2p network only supervised by a BT tracker.
And you believe that BT and the BT tracker scales up to many billions of files on millions of nodes by sheer magic? BT would probably need a lot of work to scale up. And at least USENET doesn't need any supervision by anything--it's completely asynchronous and unsupervised.
Note that I did not claim that USENET would work any better than RSS+BT--I have no idea whether it would--simply that people are basically reinventing USENET when they combine RSS and BT.
I actually suspect that there are intrinsic properties of large peer-to-peer news networks that people don't like because that's why USENET became more and more centralized over the years.
What morron modded parent as insightful?
That's what I would ask about your posting. In fact, I would ask what moron wrote it.
A better solution would be eliminating the need for syndicators to constnantly poll waiting for RSS updates by using IP multicasting to notify syndicators of when the content of a particular RSS feed has changed. Multicast protocols which provide such announcements already exist, such as the Session Announcement Protocol which would notify those curious of updated RSS feeds. A URL to the updated feed would be provided, and afterwards whatever file transfer protocol you wish could be used to fetch the RSS feed itself, even BitTorrent.