Slashdot Mirror
RSS & BT Together?
AntiPasto writes "According to this Yahoo! News article, RSS and BitTorrent could be set to join in a best-of-both-worlds content management system for the net. Possible?" Update: 03/17 21:39 GMT by T : Thanks to Steve Gillmor, here's the original story on eWeek to replace the now-dead Yahoo! link.
Hm. That's interesting. The RubyForge RSS feeds get polled every
half hour by a couple folks, i.e.:Hasn't caused problems yet, but maybe that's because RubyForge only gets about
30K-40K hits per day, and the feeds get just a fraction of that.
The Army reading list
Absolute genius. An RSS feed of torrents. I would set one up right now if I had content to share.
In opposite to *.torrent, RSS is human readable. People will stop using BT.
This could be carried further into a whole indymedia via BT. It would be even harder for governments and industry to silent dissident voices.
Trolling is a art,
It is a good concept, by all means. But, the bittorrent development community isn't that impressive. The program is great, but implementing RSS into BitTorrent would require an overhaul of the entire engine. I would love it if this got put into future versions, but I'm not that hopeful.
And why did you staple the trout to the RAM?
The article's idea is simply to make the web (at least the rss) distributed and then query the distributed server to change from 30 minutes refresh to a faster refresh. But the distributed server needs to be updated also. It may simply be cheaper / more efficient to simply run more servers.
I have planty of ideas every day. I want to see an implementation.
- "Oh look how cool I am writing a 4-page article to say the obvious"
...practical ways. It's a nice program, I've used it on occasssion but it does have its share of bugs.
And setting up a server isn't quite easy.
It really could be a lot better with some work.
clifgriffin > blog
it feels like mixing apples and bananas.
Let's see what comes from that.
:: Andrea
Anime Wallpapers
> WTF is RobyForge???
It's a free hosting site for open source Ruby projects.
The Army reading list
I'll believe it when I see it. This idea has been circulating the last few days through the blog world, the same people who think they're going to crush traditional media with the sheer power of their blogs. I say whatever.
"Windows Me offers tremendous reliability and stability improvements..." -- Paul Thurott
BitTorrent doesn't scale for very small downloads (less than a few MB, I'd say), due to the tracker.
:)
The tracker keeps, well, uhm, track, of the available pieces of the file, and every client reports in every time has got, or failed to get, a piece. So, using BitTorrent to distribute RSS feeds won't work, because the tracker will take up as much bandwidth, if not more, as a HTTP request, resulting in the "Not changed since your version" request.
Apart from that, well, yes, BitTorrent is great to distribute large files
I'd rather see real metadata assocated with bitstreams, rather than, pardon me, RSS fluff. RSS is neat and all, but is basically just a way to syndicate a bunch of yammering and blathering. Real metadata would be more useful.
but i guess bumming off bittorrent/p2p bandwidth is not a bad idea either.
I think it has been done before.
Konspire2b looks like a better option than BitTorrent for distributing news. You could have a channel mapping to an RSS feed and just wait for the news to come to you. No polling intervals and low bandwidth requirements for the operator. With BitTorrent you still have to poll for updates and this removes that requirement.
Sorry, guys, but you are basically reinventing USENET over TCP/IP.
I always thought that syndicators should take advantage of the current distributed architecture of the newsgroups to syndicate their content... but hey, maybe that's just me. The only real problem is one of authentication -- since you're downloading content from a publicly accessible source one would have to come up with some clever way of making sure you're grabbing content from the author you choose.
Using rss polling seems to me just a way to fake a subscribe push technology. Why not just use a push technology like irc. A channel per tracker, just join a channel to get the updates when they are send. Youd probably still want to use rss for events that youd miss while not online for longer periods.
Lern not to dupe your own posts.
A good analogy would be comparing the setup to Fidonet and their "echo" messageboards. It's a very efficient method to distribute news.
The key to usefulness however, is enabling technology to prioritize and authenticate the RSS feeds in some way.
I just heard some sad news on talk radio - famous marine mammal Keiko was found dead in his fjord this morning. There weren't any more details. I'm sure everyone in the Slashdot community will miss him - even if you didn't enjoy his work, there's no denying his contributions to popular culture. Truly an American icon.
Oh - and the Norwegian "secret cermony" involved tartar sauce.
It has a simple option --share that automatically shares a file or directory through bittorrent by creating the metainfo file on the fly, launching a mini webserver to serve the .torrent file and acts as the bittorent clients that acts as the initial seed.
And it is a nice and simple commandline tool.
Can somebody explain how RSS and BitTorrent equal a content management system ?
Sounds more like a (possibly improved) content delivery system.
Too bad the article didn't indicate anything about content management.
When I read "BitTorrent," I thought "Bitkeeper." Then for RSS, I came up with "Rational Software Solutions." I had this vision of them combining, Voltron-like, to crush the CVS rho-beast dead.
Like I said, weird.
I don't see the big whoop here.
I blogged about the possibilities of using BitTorrent to deliver web content back in April, but I didn't consider RSS. The idea worked out between myself and some friends was a network of transparent proxies as a way of dealing with Slashdot-style "flash crowds". When you request content, your proxy requests the content from you, and simultaneously broadcasts the request to nearby machines. If any of those machines have already downloaded the content (some form of timestamp and hash is necessary to ensure it's the correct and authentic version of that URL) then they will send that content to you, allowing servers already under or expecting heavy load to push out a new HTTP status message "use torrent", supplying a (much smaller) torrent file. This allows web servers to scale much better under flash crowd conditions.
The drawback of the WebTorrent idea is that you need some way to group all the images, text and stylesheets together, otherwise you have to make a n inefficient P2P request for each one. RSS is a great way of doing that.
There aren't many details online at the moment of the work we did on the WebTorrent idea; it was mainly an e-mail thread -- get in touch if you'd like details. The project page is available, but I stopped updating it so it doesn't have all the work that was eventually done.
Newsgroups are opt-out. RSS is opt-in. When you subscribe to a newsgroup, you get all posts from everyone in that group, and have to killfile anyone you don't like. With RSS, you individually subscribe to each poster.
And there is no division by subject. If you subscrbe to Joe, you see everything Joe posts, on any subject.
These problems are not insurmountable, but they are not insignificant, either. Thus, I don't think that RSS+BT is the instant-gratification, no-risk paradise that the Yahoo article makes it out to be.
>|<*:=
This is the first time I've heard FidoNet mentioned in... must be almost a decade. It's like the huge amateur network (which for a brief period outnumbered the Internet in raw node count, mind you) never existed.
:-/
:-)
Anyway, FidoNet was not without its share of problems. The killing bullet, I'd say today, was the social factor - there were too conservative forces clinging to backwards compatibility at the cost of anything. Anything had to work with the most basic piece of software; this effectively shot progress and evolution dead.
Not that there weren't attempts. There were. They just weren't successful.
Anyway, setting up echoes would have the same problems as FidoNet echoes. The number one problem was typical for Slashdot: DUPES!
Echoes were set up so that one node relayed a message in an echomail forum to its other connected nodes for a particular echo, effectively creating a star topology, different for each forum. However, since each sysop just wanted the echo linked, he would just hook up to somewhere, and forget about it. Then, others would hook up from him, and all of a sudden somebody had hooked up to two different valid uplinks.
The result? The star topology all of a sudden had a loop in it. Messages would keep circling (since FidoNet used dedicated dialup lines, latency between nodes was typically in the hours range) and dupe filters were created.
All of those filters and filter-enabling tags were optional, of course. After all, you couldn't mandate an operational node to change its behavior, you could just ask nicely.
Political play to no ends.
Anyway, there were many other funny effects with EchoMail. Crosslinking was another - when one echo got linked to another at a node, so that all messages in echo X would enter echo Y at that node and vice versa. The most exotic of these was when a religious echo got crosslinked with a fantasy humor one -- through crosslinked physical directories at a node (the FAT pointers for the different directories hosting the two echoes pointed to the same location on the disk). Anyway, much hilarious discussion ensued, and not many understood much what people were trying to say in the crosslinked echo.
/ former sysop and NEC in FidoNet
A) Sounds nice, but even without a torrent, using an open source hash algorithm (client and server agree on how to calculate the hash) would provide a way for the client to only download the hash value itself to check for freshness.
This way,
1 the author knows how many people have consumed the data and their general geographic distribution.
2 the author can make a decision to stop publication, which problematic but at any rate easier to enforce than if he or she starts out authorizing a torrent.
3 the author is free to pay for bandwidth if he or she will happily serve one per user just not a zillion per poller.
B) To be sure, it is easy to see who publishes an RSS feed / incites a Torrent download over somebody's infrastructure, whereas it is not so easy to discover the identity of an anonymous coward. You could also publish a pseudo-RSS feed itself exclusively over the torrent network using sequential filenames for more anonymity maybe..um.
C) Personally I have a current need for frequently updated RSS for a certain application and I'd set up a server that my internal network clients would poll frequently. But I'd still need for one machine to know the instant things change on the web too.
D) I'm wondering if a hierarchical network of servers might be useful here to publish event notifications. UDP is lossy, and we don't want to lose any events so that's out I guess. In NTP, various strata of time servers are used and clients try to sync to Greenwich time (light data) by the best route available. In NNTP, a client usually uses only one news server to get a fat feed, and different server owners often choose to handle only a subset of what's available in the whole world, which might also be the case (try serving every event of importance to someone in the world.. what is the bandwidth needed for that? How many bits to describe it in ip-like dot format?)
Probably there is another service that does what I'd like and it just flew out my left ear, but it just seemed to me that the best thing would be to combine the lightweight NTP network which lets clients synchronize their understanding of time despite general flakiness, and the NNTP network which allows different servers to decide to serve only certain segments of the worldwide aggregated feed.
And SIP does a lot of things that might be useful. And there is MDS (metacomputing directory service for the "semantic grid" - pdf / google's html). And there's Jini ..
Anyway we do want to know some things with at least one minute resolution. (A storm watch? A news headline so we can turn on the TV or video stream?) At TV stations I know people constantly are watching the TV out of the corner of their eye to see if something earth-shattering comes up. How about a chime to tell you to look instead? How else to people get First Post? ;) I'd just like to beat normal notification systems for current events and website updates, for starters, based on a relatively robust and timely mechanism.
Maybe a low bandwidth network with some of these characteristics would be useful to distribute update event notifications that filter down to everyone's devces. A big company could have one or two machines consuming a global event notification thread, add events only it knows about, and serve this information on a push or pull basis to all its employees. Hmmm, tasty. Come to think of it I want something like that for another project too, Does anything already do this?
One interesting paper (2001) I found is on an emergency notification network based on subscribe/notify messages over SIP, a widespread voice over ip prot
The main downside comment I have seen on this thread is the issue of trust: either content suppliers don't trust the network (e.g. the NYT comment,) or readers don't trust the network (CIA, Evil Bloggers, whatever) to not send them a bogus feed.
(Note I don't know details of how BT works, just general idea - fell free to take this idea and run with it however it makes more sense.)
I like the notion of this happening at the web-server level, which allows it to be generalized to other forms of content distribution than just RSS. When a client first connects to a server, it downloads (and caches) that server's public-key. When the server gets a request from a client whose HTTP header says it is BT-enabled, it can return a redirect to the torrent (presumably servers would only do this when it is a net win for them - e.g. for large static files,) which would be somehow wrapped in a digitally signed envelope.
The interting thing is, after the first connect, the client can get 'official' content from an aggregator/distribted proxy, and still be assured t is authentic. (At least, as sure as if they had gone directly to the main site - obviously, their DNS could have been hijacked, etc. You could either choose to live with the status quo and accept this level of 'security,' or use key-signing authority to verify the public key belongs to the claimed owner, as we do now with SSL certs.)
Checksums are no biggie, in principle...put the checksum in the hyperlink, and you're guaranteed to get the file linked to. Not sure if BT currently works this way, but whatever they do, it's secure enough for Linux distributions.
Bandwidth on the torrent file server has been a problem for hugely popular files, but the torrent file isn't very big. Seems to me this'll all work best for multimedia enclosures, not the RSS feed itself.
See this post on Infolets for a potentially better idea.
RSS feed = newsgroup
Aggregator = news reader
Bittorrent = RAR+PAR binaries
And best of all.. no polling! Well, between usenet servers it's mostly a broadcast kind of affair these days..
Has anyone made a rss2nntp bot yet?
Of course, IRC is also a remarkably cool medium for timely distribution of small ASCII messages.. The nick/channel bullshit sucks (though usenet "channel"/group takeovers/spams suck even more), but surely it's not beyond the realm of possibilities to build an IRC server that requires people to log in, with a central (distributed) database of credentials, and no nickserv/chanserv services/bots that just stop working every once in a while..
And hey, what about jabber? And javagroups? And multicast?
That's one big heap of cool technology with killer apps (*cough* pointcast?) waiting to be de-vaporized..
SCO employee? Check out the bounty
What I would like to see is modtorrent for apache. Where you could specify that files larger than 20MB would get sent as a .torrent instead. And it wouldn't require you to make a .torrent manually instead it would create it when a file was requested. And put it in a directory so it was ready to serve it the next time someone wanted it. Would work great if you want to have large files such as movies and demoes on your site.
I fought the corporate America, and the corporate America bought the law.
If the newsbot creates a dynamic RSS feed anyway, just punish those people who have their spiders hitting the site too often without respecting the 304's.
A simple RSS feed to the tune of, "Go bugger yourself, and either don't hit my site so often or use an RSS spider that respects 304s," would probably work wonders.
b.c
but I had the impression that my fido-setup on os/2 handled hundreds of thousands of messages a lot faster than my current box does with email, and this while the processor-speed incresed by a factor of 10 and the memory by a factor of 50... aaah, I really liked fido-technology :)
a better match might be using jabber and the p Publish-Subscribe extention (http://www.jabber.org/jeps/jep-0060.html) to allow users to subscribe to a resource and then allow the resource to announce when it has updated, creating a push notification. You could receieve the notice using a standard jabber client, but eventually someone could make an aggregator with an integrated jabber client that would handle your news subscription. Then when you start it up, you poll, and then as long as it is open you don't need to poll until you recieve a publish notification. At that point the client could automatically poll the source for you and you would always have the latest stuff. (of course if 1000's of people all poll at the same moment, you may have some good lag :) )
do you have a link to back up your claims?
... British Telecom was going to merge.
The "other" server would be a network of Squid Caches. I'm already part of one, and it's simple to setup. It doesn't help much with big files, but the small ones (which most web pages are) works just fine. Setting up a network of community NNTP servers wouldn't be hard either.
I am so sick of hearing about tin foil hats from clueless slashdotters. First of all, that article is not representative of indymedia as a whole, let alone independent media as a whole. Second, do you have any evidence that this is false, or are you just incapable of sceptical analysis? As far as I can tell, the evidence supports this scenario.
In closing, I'd like to say: never underestimate the value of a Free Press (an endangered species, at best); and, slashdot is still a fun site to read in spite of sheep like you.
A better solution would be eliminating the need for syndicators to constnantly poll waiting for RSS updates by using IP multicasting to notify syndicators of when the content of a particular RSS feed has changed. Multicast protocols which provide such announcements already exist, such as the Session Announcement Protocol which would notify those curious of updated RSS feeds. A URL to the updated feed would be provided, and afterwards whatever file transfer protocol you wish could be used to fetch the RSS feed itself, even BitTorrent.
The article proposes using BT to transport RSS files, with the goal of reducing the load on extremely popular RSS files. It's a fine idea, except that RSS files are generally very small, and BT incurs such overhead that it's a poor choice for distributing small files.
.torrent files, or .torrent URLs. Configure your news aggregator to pass all new .torrents to the BT client, and you've got yourself a media aggregator that uses open standards, and is easy on the content provider's servers.
.torrent of the latest Alias episode. Or whatever.
The real potential of combining BT & RSS is in the reverse: use RSS to distribute
That combination allows indy media producers (audioblogs, videoblogs, etc) to reach a mass audience in a scaleable way, without overwhelming bandwidth costs. And, it allows media consumers to subscribe to feeds from trusted sources -- be they official of unofficial -- instead of hunting for the
Here's my more fleshed-out response to Gillmor's article, on my weblog.
Or why not use jabber? There are rss bots, and jabber is xml, well defined and in active development. IRC is none of those.
RSS and Jabber are a natural fit. It seems like the idea of transmitting XML through an IM framework and letting the publishers server take the hit for Images/Multimedia files would make sense on the price/performance stance.
But if you really wanted to distribute everything, you could offload the image/multimedia to some sort of P2P scheme, and as the resources are published, seed the P2P network. RSS feeds would just have to send a jabber update with the P2P resource location, then the client would recieve the update and start streaming the content from the seed servers. Mirror sites could cache everything for old style web backward compatibility. Viola, instant (or close to) distributed web publishing.
---------- Hot Rats!
The Bit Torrent protocol is designed to distribute large files, not many many small files. Torrents are partitioned into usually 128k - 1mb "pieces". So unless you embed media files into the RSS stream, which would make a case for using BT, it would be ridiculous to distribute small text streams that's not even as big as the usual lowest common denominator of the piece length.
And I think it's stupid to embed large media files into RSS streams. What's the point? Media files are more static than text content, which is what RSS is intended for. If you want to distribute big media files, just make torrents out of them and use BT as is today.
And FYI, I'm totally for using BT as a media distribution transport. I'm working on a DNS based "hub" which dynamically routes clients to trackers, in an attempt to solve the dead tracker and split swarm problems that generally plague the current BT file distribution "scene".
VIVA1023.com | Political Fashion.
Yeah, it was lightweight as hell. I had a 286 running the FidoNet echomail hub for a large city, and it took some 20 minutes CPU time daily to do all the processing needed.
Distributing RSS feeds on a P2P network may be a good idea. But BitTorrent is not the way to go.
Distribution of content to clients machines is what was called "push" by software companies (Microsoft and start ups) around 1997. Finally they abandonned the idea, as few content was distributed and no standard emerged. RSS feeds is just that, and is succeeding thanks to blogs. In 1997, P2P was a therorical concept and has not been implemented for so-called "push".
The main problem with RSS feeds is not the distribution of the RSS page, but the notification of changes. Today, smart RSS readers (may be they are to few) use HEAD HTTP requests instead of GET requests to avoid loading the whole RSS feed every hour (assuming HEAD requests are correctly handled by the server to return the last modification date).
However the ultimate solution would be to replace pulling by pushing. Notifying RSS feed changes over IRC or Jabber seems to be a better idea, while keeping the RSS content distibution with HTTP.
For content distribution, a P2P network may be useful to distribute the load after the notification of a change. But the protocol should be much lighter than BT which was designed for big files.
Fidonet was great. I was one of the original systems in the network. One thing that killed Fidonet were that a few less-than-honorable people managed to take control of some of the primary hubs in the network and exert biased influence over the network - jacking people on fees and controlling which content ended up being distributed. Fidonet ended up becoming political in nature and there was a minor rebellion, at a time when usenet was gaining attention. A few overzealous fidonet backbone operators ruined it for everyone.