Slashdot Mirror

Sure, these data formats may be inefficient (although HTML and XML compress nicely, and good use of CSS can dramatically slim down pages). But efficiency costs. The solutions to your problems will therefore act as a barrier to access to those who can't afford it, among both content producers and software developers.

Take RSS and Atom. Anyone can install open source blogging software and produce a syndication feed. If we replace that with a more sophisticated protocol (which is often proposed) then folks with low-end web hosting won't be able to syndicate. Your friendly neighborhood open source developers might find the new spec too complex and simply not implement it in their (often PHP) apps; once they do get it working, the complex code is harder to maintain and improve.

Put another way: expensive broadband creates a digital divide in the audience. Fancy code creates a digital among producers. As with the impact of expensive production values on video games and movies, it would likely serve to squeeze out the creatives. It's the accessibility of the Web that keeps it from being just another mass medium.

You're right that some basic steps would help tremendously. I just read that less than 10% of feeds use gzip compression. The Apache guys put it in and no one else needs to know. But efficiency shouldn't rule. Optimization should come last, just as in programming.

I did some measurements across several hundred thousand of the most prominent RSS feeds, and I found that only a few actually return a compressed feed when so requested.

On average, compressed feeds are 30.42% of the size of the original, as you can see in graphical form here.

Better support for mod_gzip would certainly help to reduce the impact of RSS polling, but then again so would proper use of conditional get.

i believe syndic8 has about the most complete collection of feeds.

If you like comics, here are a few in RSS form:

Dilbert
Penny Arcade
Foxtrot

Search around for your favorites.
Syndic8.com is a good place to search for feeds.

Keep in mind that it is increasingly likely that Atom will kick RSS's butt, at least as long as Dave "the Whiner" Winer continues to control RSS and to refuse to improve it. Indeed, the primarily reason that Atom exists is that Winer refused to allow any improvements to RSS. If I were you, I would investigate Atom (and Atom feeds) rather than spend time on that which is likely soon to become thrown into on the rubbish bin of history.

Didn't really look at it, but found this using google: http://www.syndic8.com/

Syndic8.com is a very useful resource if you want to search for a syndicated version of your favorite site(s) or just check out some random feeds. Even if the site doesn't generate its own RSS you can often find a 3rd party "scraped" feed. (I know that FARK for instance doesn't use RSS)

Any war stories?

I have 5 million rows in my headline table at www.syndic8.com . Retrievals on the table
are very fast. This is on a 1.2 Ghz Athlon running MySQL.

The biggest issue is that I am perilously close to ext2's 2 GB limit on file sizes. I will fix this with some reorganization and some data compression.

syndic8 is a site dedicated to exactly this sort of thing. it has a rapidly-growing list of newsfeeds, and has aspirations of becoming a focal point for advocacy related to convincing more publishers to produce their own feeds.