Using AI To Filter RSS Feeds

holden writes "According to a blog post, AideRSS has moved from closed to open beta. I've been using AideRSS over the past few weeks to filter my RSS feeds (including Slashdot and Reddit) and I've been quite impressed. They talk a bit about how the filtering system works, which apparently tracks a mixture of things, from pick-up in other blogs, to some clustering technology."

Comments

Some filtered RSS feeds

[holdenkarau]

I'm not sure if it is bad form to comment on your own story, but here goes anyways :). You can take a look at the scored version of the slashdot RSS feed here, or del.icio.us or my (holden's) blog. There is also a really cool widget I've put on the side of my blog which lets people subscribe to only posts of a certain quality (you can look at it here).

Secret Sauce and GeoRSS

[Lord Satri]

From TFA: ""Some of that data we show on the site itself: Technorati, del.icio.us, etc. Essentially, we're interested in measuring the 'social engagement' of each post. To make this a little less hand-wavy, I think we'll agree that a bookmark is nice but a comment involves more work, a trackback even more so, etc. - hence, engagement). Once we have all this data, we apply our 'secret sauce', which comes in a form of statistical analysis with respect to the author's previous history/posts. PostRank is not a global score, it's with respect to the blogger him/herself.""

Secret sauce? Why do I prefer open sauce? ;-)

One other way to filter RSS is by geographic location through using GeoRSS. However, the source RSS must be offered in GeoRSS for this geolocalization filtering to work... but it's only a matter of time, we'll get there. (hey, even slash has a plugin that works for publishing GeoRSS)

Re:Secret Sauce and GeoRSS

[bergie]

Geolocation is a possible additional filter (think "local news" section of a newspaper), but I guess most people are interested in items from their field of interest regardless of the physical location where the post was made.

I made some experiments on a more open source version of the "secret sauce". It seems quite easy to determine relevance of posts using the various social news services out there.

If Only ...

[nxtr]

If only they could get the AI to do the work I'm missing out when I'm reading RSS feeds.

Potentially scary side-effects already.

[gethoht]

There are some companies out there(i.e. http://www.collectiveintellect.com/) that are using AI to mine RSS feeds and specifically the blogosphere, and selling that data to corporations for various reasons.

Lets say you're a drug company that is releasing a potentially controversial drug. You can mine the data of the blogosphere and issue press releases as a pre-emptive strike to larger media stories. This starts the real beginning of being able to effectively monitor and even potentially control some of the social aspects of the internet. I think it's a great innovation indeed, with potentially scary side-effects.

Personally it is nice to be able to filter through a billion RSS feeds to find information that I'm interested in though.

Re:Potentially scary side-effects already.

[rm999]

"This starts the real beginning of being able to effectively monitor and even potentially control some of the social aspects of the internet"

I fail to see your reasoning. Companies have always been able to "monitor" blogs and subscribe to RSS feeds. And they aren't controlling the social aspects of the internet at all. A press release has always been a standard communication means of corporations; as long as they aren't creating fake blogs, I don't think they are trying to control any aspect of the social internet.

And personally, if a company does analyze blogs, I think it's a great thing. It means they care what normal people think about them and their products. Almost every blogger who talks about a company hopes that the company is listening to them.

As an AI student, I wish people on Slashdot weren't so afraid of "intelligent" algorithms. They really aren't meant to be evil, they are usually meant to make something that is tedious more efficient. Yes, it can be abused, but just about everything can; For example, just because airplanes are used by the military to kill people does not make airplanes inherently evil.

Re:Potentially scary side-effects already.

[dada21]

There are some companies out there(i.e. http://www.collectiveintellect.com/) that are using AI to mine RSS feeds and specifically the blogosphere, and selling that data to corporations for various reasons.

Sounds like a fantastic market, actually. I recently picked up a client in the casino management market because I had made some comments on a blog regarding their lack of insight towards proper marketing and keeping a decent percentage of return customers. They actually contacted me, and I've spent a large amount of time parsing through literally hundreds of RSS feeds covering different search terms (the company, the company's competitors, key words of the market, etc, etc). They're always impressed when I hit them up to 50-100 new blogs, news reports, and websites condemning or supporting various new tactics or old dogs. I think that selling companies information that can help them (fix or spin) the problem is a huge market waiting to be tapped properly.

Lets say you're a drug company that is releasing a potentially controversial drug. You can mine the data of the blogosphere and issue press releases as a pre-emptive strike to larger media stories. This starts the real beginning of being able to effectively monitor and even potentially control some of the social aspects of the internet. I think it's a great innovation indeed, with potentially scary side-effects.

I'm not sure I'd call any of the side-effects scary, though. Honestly, I hit Google Blogsearch and Technorati before I hit Google News. What is most important to me, though, is a personal rating system so I can give weight to certain information providers that have given me good information. Lately, I've had better luck from blogs than from the mainstream media. That's a nail in the coffin for the MSM if they don't move with the speed that today's news readers desire.

The benefit of the "free market of information" on the web is that you can now see way more than the black and white sides of things. A drug company may put a spin on bad news, but the issue isn't just "bad news" and "good news." There might be 3 or 4 degrees of separation between the actual negative news and what others may say that could have a positive effect. With the growing (and shrinking?) anonymity of the web, whistle-blowers also have a chance to get the word out -- but again, without a positive history of truth, they may not have a high ranking with me.

Personally it is nice to be able to filter through a billion RSS feeds to find information that I'm interested in though.

I've been sorting through thousands of RSS feeds for a few years now, and have been using some collaborative filtering sites and systems to try to give weight to what I consider the better news sources. Collaborative filtering will be a necessary element to an AI filter because what matters to YOU personally may be missed, but if you use a collaborative ranking, you will also gather information that is important to people who have similar views/needs that you do.

I find a lot more power in the collaborative filtering market than in the AI market. The downside of collaborative filtering is that it can be gamed (see Digg). The upside is that metamoderating of other collaborators can work to fix the gaming individual at a time.

Another site using AI

[Sanity]

Thoof (disclaimer: its my website) uses Bayesian analysis (you could call it AI, so much as anything is AI) to determine what you are interested in reading, based on a variety of factors, including:

• The referring website (and what other people from that site liked)
• Your OS/Browser (and what other people with your OS/Browser liked)
• Your geographic location (and what other people close to you liked)
• What you yourself read

It also allows users to edit stories, a mechanism conceptually similar to a wiki, but with an additional voting process to help prevent abuse.

Unlike AideRSS, Thoof isn't an RSS aggregator, rather users submit stories, in a manner similar to Slashdot, Digg, and Reddit.

recursion

[shird]

What if the 'other blogs' they 'pick up' on, are in turn using AideRSS to determine what to blog. The whole blogging thing really does seem like one giant feedback loop with only a few people generating actual useful content.

Re:recursion

[jagdish]

So basically the situation would be unchanged.

Artificial Intelligence on Slashdot

[Anonymous Coward]

I suppose in these modern days when natural resources are being rapidly depleted by overpopulation and overconsumption, there had to come a time when we would start running out of intelligence... of course I wouldn't know because I'm a little short on it myself...
however
It is pleasing to see that scientists around the world have started to produce artificial intelligence to make up for the loss of natural intelligence, but I think that like everything else, perhaps it is also equally important that we conserve and recycle the little natural intelligence we have left and refine our methods to efficiently extract and use that intelligence to, uh, do something or other, but do it efficiently and without any needless waste. Yes, that's my point.

And to that end I see this Artificial Intelligence RSS Feed Filter as a great marvelous invention, because you see, it combines the old and the new, it uses artificial intelligence to extract natural intelligence efficiently and use it for something in a wonderful postmodern fashion. Now, modern invention assists the primitive natural.

Now, all we need is to have a massive SETI like project running this AI RSS Filler Feeder to search for signs of intelligence on slashdot. Oh, oh, cross my fingers, I hope my post makes it pass the filter...

Re:Download URL

[forgotten_my_nick]

Someone can correct me if I am wrong but it looks like your supposed to put your feeds into the website then link to the one feed there.

Seeing as half my feeds are internal work related and the fact I don't want someone profiling all feeds I am reading I won't be using the service.

Sux0r : Bayesian RSS filter and you can run it too

[Herve5]

http://www.nullwhore.com/sux0r/index.php?c=/0/logi n/
http://sourceforge.net/projects/sux0r/
What I find interesting is, it is one of the verrry rare examples of 'internet 2' service that you can own yourself (instead of registering here or there for more ads or worse).
A downside of Sux0r is it seems not having evolved for a couple of years (but still works, possibly that's why :-)
I for one am desperately waiting for a *local* RSS agregator which would allow *me* (and not some site's AI) to Bayes-filter my selected feeds. I'm almost sure this will happenn sooner or later.

Personalize instead

[Catil]

I think there are basically two kinds of RSS Feeds, either they show the latest news (last in first out) or they show an already sorted frontpage (e.g. "crowdsourced" like Digg); both are useful.

Using an AI to resort those feeds is definitely interesting from a coders point of view but trying to give some kind of objective view to a feed is probably not what the average user wants.

Why not do it the other way around and personalize them instead? Maybe it has been done before, but it would be nice if there was a reader to rerank (or even filter out) certain domains, keywords, tags and categories. It could take the given rank as the base score and then resort it according to the user's personal preference, e.g. if someone doesn't like politics he could give the keywords "Bush, Cheney, election, etc." a negative mulitplier and maybe the keyword "funny" gets a positive one. It could even consider the time of the day - politics in the morning and funny pictures during the lunchbreak or something.

Just a qick thought though, someone can perhaps come up with something better. Anyway, I am pretty sure that personalization is the better approach here.

Re:Personalize instead

[adam.jimenez]

spot on.

i also think their should just be a thumbs up/ thumbs down option which would save you typing in.