Slashdot Mirror

The computational effort for short word sequences is no longer much of an issue. For example, the web clustering algorithm in the free application CQ web computes clusters in corpus phrases up to seven words in length, and it runs without a hiccup on your standard Windows or Mac desktop.

4. The first time that a single query will bring a gallery of
            results equivalent to running multiple queries about the
            meaningful variations of the same topic.

  5. The first time a search engine will let users evaluate answers
            on the spot by displaying uninterrupted and coherent text
            snippets, often letting searchers forgo having to click through
            to links and saving time.

Both of these have been available for a couple of years: e.g. searching on the single query "semantic web" using CQ web, reveals clusters such as these:


fuzzy sets
fuzzy systems
neural networks
set theory
soft computing
aritifical intelligence
control systems
expert systems


And each one of which is linked to a specific page of results using sentences instead of snippets, e.g. for artificial intelligence:


1. This paper will present the foundations of fuzzy systems...noteworthy objections to its use with examples drawn from current research in the field of artificial intelligence.
Fuzzy Systems - A Tutorial
2. The most obvious implementation for the fuzzy logic is the field of artificial intelligence.
Fuzzy Logic
3. Ultimately it will be demonstrated...fuzzy systems makes a viable addition to the field of artificial intelligence and perhaps more generally to formal mathematics.
Fuzzy Systems - A Tutorial
4. The paper gives examples of the fuzzy logic applications with emphasis on the field of artificial intelligence.
Fuzzy Logic
5. A collection of articles and other technical resources for artificial intelligence.
PC AI - Fuzzy Logic

Well, as an author of one of those, er, in your words "stupid" posts, I can assure you that I didn't mean to imply UCI's research was trivial. Rather, it was the press release that was trivial, and bit of a puff piece IMHO, suggesting that:

"To put it simply, text mining has made an evolutionary jump. In just a few short years, it could become a common and useful tool for everyone from medical doctors to advertisers; publishers to politicians."

And my point still is that nobody needs to wait a few short years to do decent text mining from unstructured data. Can our software handle 300,000 articles from the NYT? Clearly not, but then again, we're not running our software on desktop machines. Fact is, a million words (or about 3000 NYT articles) is a trivial task for our software and allows people to use text mining today.

Now, back when I went to Cornell, I thought my peers expressed a bit more intellectual curiousity about software, especially the free kind that would allow them to save their $ for The Palms. But times do change, and if you think "stupid" is an accurate assessment of my post than more power to you. ;-)

And for the rest of you, yeah, I'm going to end this with a plug (natch): download CQ web for OS X or Windows if you want to see how text mining works on web search result pages.

That's all well and good, but freeware that does something even remotely similar is still cool.

The demonstration is significant because it is one of the earliest showing that an extremely efficient, yet very complicated, technology called text mining is on the brink of becoming a tool useful to more than highly trained computer programmers and homeland security experts.

On the brink? Q-Phrase has desktop software that does this exact type of topic modeling on huge datasets - and it runs on any Windows or OS X box. [Disclaimer: I work there] And there are a number of companies (e.g. Vivisimo/Clusty) that uses these techniques as well.

Going beyond the pure mechanics (this article speaks of research that is only groundbreaking in their speed of mining huge data sets), there are more interesting uses for topic modeling such as its application to already loosely correlated data sets. A prime example: mining the text from the result pages that are returned from a typical Google search. One of our products, CQ web does exactly this (and bonus: it's freeware):

Using the example from the story: in CQ web, text mining the top 100 results from a Google search of "tour de france" takes about 20 seconds (via broadband) and produces topics such as:
floyd landis
lance armstrong
yellow jersey
time trial

And going beyond simple topic analysis: using CQ web's "Dig In" feature (which provides relevant citations from the raw data) on floyd landis returns "Floyd landis has tested positive for high leves of testosterone during the tour de france." as the most relevant sentence from over 100 pages of unstructured text.

So, while this is a somewhat interesting article, fact is, anyone can download software today that accomplishes much of this "groundbreaking" research and beyond.

The demonstration is significant because it is one of the earliest showing that an extremely efficient, yet very complicated, technology called text mining is on the brink of becoming a tool useful to more than highly trained computer programmers and homeland security experts.

On the brink? Q-Phrase has desktop software that does this exact type of topic modeling on huge datasets - and it runs on any Windows or OS X box. [Disclaimer: I work there] And there are a number of companies (e.g. Vivisimo/Clusty) that uses these techniques as well.

Going beyond the pure mechanics (this article speaks of research that is only groundbreaking in their speed of mining huge data sets), there are more interesting uses for topic modeling such as its application to already loosely correlated data sets. A prime example: mining the text from the result pages that are returned from a typical Google search. One of our products, CQ web does exactly this (and bonus: it's freeware):

Using the example from the story: in CQ web, text mining the top 100 results from a Google search of "tour de france" takes about 20 seconds (via broadband) and produces topics such as:
floyd landis
lance armstrong
yellow jersey
time trial

And going beyond simple topic analysis: using CQ web's "Dig In" feature (which provides relevant citations from the raw data) on floyd landis returns "Floyd landis has tested positive for high leves of testosterone during the tour de france." as the most relevant sentence from over 100 pages of unstructured text.

So, while this is a somewhat interesting article, fact is, anyone can download software today that accomplishes much of this "groundbreaking" research and beyond.

The demonstration is significant because it is one of the earliest showing that an extremely efficient, yet very complicated, technology called text mining is on the brink of becoming a tool useful to more than highly trained computer programmers and homeland security experts.

On the brink? Q-Phrase has desktop software that does this exact type of topic modeling on huge datasets - and it runs on any Windows or OS X box. [Disclaimer: I work there] And there are a number of companies (e.g. Vivisimo/Clusty) that uses these techniques as well.

Going beyond the pure mechanics (this article speaks of research that is only groundbreaking in their speed of mining huge data sets), there are more interesting uses for topic modeling such as its application to already loosely correlated data sets. A prime example: mining the text from the result pages that are returned from a typical Google search. One of our products, CQ web does exactly this (and bonus: it's freeware):

Using the example from the story: in CQ web, text mining the top 100 results from a Google search of "tour de france" takes about 20 seconds (via broadband) and produces topics such as:
floyd landis
lance armstrong
yellow jersey
time trial

And going beyond simple topic analysis: using CQ web's "Dig In" feature (which provides relevant citations from the raw data) on floyd landis returns "Floyd landis has tested positive for high leves of testosterone during the tour de france." as the most relevant sentence from over 100 pages of unstructured text.

So, while this is a somewhat interesting article, fact is, anyone can download software today that accomplishes much of this "groundbreaking" research and beyond.

If you want to check out a free clustering search client (it clusters results from Google, msn, etc.) check out CQ web. Windows and Mac OS X versions are available.

Although trust is certainly a issue when it comes to the Semantic Web, the real problem is that its design is not a true abstraction, but is nothing more than more metadata. And like the actual textual data in a typical web page, it suffers from all the same problems, save for one: being unstructured (and thus not truly parseable).

IMHO, the Semantic Web is solving one problem (the lack of structure and descriptive context in textual HTML content) in a very hard way (asking the entire web to implement this new RDF).

Many companies (disclaimer: like my own) are approaching these problems from a different angle: working on statistical and semantic systems to extract structurue from the textual content that is already there on the web page.

Now some people will argue that trying to create a system that can understand langauge/content is insanely difficult.

But what is a more realistic time frame? The one in which an intelligent parser can begin to understand the content that is already on the web, or the one which requires the entire world to implement a solution to a problem they don't even realize is a problem?

The free search agent CQ web uses this exact strategy, but programatically rather than via human modding. For example, if you search for "tom cruise" in Google via CQ web, it will ingest the content of the first 100 results and then use all that data to determine a baseline of statistically significant keywords and phrases (e.g. "mission impossible", "katie holmes", "chuch of scientology"). Then, CQ web re-evaluates the relevance of each result based on its "closeness" to the baseline. This generally moves spam pages out of the way and pushes up content rich sites. Plus, a quick glance of key words and phrases allows you to get "good results up front" by allowing you to decide what subcategory to dig into for more information.

Ask.com has many features not available with rivals -- topic clusters

Actually, you can "roll your own" topic clusters from results in Google, MSN, del.icio.us, etc. by using CQ web, a free contextual search agent for Windows and OS X.

All the hoopla around the Semantic Web reminds me exactly of the days "XML" became the latest high-flying meme touted by "tech" writers en masse. Witness:

The semantic search engine would then cross-reference all of the information about hotels in Majorca, including checking whether the rooms are available, and then bring back the results which match your query.

And here in all its glory is the 1999 version:
The software would then use XML to cross-reference all of the information about hotels in Majorca, including checking whether the rooms are available, and then bring back the results which match your query.

Of course, the problem with this fantasy of XML was that no standardization of schemas led to an infinite mix of tagging and thus, the laypersons idea that "this XML document can be read and understood by any software" was pure bunk.

Granted, the semantic web addresses many of these problems, but IMHO the underlying problem remains: layers of context on top of content still need to be parsed and understood.

So the question remains: will the Semantic Web be implemented in a useful fashion before some develops a Contextual Web Mining system that understands web content to a degree that it fufills the promise of the Semantic Web without additional context?

Disclaimer: I work on contextual web content extraction software so yes I may be biased towards this solution, but I really think the Semantic Web has a insanely high hurdle (proper implementation in millions of web pages) before we can tell how successful it is.

for a while. The idea of sifting through search results to find related topics has been done by at least a few companies (including mine), and these products predate this tech (which was officially anounced in Sep. 2005) so I don't think Google will be able to defend a US Patent on this.

If you want a demo of a product (mine, natch) that's been around in one from or another since 2004, check out Q-Phrase's ConceptQ Pro product. A free version which does just web search will be coming soon.

Here's a screenshot of a search of the entire 9/11 report, broken down into relevant topics.

for a while. The idea of sifting through search results to find related topics has been done by at least a few companies (including mine), and these products predate this tech (which was officially anounced in Sep. 2005) so I don't think Google will be able to defend a US Patent on this.

If you want a demo of a product (mine, natch) that's been around in one from or another since 2004, check out Q-Phrase's ConceptQ Pro product. A free version which does just web search will be coming soon.

Here's a screenshot of a search of the entire 9/11 report, broken down into relevant topics.