Slashdot Mirror
Text Mining the Multiverse
The NYT has a decent piece about text-mining, skimming large volumes of miscellaneous text to extract some sort of refined knowledge from it.
← Back to Stories (view on slashdot.org)
← Back to Stories (view on slashdot.org)
(Yes I am a lazy /. reader)
skimming large volumes of miscellaneous text to extract some sort of refined knowledge from it.
Like those ppl who actually RTFA and try to get "FORST PIST!!!"?
do() || do_not();
The same article is also posted at CNET, which doesn't require registration. They also have it in a nice single-page format for those that don't like to keep hitting "next".
To make sense of what it is reading, the software uses algorithms to examine the context behind words.
They make it sound like Semantic and Contextual modeling is done on the fly -- the way I see this system, it does this based on a preset lexicon or database.
Thats again brute forcing the problem -- a lot of researchers in the field feel that real solution does not lie that way. We need to analyze this from ground up, to gather meaning from data.
The above method fails the moment you have spatial and temporal data -- my lexicon may evolve over a period of time.
You're looking at all the information and then deciding whats for you -- a better way is to develop an "instinct" for the right kind of information and refine it.
If you really want to know where data mining is going to, look at KDD or SIGMOD -- thats where all the real action is.
The scary part (that took mathmeticians a long time to accept and longer to figure out) is that the distribution is the same for any sufficiently representitive sample of text....
"to extract some sort of refined knowledge from it." hum.... ....Infinite number of shot guns & shotgun shells.... And an infinite number of stop signs, you will eventually get Shakespeare in brail.....
If you have an infinite number of red necks
Julius Caesar - Act I, Scene i: "What mean'st thou by that? Mend me, thou saucy fellow!"
...skimming large volumes of miscellaneous text to extract some sort of refined knowledge from it.
Dear Text Miners,
Please start here: http://slashdot.org
Thanks so much.
Operator, give me the number for 911!
May I offer that computers make no sense of what they are reading & that "smart people with knowledge of the particular subject" aren't optional if the results of text-mining are to be of any usefulness whatsoever, at least in any kind of reasonable time frame.
Otherwise, the text-mining computer is playing the old "99 monkeys with typewriters" game...
"Obviously, I'm not an IBM computer any more than I'm an ashtray" (Bob Dylan)
i always wondered about this
allright, you can take huge amounts of text and apply some smart tricks to extract patterns from it.
but how can you determine whether the original data was trustworthy?
take the example of genome annotation (description of gene function), which would be helped greatly by including more functional descriptions from scientific literature. how do you determine whether the original publication was backed by solid experimental research?
by the reviewers of the articles? i don't think so, peer review is a snakepit filled with politics. by the amount of people who cited it? hmmmm... so hip subjects are more true?
me personally, because i'm experienced, can recognise bullshit articles when i see them. but how to translate this into an algorithm... anyone any ideas about this? or even working solutions?
(of course this is an example from my field of expertise - biology, but it applies to any set of text data/articles IMO)
(1)"Of course, no one, Dr. Liebman included, is arguing that these products are actually reading anything. What they are engaged in is "text mining,'' "
... Just a thought.
Dijkstra once said "The question of whether computers can think is like the question of whether submarines can swim."
(2)As noted in the article sarcasm is very hard to detect. If you think about it even many people have a hard time recognizing it. How are we supposed to develop an intelligent system when we "intelligent" humans don't even "get" it?
(3)"There is a need to make these technologies available for publicly available information," he wrote at his site.
Yes of course. Anyone who has done research knows how frustrating it is to read through abstract after abstract, let alone the entire publication, to find what you are looking for. In research when you are looking for facts or raw information text mining seems highly promising. Yet, for interpretive processes it grows increasingly difficult to envision a correct system. As noted nuances are difficult to detect. In addition to sarcasm, words like "still" allow for multiple meanings for the bigrams, trigrams, etc. to which they belong. Natural language ambiguity is the most important problem to overcome in NLP. After all, how would you like to write a printf statement and not know whether you would get the intended output or some other arbitrary call.
sig
The knowledge discovery and datamining cup challenge this year was looking at the arxiv.org papers for this sort of analysis - some very interesting results. The Task 4 winnder looked at the structure of the papers as a sort of relational database and uncovered a lot of statistical patterns and metrics that could be quite useful for scientists.
Energy: time to change the picture.