Slashdot Mirror
Text Mining the New York Times
Roland Piquepaille writes "Text mining is a computer technique to extract useful information from unstructured text. And it's a difficult task. But now, using a relatively new method named topic modeling, computer scientists from University of California, Irvine (UCI), have analyzed 330,000 stories published by the New York Times between 2000 and 2002 in just a few hours. They were able to automatically isolate topics such as the Tour de France, prices of apartments in Brooklyn or dinosaur bones. This technique could soon be used not only by homeland security experts or librarians, but also by physicians, lawyers, real estate people, and even by yourself. Read more for additional details and a graph showing how the researchers discovered links between topics and people."
For every time homeland security is mentioned as benefitting of a new technology, you should get a swift kick to the nuts. Goddam, there is more than just terrorism in this world.
Nobody likes you.
For example, the model generated a list of words that included "rider," "bike," "race," "Lance Armstrong" and "Jan Ullrich."
From this, researchers were easily able to identify that topic as the Tour de France.
I imagine "testosterone", "doping", and "supportive mother", would have found the Tour de France topic even faster.
In post Patriot Act America, the library books scan you.
A relative new method? A difficult task? Sorry, but these are almost laughable, even for a poor spaniard like me.
"Home atlast after another long day in the salt^H^H^H^Htext mines.
We lost four more miners today, bless their souls. The foreman kept insisting they'd dig another tunnel between bicycling and Tour de France. They told him it was too dangerous, but no... he never listens. One of these days... They've got us working 20 hour shifts in the abyss that is the text mines, barely pay us enough to afford the rent, I'm telling you, one of these days..."
GAAH! MY PRINTER IS ON FIRE!!! PUT IT OUT! PUT IT OUT!
If this can be implemented into research in academia, is searching through decades of articles and abstracts finally going to be more efficient? Provided that they are electronic of course. Poor citations, inaccurate keyword tags, obscure sources...ahh reminds me of grad school.
But does it also ditch the ads?
I thought this was fairly easy to do with a Support Vector Machine. (http://en.wikipedia.org/wiki/Support_Vector_Machi ne )
Or even simple Decision trees by setting the threshold for certain words.
(http://en.wikipedia.org/wiki/Decision_tree)
You mean they can group data by topic? Like clusty.com does when you search?
I just read the stub of the article... because it seemed like it does exactly what clusty does and I don't care to read anymore.
--------========+++Dont Feed The Lab Techs+++========--------
Google buys the University of California computer science school
"This research work has been presented by Newman and his colleagues during the IEEE Intelligence and Security Informatics Conference" .... Hello Newman....
Has anyone realised that english is one of the most screwed up, stupid languages ever created? its just been stretched and modified in any way possible and some aspects of it are practically useless. Maybe the world would be better off inventing a better language than analysing a horrible one :P
I have available to me quite a large database of historical research spanning back to 1991, being freeform copies of emails between researchers and acedemics on a wide variety of topics to do with a specific topic from the 15th century. Dry stuff, but a very exciting topic.
;) but it does open up some interesting possibilities.
;)
At the moment the data is mined with wildcard text searching, which means you need to know the subject before you can participate. It's a very valuable resource, but it's also not used to it's potential due to the clunky methods of interfacing with it.
It will be quite interesting applying this technique to the dataset to see if unknown relationships become apparent or known relationships become clearer.
Looking at the paper and samples would indicate this tool (if it does what it promises) might be able to not only work out the correlation between datum but to create visual diagrams linking people, places and events quite well. A handy tool for my dataset.
I'm now sitting here crystal ball gazing; if we were to expand this to a 3D map. Say by displaying a resulting chart and allow a researcher to hotlink to the data underneath it would be an interesting way to navigate a complex topic, more so than a text based wild or fuzzy search. Of course I won't know if this is possible until I look into the program more, and I won't be able to look into the program more until I massage teh dataset again
Click on the Anthony Ashcam box and see the hotlinking and unfolding of data specific to him. Drill in more... then more... and eventually get to a specific fact.
The only problem will be that I would need to pre-compute all the charts. Oh well, one day
Orationem pulchram non habens, scribo ista linea in lingua Latina
An artificial intelligence could maybe use these new methods to grok all human knowledge contained in all textual material all over the World Wide Web.
Technological Singularity -- -- here we come!
We're doomed! DOOOOOOOOOOMED!
So how is this not simply automated discourse analysis?
http://amishthrasher.blogspot.com/
I have to agree with the first response (swift kick in the nuts to whomever came up with that). It's called Google or Regex, whatever you want to use to strip unwanted content from a search.
Not revolutionary. In fact, they're late.
Google AdSense network has done this for years to serve contextually-relevant text ads across thousands of websites. Yahoo now, too.
Wow, they figured out how to use grep!
"Maybe the world would be better off inventing a better language than analysing a horrible one :P"
So how would we read your posts then?
Agreed. We jump all over politicians and their "think of the children", and we do no better with our "homeland security". As I posted elsewere the advance of technology can free a public as much as it can enslave it.
It's hard from the press release to understand what's the innovation here. Certainly unsupervised text mining techniques have been around for a long time. Latent Semantic Indexing has been around for fifteen years.
...a load of grep.
Autonomous Retard -- Is your camp safe? UnsafeCamp.com
How is this hard to do? It seems like this could be done with relatively simple algorithms.
Phil Schrodt at the U of Kansas has been doing something similar for years using The Kansas Event Data System (and its new update, TABARI). He started using Reuters news summaries to feed the KEDS engine back in the 1990s.
Following Schrodt's work, Doug Bond and his brother, both recently of Harvard, produced the IDEAS database using machine-based coding.
These types of data can be categorized by keywords or topic, though the engines don't try to generate links. The resulting data can also be used for statistical analysis in a certain slashdotter's dissertation research...
"Search" is a word used by "commoners".
"Google" is a word used by "commoners that play on 'puters".
But "text mine"? Why, that's a word meant only for science's finest.
A rose may still smell as beautiful when it's named shit, but who names their daughters "Shit", like, ever?
The new method that they figured out was
"site:newyorktimes.com "Tour de France" "
Visit my site @ http://www.madtorrent.com
We were doing this in 1989 with long free form responent answers to marketing questions to gain information about their actual preferences. Full natural language processing. We didn't patent the technique because we thought it was obvious - and we were too dumb to know how difficult a thing we achieved. It worked wonderfully. Ours worked in Japanese, German, and Thai, too - I bet their's only works in English, and American English at that. Of course it took us several months to teach it the decoding matrix for each language. I always think of this as the coolest computer related thing I ever did.
As William Burroughs suggested, the goal of the Aftosa Commission is not to rid the world of bovine aftosa. It's goal is to justify its existence and continue to enlarge its budget and its manpower until the world understands that bovine aftosa is such a critical issue that there needs to be a cabinet level Office of Bovine Aftosa with a budget only surpassed by that of the military. No one in government ever does anything that could conceivably put them out of business. This is why relying on the military and the "defense" contractors to bring peace is such a dangerous activity.
The demonstration is significant because it is one of the earliest showing that an extremely efficient, yet very complicated, technology called text mining is on the brink of becoming a tool useful to more than highly trained computer programmers and homeland security experts.
On the brink? Q-Phrase has desktop software that does this exact type of topic modeling on huge datasets - and it runs on any Windows or OS X box. [Disclaimer: I work there] And there are a number of companies (e.g. Vivisimo/Clusty) that uses these techniques as well.
Going beyond the pure mechanics (this article speaks of research that is only groundbreaking in their speed of mining huge data sets), there are more interesting uses for topic modeling such as its application to already loosely correlated data sets. A prime example: mining the text from the result pages that are returned from a typical Google search. One of our products, CQ web does exactly this (and bonus: it's freeware):
Using the example from the story: in CQ web, text mining the top 100 results from a Google search of "tour de france" takes about 20 seconds (via broadband) and produces topics such as:
floyd landis
lance armstrong
yellow jersey
time trial
And going beyond simple topic analysis: using CQ web's "Dig In" feature (which provides relevant citations from the raw data) on floyd landis returns "Floyd landis has tested positive for high leves of testosterone during the tour de france." as the most relevant sentence from over 100 pages of unstructured text.
So, while this is a somewhat interesting article, fact is, anyone can download software today that accomplishes much of this "groundbreaking" research and beyond.
Follow that up by reaching down his throat so you can rip out his spine and strangle him with it. Then tear off his head and go bowling with his dead skull. Have a few beers and enjoy the experience. After that, shit down the bloody stump of his neck - big, nasty, stinky beer shit.
Just on general principles.
There's no need to use any excuse.
330,000 articles at $3 each comes to $990,000, almost a million dollars for their data mining experiment. No wonder tuition costs are so high when this is what they're spending their money on!
I can text-mine the NYTimes without even accessing the text:
It's all Bush's fault.
Business is evil.
Tax cuts are bad.
Republicans are fascists.
Google News does a rather good job of associating all the stories on the same topic. I'd thought this was a solved problem.
The fastest way to correct many of the problems with the American English language could easilly be solved by switching to a pure phonetic spelling instead of the misbegotten methods we're now using. In other words, spelly it the way it sounds. EG: Fone instead of Phone, Duk instead if Duck and so on. Yes it's going to create havik but I think it will ease not only the learning of English (remember there are only 33/35 sounds in the language) but it will also increase Speech Recognition as you will then be pronouncing a word as it's supposed to be spelt.
As William Burroughs suggested, the goal of the Aftosa Commission is not to rid the world of bovine aftosa. It's goal is to justify its existence and continue to enlarge its budget and its manpower until the world understands that bovine aftosa is such a critical issue that there needs to be a cabinet level Office of Bovine Aftosa with a budget only surpassed by that of the military. No one in government ever does anything that could conceivably put them out of business. This is why relying on the military and the "defense" contractors to bring peace is such a dangerous activity.
The same could be said of registered charities.
It doesn't come bundled with an analysis engine, but if you're looking to build your own corpus of material (e.g., by automating searches or harvesting large volumes of your research web pages) and you're on MacOSX, check out Anthracite web mining desktop toolkit... It makes it easy to build spidering and scraping systems, structure the output and feed it into a database like MySQL...all without requiring you to write a single line of code. Take that output and feed it into any number of the analysis and search systems on SourceForge or Freshmeat and you're going to get comparable results without all the fuss, although you should definitely write a press release about it! The Google API and regex support are built-in, and you can even run the data through any UNIX command (e.g., grep or Perl) without leaving the program if you need even more. As for speed, the new release is going to feature a throttle because a few customers are getting overwhelmed by the URL loading throughput. Yes, by way of full disclosure, I wrote the software and that's why I'm always busy promoting it.
Edward Herman and Noam Chomsky may or may not have had a fancy computerized search system, but association of loaded keywords was a major topic in Manufacturing Consent (ISBN 0375714499) where the influences of commercial interests on the media and government was analyzed using the New York Times. The great improvement in the rate at which text can be analyzed should make for an excellent third edition.
I'll be your candy shop of infinite deliciousity if you'll be my discotheque of endless rump-shaking.
Some people have suggested to combine both: make a new version of english, with dumbed-down grammar and a reduced vocabulary. Egyptian-taxi-driver-english if you want. That I believe would be a good solution, as everybody could learn it, and those with time/talent could "move on" to normal english.
10 ?"Hello World" life was simple then
Out of the thousands of papers published on this subject every year, Roland Piquepaille picks this one.
This is interesting, but the idea has been around for more than 50 years, and practiced using automated computers (as opposed to human coders) since the 1960s. Lerner and de Sola Pool came up with the idea of using "themes" to analyze political texts at Stanford in 1954, and hundreds or even thousands of studies using automated text analysis tools have been performed since then. You can download a free text analysis tool called Yoshikoder, which will perform frequency counts of all words in a text, as well as dictionary analysis, and several other functions. So why is this news now? I think the press release is really leaving out some key information. I think the more relevant questions that should have been addressed in the original release is how the text was prepared for analysis, because most websites and online databases of news articles (LexisNexis, Factiva, etc.) don't allow batch downloads of huge amounts of news text in XML or some other format that can be easily parsed by text analysis programs.
- basically, the model assigns a probability distribution over topics to each document
- topics are learned from the documents automatically, not pre-defined
- the technique can learn which authors are likely to have written various pieces of a given document, or which cited documents are likely to have contributed most to this document
For a good high level description of what these models are doing, see Mark Steyvers' research page (MS is one of the authors); that page also has links to a number of the preceding papers. Those interested in seeing what the output of a related model looks like might like to check out the Author-Topic Browser.i.e., documents aren't assigned to a single topic (as in latent semantic analysis (LSA))
this means, incidentally, that they're not automatically labeled, although a list of the top 5 words for a topic generally characterizes it pretty well.
side benefit: you can also discover misattributions (e.g., authors with the same name)
i heard on the (fox) news that NY Times writers ARE the terrorists.
The Klingon Language Institute
http://www.kli.org/
I wonder how the classifier program would cope with text like that in the parent post... probably sprain its parser, or something.
"Is life so dear, or peace so sweet, as to be purchased at the price of chains and slavery?" - Patrick Henry
Context-sensitive adaptive parsing seems to be effective in parsing English even with very small (http://www.sand-stone.com/Meta-S.htm for an introduction. (The 2nd reference is on natural language parsing.)
"Is life so dear, or peace so sweet, as to be purchased at the price of chains and slavery?" - Patrick Henry