Slashdot Mirror
Text Mining the Multiverse
The NYT has a decent piece about text-mining, skimming large volumes of miscellaneous text to extract some sort of refined knowledge from it.
← Back to Stories (view on slashdot.org)
← Back to Stories (view on slashdot.org)
Quick, someone patent it before Microsoft does or else Slashdot is going to be the next casualty.
Then again, we could just skip the patent and let WWdN die too. Seems like the internet community would break even.
A programmer is a machine for converting coffee into code.
MICHAEL N. LIEBMAN knows his limitations. Even with a Ph.D. and a long career in medical research, he cannot keep up with all the developments in his area of interest, breast cancer. Medline, the database that already houses more than 10 million abstracts for journal articles, is adding 7,000 to 8,000 abstracts per week. Only a fraction of these are about cancer, but the volume of information is daunting nonetheless.
"There is just too much literature to be able to go through it all," said Dr. Liebman, the director of biomedical informatics at the Abramson Family Cancer Research Institute at the University of Pennsylvania.
Yet Dr. Liebman is convinced that new cures could someday emerge for breast cancer if only someone could read all the literature and synthesize it. So he has found a solution: enlisting a computer program to read the articles for him.
"The software is not going to get tired," he said. It also happens to be a speed reader: The product he is using, from a Chicago-based software company called SPSS, can zip through 250,000 pages an hour. Another product, from the text-mining company ClearForest, boasts a speed of 15,000 pages an hour, still far surpassing the human rate of a mere 60 pages.
Of course, no one, Dr. Liebman included, is arguing that these products are actually reading anything. What they are engaged in is "text mining,'' a technique that academics have been experimenting with for years but for which tools have only recently become commercially available. The prospect of rapidly scanning through reams of documents is stirring interest among researchers and analysts faced with more material than they can handle.
To the uninitiated, it may seem that Google and other Web search engines do something similar, since they also pore through reams of documents in split-second intervals. But, as experts note, search engines are merely retrieving information, displaying lists of documents that contain certain keywords.
Text-mining programs go further, categorizing information, making links between otherwise unconnected documents and providing visual maps (some look like tree branches or spokes on a wheel) to lead users down new pathways that they might not have been aware of.
Currently these programs are used by academic researchers and companies, but information scientists expect that to change. Lower-cost text-mining tools eventually will be offered to ordinary people who want to dig into medical or political issues using public documents. Madan Pandit, an expert in text analysis in Bangalore, India, who runs a Web site called K-Praxis (k-praxis.com), has suggested that text mining could help people make sense of voluminous documents that are already on the Web, like the 858-page report on the congressional inquiry into intelligence failures regarding the 9/11 terrorist attacks.
"There is a need to make these technologies available for publicly available information," he wrote at his site.
In most cases, text-mining software is built upon the foundations of data mining, which uses statistical analysis to pull information out of structured databases like product inventories and customer demographics. But text mining starts with information that doesn't come in neat rows and columns. It works on unstructured data - e-mail messages, news articles, internal reports, transcripts of phone calls and the like.
To make sense of what it is reading, the software uses algorithms to examine the context behind words. If someone is doing research on computer modeling, for example, it not only knows to discard documents about fashion models but can also extract important phrases, terms, names and locations. It can then categorize them and draw connections among the categories.
How well computers truly make sense of what they are reading is, of course, highly questionable, and most of those who use text-mining software say that it works best when guided by smart people with knowledge of the particular subject.
"I was an F.B.
CMDRTACO CHECK YOUR EMAIL!
(Yes I am a lazy /. reader)
I've always wanted to ask the computer to find all references to some complex interplay of topics at hand the way those Star Fleet engineers were always able to in TNG...
yea...
.sig
text mining is fun until someone creates something to generate a bunch of junk to feed to the text miners..
take a look at my
anime+manga together at last.. in real time.
I fail it, lah. :~(
-- The WIPO Avenger
skimming large volumes of miscellaneous text to extract some sort of refined knowledge from it.
Like those ppl who actually RTFA and try to get "FORST PIST!!!"?
do() || do_not();
skimming large volumes of miscellaneous text to extract some sort of refined knowledge from it.
If you replace "miscellaneous" with "garbled" and "refined knowledge" with "coherent sentence", this would be describing slashdot editor postings!
Multiverse doesn't appear anywhere in the article. Multiverse is a technical word, for interpreting Quantum Physics. It is totally misplaced in this news submission.
Did the poster even know what it means ?
The same article is also posted at CNET, which doesn't require registration. They also have it in a nice single-page format for those that don't like to keep hitting "next".
If I apply this to slashdot, I'll have only 3-4 posts to read everyday... What will I do all day at work?
Fwiw....
I've noticed this as well.
Netcraft confirms that the Bleagured Slashdot is Dying....
For seriously, yes. Not only have i noticed much fewer posts to articles, but much less modding up, or even modding at all in many articles. Even those posts desperately deservant of it.
do() || do_not();
This is OT, but read this journal entry from CmdrTaco.
I watched C-beams glitter in the dark near the Tannhauser gate.
Agreed.
Thanks!
yes, I have noticed the metamod thing too..
in addition there is also this tasteless group of guys who keep making posts about greased-up yoda dolls which has also forced me to start browsing at +2..
seems that mod points are being handed out with less frequency than they were before.
I think they should start handing them out for people with "excellent karma" and then track if the metamods agree with the point distribution..
that is just me..
anime+manga together at last.. in real time.
To make sense of what it is reading, the software uses algorithms to examine the context behind words.
They make it sound like Semantic and Contextual modeling is done on the fly -- the way I see this system, it does this based on a preset lexicon or database.
Thats again brute forcing the problem -- a lot of researchers in the field feel that real solution does not lie that way. We need to analyze this from ground up, to gather meaning from data.
The above method fails the moment you have spatial and temporal data -- my lexicon may evolve over a period of time.
You're looking at all the information and then deciding whats for you -- a better way is to develop an "instinct" for the right kind of information and refine it.
If you really want to know where data mining is going to, look at KDD or SIGMOD -- thats where all the real action is.
Answers to your questions: HERE
The scary part (that took mathmeticians a long time to accept and longer to figure out) is that the distribution is the same for any sufficiently representitive sample of text....
I'm glad to know I'm not the only one that hasn't gotten mod points in ages.
..until no student ever has to research any topic again?
Just head over to tellmewhatthisthingyisabout.com > Print
Sounds like the start of Xanadu.
I'd like that.
I took a speed-reading course and read War and Peace in twenty minutes. It involves Russia. -- Woody Allen
quick someon mine this and give us the refined knowledge that the article has
30% Troll, 50% Underrated, 10% Interesting
Score:5, Troll
Yes, yes and yes...
And slashdot seems to be heavily slashdotted lately... What's up with that?
I think part of it has to do with the number of posts lately. The last 10 articles (not including the current) have had an average of 224 posts. I estimated the average in early-mid summer and came up with 550. Twice as many posts = twice as much required moderation? I'm not sure how slash works in this regard.
:)
Now, as far as the reason for fewer posts, I know that the editors have said that late summer-fall tends to be slower for news, but I also think they've been putting up some boring articles lately. Granted, they may be exciting to some people, but I'm just generalizing. Hey editors... instead of posting articles just for the sake of posting articles, how about sticking only with the interesting ones and taking the leftover time and hunting for other interesting websites to feature? Maybe even write an editorial or two? A State-of-the-Slashdot address? Something fun?
You have enemies? Good. That means you've stood up for something, sometime in your life. --Winston Churchill
That the text has to first contain some knowledge in it to begin with?
Maybe this is just an attempt at getting a machine to generate core knowledge but then haven't they been working on common sense, which is sorta needed first?
Tired of going through their stupid registration? CLICK HERE
"to extract some sort of refined knowledge from it." hum.... ....Infinite number of shot guns & shotgun shells.... And an infinite number of stop signs, you will eventually get Shakespeare in brail.....
If you have an infinite number of red necks
Julius Caesar - Act I, Scene i: "What mean'st thou by that? Mend me, thou saucy fellow!"
kimming large volumes of miscellaneous text to extract some sort of refined knowledge from it.
like grep?
I'm sorry, reading this text requires meta-technology.
how long until
...skimming large volumes of miscellaneous text to extract some sort of refined knowledge from it.
Dear Text Miners,
Please start here: http://slashdot.org
Thanks so much.
Operator, give me the number for 911!
We changed a bunch of stuff and we think something might be broken but, we dont know what or where and we can't reproduce the problem so stop telling us about it. There are more people using the site so everything must be Ok.
That explains some of the problems, but not everything. For instance, why haven't I had mod points in nearly two years, despite having good karma and contributing to conversations (rather than trolling)? Yes, I know the rules about getting moderation points, but even with those I'd expect to get points at least two or three times a year, not every two years. As well, I recently started noticing that Slashdot has popup ads now (I saw one in the last day, and then added slashdot into my popup blocker's blacklist because there's no excuse for that). I usually don't pay attention to trolls, but one has to wonder if the trolls about Slashdot not being able to pay their bandwidth fees may have a kernal of truth to them? That would certainly explain why they've had connectivity issues, as well as why they've added more annoying advertisements in an attempt to scare up more advertising revenue.
Text-mining programs go further, categorizing information, making links between otherwise unconnected documents
For any google results
"Category" is shown right on top of the results.
"Links" - try link:slashdot.org & related:slashdot.org as google queries.
If someone is doing research on computer modeling, for example, it not only knows to discard documents about fashion models but can also extract important phrases, terms, names and locations
Try the google advanced search you can search with "all of these words" and "without these words".
Google shows dictionary definitions for every searched term if they exist. There's also already a location based search and a
phone book
Finally as the article says it all comes down to asking the right question.
I agree about the quality of recent stories. I use to look forward to refreshing the main page all day and seeing an interesting story pop-up every hour or so, one that generates a couple of hundred comments and several deeply-nested threads.
Now I refresh and see a review of a pirate book with ~70 "+2" comments and "Third Anniversary of Bezos-Backed Patent Reform," which went completely ignored. Meh.
Of course, I'm not helping by posting near-useless comments like this...
Hey anyone else think this picture was really cool?
I have been noticing as well, I've not had modpoints for quite some time, and most of my posts have been modde redundant for some reason, when I feel they're not. No major problems, I usually meta-moderate 10 posts, but yes there is some weird issues.
Note to self: get smarter troll to guard door.
May I offer that computers make no sense of what they are reading & that "smart people with knowledge of the particular subject" aren't optional if the results of text-mining are to be of any usefulness whatsoever, at least in any kind of reasonable time frame.
Otherwise, the text-mining computer is playing the old "99 monkeys with typewriters" game...
"Obviously, I'm not an IBM computer any more than I'm an ashtray" (Bob Dylan)
Why does every story linking to a New York Times article have people such as yourself complaining about it? Your comment lends nothing to the discussion and identical sentiments have been expressed countless times. Write a journal entry if you want mindless discourse regarding the New York Times registration requirement. Complain to the Times and tell them what horrible, horrible people they are for making you take ten seconds of your time to provide them with false information.
The fact is that the New York Times often has excellent content and they do not syndicate. This article isn't available elsewhere, but it's worthy of discussion, so either register and read it or do not participate in the discussuion.
I find it hard to believe that you're too lazy to fill out their registration form but somehow found the energy to come here and complain.
--
the strongest word is still the word "free"
And naturally, the few mods that are around have basically wasted a dozen or so points by methodically modding this entire thread Off-Topic. Not that I mind, but for Pete's sake, there's not a single +5 Mod in this topic yet! There's only 4 "+3"s posts! At least try to be constructive, for crying-out-loud! =P
Something tells me at least six /.ers are already working on the case mods :-D
"Obviously, I'm not an IBM computer any more than I'm an ashtray" (Bob Dylan)
Like in the complex plots found in comic books and sci-fi? I'm sure you need a lot of help.
Do you have to where a minning hat:p
-Seriv
And why should you not have to register for information that someone worked for? Freeloader.
BOO! TERRO
Mining for data that might be related based on proximity, either temporal or locational, starts to get interesting when you are dealing with millions of interactions like in a call center on voice data (check out www.callminer.com) and suddenly you find out that when a customer says "hurricane" in an insurance call center, your agents are 5x more likely to hand them off to a supervisor, is real money saving information. This is what this technology is good for, and is being bought and used by a lot of companies hoping to find out that kind of information and save money by training those agents to handle those calls better.
never did this before
/. comment that makes sense
MOD PARENT UP (and me down i don't care)
finally a
Slashdot is a business. If they started using Google's partner link, the NYT would hand their asses to them in court.
Perhaps Slashdot should get in touch with the NYT and see if they can get a partnership set up, but stealing someone else's wouldn't be such a hot idea.
--
the strongest word is still the word "free"
The Slashdot Text Mining Challenge:
Translate into plain English, the never-ending
Bushisms.
Thank you and have an Ashcroft-free weekend,
W00t
i always wondered about this
allright, you can take huge amounts of text and apply some smart tricks to extract patterns from it.
but how can you determine whether the original data was trustworthy?
take the example of genome annotation (description of gene function), which would be helped greatly by including more functional descriptions from scientific literature. how do you determine whether the original publication was backed by solid experimental research?
by the reviewers of the articles? i don't think so, peer review is a snakepit filled with politics. by the amount of people who cited it? hmmmm... so hip subjects are more true?
me personally, because i'm experienced, can recognise bullshit articles when i see them. but how to translate this into an algorithm... anyone any ideas about this? or even working solutions?
(of course this is an example from my field of expertise - biology, but it applies to any set of text data/articles IMO)
GOAT!
(1)"Of course, no one, Dr. Liebman included, is arguing that these products are actually reading anything. What they are engaged in is "text mining,'' "
... Just a thought.
Dijkstra once said "The question of whether computers can think is like the question of whether submarines can swim."
(2)As noted in the article sarcasm is very hard to detect. If you think about it even many people have a hard time recognizing it. How are we supposed to develop an intelligent system when we "intelligent" humans don't even "get" it?
(3)"There is a need to make these technologies available for publicly available information," he wrote at his site.
Yes of course. Anyone who has done research knows how frustrating it is to read through abstract after abstract, let alone the entire publication, to find what you are looking for. In research when you are looking for facts or raw information text mining seems highly promising. Yet, for interpretive processes it grows increasingly difficult to envision a correct system. As noted nuances are difficult to detect. In addition to sarcasm, words like "still" allow for multiple meanings for the bigrams, trigrams, etc. to which they belong. Natural language ambiguity is the most important problem to overcome in NLP. After all, how would you like to write a printf statement and not know whether you would get the intended output or some other arbitrary call.
sig
The compelling dream is that you laboriously load up a computer with enough facts so that it can glean understanding of what it's reading, and one glorious day the computer has enough smarts to make sense of things on its own, and two weeks after crawling the entire Internet, it knows everything.
Hence Doug Lenat's Cyc, now partly open source. Unfortunately that glorious day has been "a few years away" for over 13 years.
But I haven't come across any postings from Cyc on Slashdot correcting misinformation and lies.
Clearly this is possible because all those darn human kids do it; maybe you have to use a more complex computer and leave it for a few years crawling on the floor putting things in its mouth.
=S
How about a computer parsable language like lojban? You can't polish a turd, and you can't systematically extract information from English.
-Libertarian secular transhumanist
That sounds painful!
You know how those guys at MIT are constantly trying to figure out ways to teach their robots how to interact with people? Let the robots roam the Internet with a topic in mind. If I'm at a party with a bunch of dog groomers I'm probably not going to say much. I'm sure robots have the same issue; they have nothing in common with us. If we start by making a Cancer-Expert-Bot then let it try to have a conversation with an oncologist I think AI will have more success.
What if Digg added local news and a Slashdot inspired comment karma system? ---
http://houndwire.com
They just had to get it in somehow
like the 858-page report on the congressional inquiry into intelligence failures regarding the Sept. 11, 2001, terrorist attacks.
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
I gave up a few years ago, now I mostly use statiscal approaches (markov processes, word counts, huge databases of proper names, etc.)
-Mark
NYT won't be contributing to this large body of text, because registration is STILL required.
These are my friends, See how they glisten. See this one shine, how he smiles in the light.
skimming large volumes of miscellaneous text to extract some sort of refined knowledge from it.
/. for years. Guess /. would be the equivalent of open source text mining.
You mean people are getting paid for this? And I have been doing this for free on
skimming large volumes of miscellaneous text to extract some sort of refined knowledge from it.
I like to call it High School.
but could a person who had sufficient knowledge of the program(s) build a large document (say, the giant 9/11 intelligence-failure doc mentioned in the article) so as to fool the text-miners? Subtle misinformation -- let's say that widespread use of text miners results in larger docs being published, then unscrupulous types bury information in such a way that a ridiculously long human endeavour will turn them up, but the programs won't, so those responsible can say: "See? It's all in there. Your program is at fault."
MS Word has a surprising "summary" feature that has given me impressive results in portuguese. How the hell do they do that?
Do you Americans have other newspapers online WITHOUT the free registration requirement?
Is it "free" as in "I give my personal data to them and don't have to handle them money to keep it" or "free" as in "I have the inalienable right to store my data with them?
My cable lineup includes a science channel which has what I would consider to be at least halfway decent shows -- the problem is that even on this channel, and even though there have to be 3 or 4 channels a-la History Channel, the shows that dominate are about archeology. Now I won't pan archeology as "not a science" of course, but people who want to watch science shows generally get bored with that incredibly fast. Also, though I appreciate historical background of scientific thought, too much of that. Not enough theory. TV is a terrific medium and could work very well to convey theory to the public, but I just don't see it being done since documentary makers go for the easy route of regailing us with tales of the past.
I, along with most other Slashdot (adult) readers, am tired of morons like this whining about having to sign up for the NYTimes -- particularly since the whining usually comes from people who are REGISTERED USERS of freakin' Slashdot!!!
But that aside, what in the hell is "insightful" about this shit? He's not the first idiot to spout this immature little complaint. He's the ten-millionth. EVERY fucking time a NYTimes story shows up, one of these dolts (always toting a latecomer's userID) feels compelled to complain. If it was ever insightful -- which I don't think it was -- surely it ceased to be so YEARS AGO.
So I repeat, and I plead: STOP MODDING THIS SHIT. Or at least, use the "Troll" or "Flamebait" options. (Although you should always try to mod up, rather than down...blah, blah, blah.)
Taco, et. al, can simply enter in the article into Google News and get a NYT link that does NOT require registration.
For Fuck's sake, why? Just register at NYT and get on with your meaningless life.
Pretentious git.
The knowledge discovery and datamining cup challenge this year was looking at the arxiv.org papers for this sort of analysis - some very interesting results. The Task 4 winnder looked at the structure of the papers as a sort of relational database and uncovered a lot of statistical patterns and metrics that could be quite useful for scientists.
Energy: time to change the picture.
"I was an FBI agent for 20 years," said Randall Murch, now a researcher at the Institute for Defense Analyses, which works for the Office of the Defense Secretary and other government agencies. "And I have yet to see anyone who is able to model the way an agent thinks and works through an investigation."
Apart from suggesting the jibe that, of course, only an ex-fbi dick could think that anyone would want to model his/her behaviour, this misses the point that text-mining is intended to find precisely those connections which are too weak to attract human attention. A human being approaches an investigation with preconceptions that can colour their findings powerfully. The power of text mining lies in the fact that is non-human and stupid. Software doesn't get tired and is very fast. Attempts to make software "smarter" are misguided.
Doesn't it bother anyone that copying the article is probably illegal?
If you don't understand the benefit of maintaining pressure against something on a well-known public third-party site as opposed to sending complaints direct into a black hole then you don't understand how the world works.
Micheal Moorcock has been text-mining the multiverse for decades now.
One line blog. I hear that they're called Twitters now.
that the whining cretins are really vastly intelligent creatures railing against the machine? What a relie.
Any preoccupation with ideas of what is right or wrong in conduct shows an arrested intellectual development. (Wilde)
think before they use a hammer. Using software to fix problems that exist within their human intelligence arena is soooo typical. The bit about subrogation is so idiotic, I can't believe it. Any idiot can check a box on the report if there is a basis for subrogation. If there is enough data in the report to determine a basis for subro. then the adjuster obviously knew that it should have been handed to the subro. dept. from the outset. There is obviously an issue here. The adjusters are reluctant to send claims to subrogation. Why? Maybe they got yelled at for making work for the lawyers who are paid much more than adjusters and they need to be able to blame it on the computers.
Any preoccupation with ideas of what is right or wrong in conduct shows an arrested intellectual development. (Wilde)
At dinner last night, my friend asked how I could justify spending a lot of time putting in massive amounts of information on a project. I told him that's what students are for! (wink wink, nudge)
Any preoccupation with ideas of what is right or wrong in conduct shows an arrested intellectual development. (Wilde)
...short for "Marvel Multiverse". Text-mining all the comic books in existence to find out which timelines conflict with the others would be an excellent research project.
8-PP
I've been getting 500 errors almost every few pageloads. Isn't anyone else getting the 500 errors?