Slashdot Mirror
Open Source Automated Text Summarization?
TrebleJunkie writes "I've spent some time recenting looking for open source projects dealing with Automated Text Summarization -- automatically generating detailed summaries from longer documents -- to no avail. I can find a lot of research papers and several commercial projects, but no open source code or projects? Does anyone out there know of any?"
It's too bad the Internet didn't keep with the way it was supposed to work back in the beginning with the and such. If that were the case, you could look for headers and body blocks to determine content.
If you know how the text will be formatted you can have the script look for a proliferation of words that are not things such as "a" or "and" or "of" and perhaps search for common threads, but other than that, I got nothing!
is there a non-free program that does this?
I know this isn't exactly what you are looking for, but I remember SAT prep books that teach you to read the first line of every paragraph to get a quick summary. Granted it works better for the SATs than it does IRL, but it often works pretty well and it's better than nothing. You could whip up a simple perl script to extract the first line of each paragraph in no time.
Surely the whole point to summaries is that they are a shortened version of human-generated english (or whatever human language) that embodies the general context of the document.
This just seems like one of those things that:
- Is best not automated
- Probably cheaper done by hiring a clerk to read and summarise, than use a computer
Seriously though, I can't think of any reason why you would really need to automate this. Is there one?Why not provide us with the research you found. Maybe one of us would be willing to hack up a quick and dirty prototype based on the research.
First off, I'm doubtful that there are any open-source programs that do this well, as it's a very difficult problem! It has to do with understanding a document, which computers really can't do.
So I'd like to take a moment to point out a good resource for some existing summaries, at bookaminute.
The HTML::Summary and/or Lingua::EN::Summarize modules probably do what you need.
I know it's not open source, but have you tried the Summarize feature in Microsoft Word? I fed it the entire contents of the GNU website and it came back with:
http://www.mitre.org/technology/alembic-workbench/
Might do exactly what you want. You probably have to train it first but it works quite nicely.
Mike
I think one of the problems is that such a piece of software would be big business. I think I found something in the Natural Language Processing Software Registry: http://registry.dfki.de/ Check under sections->written language->summarization Another poster described systems that simply filter through relevant sentences. They're also sometimes known as abridgers. You might want to include that term in any keyword search you're doing
Try it on man pages:
man awk | perl -ne 'split;foreach(@_){print $_." " if (rand()>.9)}'
and it still makes sense! :)
I developed NetOwl Summarizer 1.5 (way at the bottom), and there's a lot that needs to be done. You need to score enough documents, and need to have a good entity extraction mechanism (which NetOwl Extractor does) and you need a good on-line learning system. It's a lot of work, and even still, we don't get very good results, only good results. Microsoft's text summarizer does far worse, actually, but neither of us is perfect.
From my research, there appears to be two primary methods of performing this kind of processing:
Of the two, statistical parsing is more popular these days because it doesn't require knowledgebase, expert system shells, grammar modeling and extensive dictionary. One of the primary method of determining the relative importance of words in a sentence is valence. The main challenge with natural language parsing and statistical technique is it depends on the training dataset. The more specific the dataset is, the better it will perform.
Statistical analysis can also use expert system shells and other AI technologies to improve accuracy, but it doesn't have to.
From my understanding (which is limited), it stems from a principle from linguistics. By counting the frequency of words or more specifically nouns, the program is able to rate each nouns importance. Once it got done, it could then look at the sentence that best describes the document by doing a comparison between the most importance words and the appearance of those words in the sentences. I remember this from my literature and linguistics classes. Congnitive science has also attempted to solve this problem, but it is very difficult.
In either case, if you dealing with well structured documents, your best bet is to grab the first 3 paragraphs assuming the author followed standard thesis/essay structure. If you're planning on summarizing new articles, it might not be that hard if the author followed the inverted pyramid, which many do not. One of the big tools of natural language parsing in the early days was prolog. It is still used a lot in academic settings for natural language processing. You're best bet is to get an intern to read and summarize for yo
It's not Open but it is scriptable, is not an additional cost, and is available on a Unix OS (MacOS X) Indeed through Apple's Open Scripting Architecture (OSA) one can use any number of scripting languages such as Python, Perl, and even JavaScript to interact with the application.
Feed it a document, tell it to summarize and back will come a generally useful précis. For folks directly on a Mac (MacOS 8.6 or newer incl. X) simply highlight a document or portion of text and select "Summarize" from the contextual menu.
I don't read ACs: If a post isn't worth so much as a nom de plume to its author then I wont bother either.
It's a pretty hard problem; people are still actively researching this and the best results are only so-so.
The best way to find code would be to e-mail the authors of the papers you've found. They probably have implementations, and academics are usually willing to share under something like a BSD license or the GPL.
Are you DevilM from GamesNET?
You're just seeing the problems with the Perl Summarize::Moderate module. Its redundancy detector needs improvement.