Slashdot Mirror
Open Source Natural Language Processing?
fieldmethods asks: "One area where Open Source and Free Software doesn't seem to have really taken off is Natural Language Processing (using computers to deal with human languages). There are a few projects that are open source, such as Festival (a speech synth system, now ported to Java), NLTK, a general-purpose NLP system in Python, and the Linguana project, a Perl implementation of a semantic network not unlike Wordnet (but better). Generally, though, there doesn't seem to be a lot of Open Source momentum behind the field as a whole. It's a challenging, difficult field that would benefit from collaboration, especially given the potential of replacing static corpora with on-the-fly corpora developed by search engines. Is anybody else interested in this?"
Actually, I am interested in this. I did some computational linuguistics work while I was doing my BSc/MSc and really enjoyed it.
You should also have mentioned your [?] interesting website fieldmethods.net as a good source for exploring all things NLP [which I thought referred to Neuro-Linguisitc Programming when I first saw it...].
If machines that attempt the Turing Test count as NLP, then NLP is a solved problem. You just need a random number generator to choose from a list of prechosen responses (face it, there's nothing less believable than talking to someone who actually listens to you.) Therefore, I submit Virtual Boglin:
#include
void main() {
int i = 1;
printf("Hello\n");
while(i) {
scanf();
switch(i){
case 1: printf("Microsoft Sucks! Use Linux!\n");break;
case 2: printf("I need to boot back over to the Windows side to play System Shock 2.\n");break;
case 3: printf("Sony is an evil monster who won't be content until we have lost all our rights.\n");break;
case 4: printf("Have I shown you my Clie? Look, it can play the Spiderman Trailer!\n");
}
i=rand()%5;
}
printf("Leave me alone; I'm about to get a new high score.\n");
}
There's a huge amount of open-source NLP resources and software for many languages on the web.
Last but not least:
Will.
Why oil price increase equals economic trouble (Score: Interesti
How is Linguana better than WordNet when they haven't actually done anything?
...the people that are knowledgeable in this field enjoy getting paid for their work.
http://www.speech.cs.cmu.edu/
There are probably others ( search google.com, freshmeat.net, sourceforge.net )
Based on upvotes, Ageism is the only "-ism" Slashdotters care about and think isn't SJW
The POESIA (an opensource internet content filter, partly funded by the European Commission, safer Internet Access Plan IAP2117/27572) project will have some opensourced NLP components (for English, Spanish, Italian...).
See POESIA site for details.
POESIA (Public Opensource Environment for a Safer Internet Access) aims to protect European youth (in educational institutions) against harmful or inappropriate Internet content, and use several techniques (including NLP, Image processing, ...) to achieve this goal.
Judging by the responses to this post (or rather the lack thereof), NLP is not a very hot topic. Most of natural language processing research is in a very academic stage. Quite some universities study some NLP related small little subtopic, but there are hardly any real large departments - say the size of a computer science faculty.
With Lernhout & Hauspie - the one major commercial software supplier in this field - gone bankrupt, there are only some small companies, trying to get by. Some have success in a very specialized sub-subject, like OCR, voice response or information retrieval.
As a former Computational Linguistics student, I'd say the main problem is either the lack of computational power or the lack of manual labour. Ie.: even a very well defined liguistic area needs to be defined with too many rules (in a complex system) or needs too much data and CPU time (in a brute force) to be feaible, commercially viable, interesting in the Turing-sense... too much effort to just make it work.
Where you'd expect high-level NLP to work, simpler techniques usually work better. Ask Jeeves and Q-go are great, but most people agree no search engine currently beats Google, even when it's taylored for a very small subject. NLP is just way immature, compared to most other computational topics - primarily because it is intrinsically complicated.
I guess we'll have to wait for the killer-app for another decenium or two, though I'm a pessimist. Until that time I'd agree all institutions to collaborate as much as possible, and I really don't understand where some universities are going with their closed source research projects.
--
Recursive: Adj. See Recursive
> [...] - primarily because it is intrinsically complicated.
Yeah right. And databases, math and user interfaces are not?
" Is anybody else interested in this?"
judging from the lack of comments on this story, i'd say the answer is a resounding "No."
One of the best speech understanding systems in existance is OpenCyc - and it is open source!
/..sig file not found - permission denied.
http://opennlp.sourceforge.net
http://nlpfarm.sourceforge.net
If you're looking for speech software there isn't that much good software as open source, since just about every aspect of modern speech processing is patented.
As a former Computational Linguistics student, I'd say the main problem is either the lack of computational power or the lack of manual labour. Ie.: even a very well defined liguistic area needs to be defined with too many rules (in a complex system) or needs too much data and CPU time (in a brute force) to be feaible, commercially viable, interesting in the Turing-sense... too much effort to just make it work.
This is a very popular opinion, conirmed by the amount of money corporations through at manual processes like taxonomy maintenance and training, but it's out of date. There are several scalable, commercially viable approaches that do not require manual labor or prohivitive processing to be feasible.
Google News is an example of a large-scale application built with automated NLP. Think Tank 23 makes a NLP-based, ad-hoc categorization engine that powers, among other sites, the Waypath Project.
Talk about a killer app!
Neuro-Linguistic Programming is really different. It's actually something of a sore spot, I'd say, within the Natural Language Processing world, that this other field, which overlaps only under the most generous of definitions, has swiped some of the mindshare of the acronym. To be honest, the NeuroLinguistic Programming thing seems a little iffy to me. But I'll withhold judgement. The bottom line is that its goals are quite different from those of Computational Linguistics. (Also: I'm glad you think the site is interesting. It is my site, but I didn't want to shamelessly promote it.)
I'm not sure about ThinkTank23, but isn't Google news maintained by human droids? IMHO there's some really interesting NLP research going on at Google, but its all very pragmatic: focus is on assisting manual labor and getting proven, but simple, techniques to work on larger and larger problems.
Nothing wrong with being pragmatic, but in the spectrum of unsupevised-NLP to automatic-aids-for-jobs-that-humans-find-tedious, only the left-most extreme is a cheap solution in the long run. And it's these unsuperviced techniques that just don't take off. Persoanlly, I tend to agree with the popular opinion that this will take a very long time - mainly because NLs are just intrinsically very complex, compared to the problems faced by mainstream computer science.
But I certainly agree that NLP research has contributed to some killer apps.
--
If pro is the opposite of con, what is the opposite of progress?
Hi,
my emdros text database engine is built specificially for storing and retrieving annotated or analyzed text. This makes it ideal as a back-end for certain classes of NLP projects.
Ulrik Petersen
Database engine for analyzed or annotated text