Slashdot Mirror

I'd love to, but the core of it was licensed. It was a context-aware full text semantic search engine called Metamorph, produced by Thunderstone, which has since then been built into some other commercial tools. I think they may have a free-for-non-commercial-use non-open version of some sort. It really was quite a cool engine and probably still is. I haven't looked into it for a number of years.

Basically my product used the Metamorph search engine to spin through all the mail files, and using the NeXTstep object system was able to link right into the Mail app and display the email in question, with highlighting. It could run through ten thousand emails (a lot for the early 1990s) in a second or two, on a 25 MHz NeXTstation. So it was basically a relatively simple user interface (although tuning that to make it dirt simple to use and powerful was an interesting project), a driver for the search engine, and of course the packaging into an installable app. But the NeXT system made that part pretty straightforward.

I suppose that by now someone must have built an equivalent full text semantic search engine. As I recall it was originally based on a huge semantic network, constructed by 'reading' various dictionaries, thesauri and other sources. Bart Richards, the inventor (and one of the smartest people I've ever known) originally built Metamorph as a tool for processing research literature and looking for papers relevant to his interest, which (IIRC) was scanning tunneling electron microscopes. But I believe it was later used by the CIA for filtering newspapers around the world, and as of a few years ago was still used by some of the big legal/academic reference databases.

Context-aware semantic search is cool - I haven't seen any other tool that matches it in usability, at least for me. You could look for 'Congress' and it (if you wanted it to) would find both 'Representative' and 'Senator' - or vice versa. I suppose that, given an updated semantic network, you could look for 'geek' and find 'programmer'. :)

+1 on the database. If you can get the data into some sort of mail server temporarily, you can use procmail to parse the mail headers and generate SQL insertions. There's probably something newer - I used this method in 1998 to parse incoming mail from a remote server, that sent status updates every hour.

Mail headers are not that difficult, so if you can get the data into a few standard formats (I don't know about the Outlook formats), you could even do this with a scripting language of your choice, directly from the file. Procmail is nice because it's very good about splitting the mail at the correct points. But, like I said, there's probably newer tools.

In the database you only need fields for (off the top of my head) Date Sent, Date Delivered, To, From, Subject, All Headers, Body and Attachments, plus probably one separate table for the raw data with the same indices so you can augment it later with stuff like mail ID and threading, etc. Then run a Full Text index on the body and subject. You could get fancy with separate tables for all the different To and From, etc.

In the very early 1990s I built and sold a tool for the NeXT called MailQuery, which combined NeXTMail with a 'context aware full text semantic search engine' called Metamorph - presently part of the Texis text search system. That was cool. It was phenomenally good at letting you type in key words that were related to what you were looking for, and finding exactly the right email. You didn't have to remember the exact words - just the ideas, more or less.

...but there are people who know how to actually work the knowledge contained in newsgroups, and these people will be be severely obstructed.

I subscribe to a number of newsgroups covering many technical subjects I'm very interested in. Using just the Microsoft newsgroups as an example, I have scripts that allow me to input keywords and conceptual ideas, search all the newsgroups simultaneously, and then present the relevant posts and threads prioritized by content. This cuts my browsing for relevant info by about 90%. (This saved me weeks of time on Siverlight and Expression alone!) Even the web-based MS Groups search system is faster than any tools I've seen in the forums.

My alternative seems to be to visit each individual forum separately, even if I do a forum search from the MS websites. Crap! what a time-waster. I expect I'll have to spend a lot of time browsing rather than getting relevant results. Until companies start implementing more OWL and semantic web enhancements, forums seem to be largely deficient compared to newsgroups.

BTW, I use a Bayesian filter for spam filtering, a Bayesian search function for first-level search, and a Neural Net for second-level relevancy. I almost NEVER have to contend with spam. (I wrote my own NN, but I got the idea from Logic Line by Thunderstone. http://www.thunderstone.com/texis/site/pages/ )

I hope MS reconsiders.

Check out Thunderstone. It's what they do, and they do it very well.

But there's more!

...already supports this (you most often see it in a free search engine called Webinator). It's the search db behind Dogpile, some (all?) of Ebay, parts of ZDNet, and a whole bunch of other stuff. Not cheap by any stretch but solid.

Check it out: http://www.thunderstone.com/

Most CMSs actually don't really bother about searching via their internal datastores, mostly because of the problems you've raised above.

What you tend to get are two (not mutually exclusive) approaches:

1. Spidering of content using a packaged solution from vendors such as Verity, Autonomy or Thunderstone
2. Internal datastore searching of taxonomy - metadata about how the content is organised. This is the harder one to crack as an effective taxonomy takes a looooong time to get right for a half-way complex site.

Sidenote - you can get around those query string URLs with most serverside scripting environments (as well as that PHP tutorial, it's worth bearing in mind that evolt's site is built in a similar way using Cold Fusion)

This was my final year project thesis. Just remember the golden rule unstructured 2 structured == convert 2 XML I wrote a [very bad] program in C++/Perl/tcsh IPC=pipes to add XML tags to English, and then index them into a search engine which would use the lingual data stored in the XML tags to help the search.

NIST does a MASSIVE competition on this annually. I don't want to be an XML-buzzword whore <Arnold Schwarzenegger accent> (XML commando eats Green berets, C++, Java, Perl, COBOL for breakfast)</Arnold Schwarzenegger accent> but you can't beat XML for easily converting anything that you can make sense out of into computer readable format. Real h3cKoRs use SGML, but us underlings have to stick with things we can understand like XML. As for expandability, if we want to encode something else into the document, then just tag-it-and-go

It took me 200 hours to fish out all these links (before the Google days), I don't want anyone to have to waste as much time as I did feeding the search engines exotic foods. It's a year old so pardon me for the odd broken link, armed with these you could probably turn jello into XML ;-)

My favourite bookmarx
PROJect[21 links]
Beginners' Guide[13 links]
Berkeley Linguistics Dept. Course Summaries, general stuffzzzzzzzzzzzzzzCryptic IR Vocabulary defined
Explanations of weird words like hypernym zzzzzzzzzzzzzzHow do we produce and understand speech
How Inverted Files are Created - Univeristy of Berkeley zzzzzzzzzzzzzzNLP Univ. of Indiana, very good basics e.g. word sense d
Simple langauge - useful.... zzzzzzzzzzzzzzWhat is Natural Language Processing, links
What is POS tagging........ zzzzzzzzzzzzzzWord Sense Disambiguation defined
Word Sense Disambiguation in detail, scroll down far zzzzzzzzzzzzzzWord Sense Disambiguator - LOLITA (tested at MUC-7 and SENSEVAL competition as best)
XML for the absolute beginner

HTML, XML stuff + parsers[19 links]
Apache plug-in that uhhh does stuff with XML zzzzzzzzzzzzzzConvert COM to XML
convert XML, HTML to Unix pipeable formats zzzzzzzzzzzzzzconverters to and from HTML
expat XML parser zzzzzzzzzzzzzzHTML Tidy - converts HTML 2 XML + source code!!
Parse DB (RDBMS, whatever) to XML zzzzzzzzzzzzzzPerl-XML Module List
PHP Manual XML parser functions - what the hell are they talking about, PHP Virtual M... zzzzzzzzzzzzzzPublic SGML-XML Software
Pyxie - XML Processor for Python, Perl, etc. zzzzzzzzzzzzzzSGML+XML tools.org
The XML Resource Centre - massive number of links zzzzzzzzzzzzzzW4F wrapper - wrapper converts XML to HTML
XFlat - convert flat file into XML zzzzzzzzzzzzzzXML Parsers and other XML stuff
XML.com - Parsers, etc. zzzzzzzzzzzzzzXML-Data Catalog System - uhhhh looks close
XTAL's general converter - convert anything 2 XML

other Background[8 links]
Is Linux ready for the Enterprise, scalable... zzzzzzzzzzzzzzLinux reliability
Linux Versus Windows NT, Mark(sysinternals bloke) zzzzzzzzzzzzzzPC reliability (pcworld)
SPEC - Standard Performance Evaluation Corp. zzzzzzzzzzzzzzSystems benchmarks
TPC - Transaction Processing Performance Council zzzzzzzzzzzzzzUnix Beats Back NT In EDA Workstation Arena
Proper TREC(-8) QA systems[2 links]

pg. 387 LIMSI-CNRS pretty deep parsing[2 links]
More links....
NLP, IR links - lots to corpii, etc.

pg. 575 U. of Ottawa and NRL (shit system, got 0%)[1 links]
LAKE Lab
pg. 607! University of Sheffield (crap system, but OPEN SOURCE!)[2 links]
GATE - FREE IE app w`source code
LaSIE - ER, coreference, template (cv)

pg. 617 Univ of Surrey (inconclusive matches)[2 links]
System Quirk - Or is this their search system..... Hmmmmmm
Univ of Surrey - pointers (hopefully this is their WILDER search system...)

SMU - Pg. 65[1 links]
Natural Language Processing Laboratory at SMU

Textract[2 links]
Cymfony - Technology
Textract - State of the Art Information Extraction

Xerox uhhhhh maybe[1 links]
Xerox Palo Alto Research Center
(OVERVIEW) 1999 TREC-8 Q&A Track Home Page
NLP bloke, Univ Sussex


Tcl-Tk[4 links] Tcl tutorial
Tcl-Tk Contributed Programs Index
Tcl-Tk Resources, sources
TclXML - manipulating XML using Tcl-Tk
Artificial Natural Language - Is this what I'm trying to parse into...
Comparison of Indexers - Prise vs. Inquery vs. MG, etc.
Eagles - Language Engineering Standards
Language Technology Group - lots of modules!
LDC - Linguistic Data Consortium, lots of corpora
Lexical Resources
Links 2 resources, indexers.....
Lots of IR stuff, University of uhhh
Managing Gigabytes Indexer
Managing Gigabytes Manuals and stuff
Htdig search system
NLP & IR (NLPIR, NIST) Group
OVERVIEW OF MUC-7-MET-2
Perl XML Indexing - XML search engine type thing
Phrasys Language Processing Software Components (money)
QA HCI bullshit
SIGIR - TREC-type thing, resources
SMART indexer system documentation
Text REtrieval Conference (TREC) Home Page
The Natural Language Software Registry
Thunderstone IE and IR products
WordNet - FREE DOWNLOADABLE lexical English database

Page created with URL+, nice utility for working with internet shortcuts

Thunderstone Texis... web spider, regex scraper and SQL-compliant RDBMS. Oh, and a _tad_ more maturity and market experience. Of course, the price tag is a bit steep...

WebQL seems kinda like a friendlier name for an already-established market.

PDHoss

Your project sounds like a project for Thunderstone's Texis system. I've worked with quite a few other search engines, and for my projects, Texis always beat the other players in performance/flexibility.

1. Keyword/Phrase searching (and,or,must include,must not include,must include N or more, etc.)

2. Prefix and suffix processing

3. Search by regular expressions

4. Concept searching with precompiled thesaurus as well as a user-defined thesaurus (you can customize it for your industry's jargon, for example).

5. Integrated P-Code compiled scripting language

6. Available for a bunch of platforms

7. Goes like a bat out of hell on huge indexes

8. Free [beer] version available for smaller projects

Their client list is pretty impressive; Ebay, NASA, ZDNet, etc., and lots of city/state sites. The downside is that it's not cheap: $5000 to $10000 or more just to start.

My only connection to Thunderstone is my prior use of their product.

that Slashdot has resorted to analyzing reports and trying to improve Linux rather than posting baseless flames along with flagrant use of the term "M$".

I guess the serious readers have been there all along but like me, were driven out of the shadows by the ridiculous behavior towards MindCraft.

If you are still having trouble understanding Linux has a design problem, try this little exercise. On Slashdot, go to the search box down there at the bottom and enter something in. When the results come back, select to search in the comments and time how long the search takes. I seem to get results back in about 10 seconds. Now, go to search.thunderstone.com: a web index similar to AltaVista which is driven by their SQL server Texis. Searches on here come back in less than one second, even though they are searching much more data (350 gigs in Oct. 98).

MySQL simply is not designed to efficently search through text so it is slower. Just like Linux does not have the multi-threaded TCP stack and so it runs slower. Flames won't help at all.

Now, for the offtopic part.. Are there any good SQL servers out there (DBI/DBD support is a huge plus) with good text searching abilities? One website I run on MySQL has the same problem with long searches. We looked at Texis but the license is simply too expensive. They base it on transactions/day and rows in the database, going from a couple thousand all the way up to $12.9 mil. Just to convert over, it would have cost us about $50,000. On the plus side, they do support Linux!

that Slashdot has resorted to analyzing reports and trying to improve Linux rather than posting baseless flames along with flagrant use of the term "M$".

I guess the serious readers have been there all along but like me, were driven out of the shadows by the ridiculous behavior towards MindCraft.

If you are still having trouble understanding Linux has a design problem, try this little exercise. On Slashdot, go to the search box down there at the bottom and enter something in. When the results come back, select to search in the comments and time how long the search takes. I seem to get results back in about 10 seconds. Now, go to search.thunderstone.com: a web index similar to AltaVista which is driven by their SQL server Texis. Searches on here come back in less than one second, even though they are searching much more data (350 gigs in Oct. 98).

MySQL simply is not designed to efficently search through text so it is slower. Just like Linux does not have the multi-threaded TCP stack and so it runs slower. Flames won't help at all.

Now, for the offtopic part.. Are there any good SQL servers out there (DBI/DBD support is a huge plus) with good text searching abilities? One website I run on MySQL has the same problem with long searches. We looked at Texis but the license is simply too expensive. They base it on transactions/day and rows in the database, going from a couple thousand all the way up to $12.9 mil. Just to convert over, it would have cost us about $50,000. On the plus side, they do support Linux!