Slashdot Mirror

I refer to the Robot.txt used to tell search engines what's out of bounds. http://www.searchtools.com/rob...

There're also things like the Open Directory and the software listed here. However, I imagine it is hard to really break into the search engine game without lots of money!

Rather than "Starter Edition," here's some suggestions, if anyone from Redmond just happens to read this. (I know they won't do it - it's more a mental exercise while I eat)

1. Go download this, and make it natively multi-user if it isn't already. Give it a strong native security model, too...you can get some ideas here, and the best part is, they won't mind you doing that if you don't try and patent said ideas. Also, modularise your GUI, and don't prevent users from accessing the CLI when they want to.

2. Have the CLI composed of this and this for us CLI types.

3. Make the Add/Remove Programs panel essentially a net-aware frontend for either this or this.

4. Use this for hardware detection. Also re drivers, get rid of the suicidal policy of seeing third-party hardware vendors as the enemy, and actually support them...via tools, docs, etc. These people are your friends...they'll help you stay relevant.

5. Download this and use it as your default FS, and then get this and this, (although you already seem to know about this last one) and incorporate both of those into your stock UI. You've essentially got WinFS right there, without all the added complexity you'd no doubt throw into it if you tried to code it from scratch.

6. For the Agent angle, incorporate the last point, as well as putting help/docs in a non-binary format, making them searchable with this, converting said search results for use with this, and then use the AIML output as input for something like this. Also, instead of making the agent a tightly anthropomorphic personality, make it more generic, and more as though it's simply "the operating system" communicating with a user, rather than that dog or Clippit instead.

7. Give Outlook a major overhaul. This and this are examples of directions it IMHO should go in.

Just some random ideas, anywayz. Dreaming's fun. ;) I'll probably get modded Offtopic, but it was worth it.

This is a very pretty demo and I am looking forward to digesting the linked papers. At first glance it seems not so interesting since the functionality provided by the demo would seem to be reproduceable with a simple sql engine.

What I still am not so sure about (and is why I want to read the articles and the code I downloaded - THANKS!) is the following perceptions I had (possibly erroneous) on first glance:

- It is not clear how an semantic web ontology is being used, presumably there is rdf with some rules about e.g. what period is baroque and then some other semantic rules are used as a guide on how to organize the interface (or I suppose you would say along which dimensions to slice the whole n-dimensional mishmash and in what order).

- Having been quite interested in faceted metadata search/navigation engines, I pounce upon seeming drawback of this presentation, in that it forces a hierarchy and browsing direction on a user, whereas it is difficult to discover information laterally starting from a leaf node (basically it feels like you are viewing yahoo a few levels in advance). And also that you don't know how many items are available in a set until you click, though I suppose the "mouse hover and wait" function is supposed to solve that, but it doesn't.

- wrecked keyboard-based advancement of slashdot thread using space bar after viewing in firefox, but could just be some memory flakiness on my machine I suppose.

- why not use perl and a js-based dhtml module instead of php? (since I like perl and also because it would be nice to have programmatic access to your semantically informed rdf browser, maybe)

- relationship of this philosophically and scientifically to faceted metadata browsers such as flamenco (to become OSS we are told) and the work of companies like Endeca, Siderean, etc.

- Why is the user not told about the semantic rules being used? Wouldn't that help inform exploration of a subject?

Otherwise it is an interesting project and I wish you luck.

Matt Rosin

No need to write one from scratch, there are plenty out there including some not on the list. Some of these are quite customizable, you can prune various servers, directories or file types from indexing. It's even possible to custom pre-processing, for example getting rid of all navigation menus identified by 'class' from the index. At the low end- there's even Swish and htdig

If you're a sucker for punishment you can even front end one of the higher end search engines with other protocols. For example, Z39.50 allows search clients like BookWhere, Procite and Endnote to do the search, something which is useful if you have a lot of research documents. Perhaps there is a use for LDAP here, too.

However, no way would it take months to install and configure an existing search engine in its basic form. If you have a machine, it takes 20 minutes to slap Debian (or your favorite Linux or BSD) on it and a few more to install the search engine and its prerequisites. Then you spend the rest of the week reading about it and tweaking it.

No official word yet on whether the company thinks there are any infringers.

There is the Google Search Engine Appliance thing, which must have a healthy profit margin. I doubt that's going to be all that significant a contribution though - so I guess there is more profit in throse Sponsored Links than might be expected.

I agree entirely. The search spammers abused the system and now it's gone -- a tragedy of the commons.

However, metadata on web sites is a powerful and useful tool, so smaller search engines can take advantage of it even though the web-wide ones can't trust anything that's not visible.

There's a new approach to fielded searching called "faceted metadata" search that's really designed for rich metadata systems and well-populated databases, such as online catalogs, recipes, auctions and technical documentation. It shows the applicable metadata fields in context, a dynamic taxonomy. So if you search for "pepper" in a recipe database, you can then navigate based on other ingredients, cusine, holidays and so on.

I think this is great for all those databases where there's tons of information but no easy way to navigate it. I written up a Faceted Metadata backgrounder with some examples. It makes sense, it's usable, I think it's the Next Big Thing in search.

Avi

I'm not a huge fan of PDF in general, and it's tricky to index because the files tend to be big, so you can get word co-occurances where they don't have much to do with one another.

That said, I've found a lot of PDF search engines, my current favorite is PDFWebSearch whichis smart about PDF pages and should handle local file system indexing as well as web site crawling. I have a whole page all about PDF search engines, including links to all the ones I know about at the SearchTools PDF searching report.

If you want to get in touch with the Glimpse people, I have a page about that too.

Avi

I'm not a huge fan of PDF in general, and it's tricky to index because the files tend to be big, so you can get word co-occurances where they don't have much to do with one another.

That said, I've found a lot of PDF search engines, my current favorite is PDFWebSearch whichis smart about PDF pages and should handle local file system indexing as well as web site crawling. I have a whole page all about PDF search engines, including links to all the ones I know about at the SearchTools PDF searching report.

If you want to get in touch with the Glimpse people, I have a page about that too.

Avi

As for building out a Topical or Hierarchical structure like Yahoo's or DMOZ, you need to have meta data about the document. You can pull meta data from a URL, as was suggested earlier, but I wouldn't advise that. In order for this to work, your URL's start to look like 'http://city.gov/CityCouncil/MinutesOfMeeting/2000 /10/15/index.htm', which also makes it difficult to move things around, and makes it difficult to return the string 'CityCouncil', properly formatted when you build the hierarchy on the fly. If you include the hierarchy information in the page, you can get around some of these problems. I think the standard way to do this would be using meta tags. View the source of this page for an example of this. The downside to this is that you have to structure your hierarchy in advance, which goes back to getting & building requirements early, you also have to convert old documents to include these tags, and you have to make sure that future documents will have these tags properly implemented. If you're using URL paths to qualify the documents, you may be able to take advantage of an already existing directory structure to build your heirarchy. I just don't like using document locations to figure out what a document is about--its always seemed like a brittle solution to me. On the other hand, I've used the URL string to help make a first pass at placing meta tags into documents, usually using a Perl script. I still had to go back in and check the documents though.

I've used both FreeWAIS-SF and Verity for implementing searches like this, as well as home-grown solutions. I wouldn't advise using Verity since I think that it's prohibitively expensive, unless it has some feature that you require and are willing to pay for it. And I didn't think the home-grown solutions worked as well as the off the shelf products that were customized.

The default (no robots.txt) is to crawl your site. If you have a robots.txt, it follows the rules therein.
http://www.searchtools.com/robots /robots-txt.html
List of rules - found with google. :>

- the IB project at Basis Systeme netzwerk,

- the former commercial products from PLS that AOL is now giving away,

- the ASF project,

- the Webglimpse pages,

- the pages for the mg system.

For a comprehensive presentation on the subject, see the searchtools site.

I'm obviously a bit biased, but there *are* strong, open-sourced search engines. Try ht://Dig for example www.htdig.org or if you don't like that, you should check out the excellent SearchTools.com website. Cheers, -Geoff