On Measuring the Effectiveness of a Search Engine?

younix asks: "I am working at a major telecom provider in the Netherlands and we are implementing a new system for knowledge management. Our system works quite good at the moment, but there is no way to actually measure the effectiveness of the search engine we are using. With effectiveness I mean how much effort do I have to make to find a particular solution that is present in the knowledgebase. Is it possible to express this in a number? Has anyone in the slashdot community done anythin similar before? I would really appreciate any pointers to work done by others on this subject." I think that the single most important aspect of a search engine is the mean time it takes to get useful results out of it. Now, about regarding measuring such a thing (or any other usable characteristic) in an objective manner?...Therein lies the problem.

Comments

Ideas to think about

[duplicate-nickname]

First off, you may be able to determine some "satisfaciton" factor for your search engine, but its meaningless unless compared to the same rating of other search engines. For example, if you creat some rating system on a scale of 1-100, and it rates a 10 as in worthless piece of garbage, a 10 maybe great if all other solutions average under a 5 (becuase search engines suck in general).

The obvious problem is now how do you come up with a rating? The non-tech, and probably most effective, is to survey the users. Ask questions like: Did you find a suitable result? How many times did you refine your search? Where was the result located (1st, 50th, etc)? yadda-yadda.

Another way is to keep track of the average rating of the selected links that the users click on. The search engine probably has a numerical relevance number for each result. When a user clicks a link, store that number. You may also want to store the rank of the link (1st, 50th) link in the survey becuse you may have 50 links with relevance of 99-100%, but the fact that the user has to scroll to link 40 to find what he/she is looking for isn't so great. The problem with both of these methods is how do you know the link the user chose is what they wanted? Maybe it was a totally wrong result and they hit the back button and tried another one. Some session tracking would be in order I would guess. With that, you could track the number of searches a user executes.

Stats would look something like this:
Session 1
=======
Searches: 4
Links followed: 10
Final Link Relavance: 85.4%
Final Link Rank: 3

Use these numbers, compare them to other search engines (along with surveys) and see how you're doing....

ÕÕ

It's all relative.

[grammar nazi]

It all depends upon the person who is searching.

Suppose I want to find out information about the Ikeda Attractor. I do a search on 'Ikeda Attractor' and it results with a few pages with pictures of it, but nothing really describes what it is. From the images I realize that it is a chaotic attractor, so I do a search on 'chaotic attractors'. The 2nd search results with more pages that describe the phenomenon of chaos. These pages, however have links to other pages that describe chaos in general and how to represent chaotic attractors. I have to then follow these links to get to the pages that describe exactly what an attractor is. Many of these include the Ikeda attractor as a specific example. Search completed.

This is similar to any research oriented search. In a university library, the preliminary survey of Scientific Citations Index, or whatever literature search tool that you use rarely turns up good results. These initial results, typically, are only good for directing you to further results via the bibliography (the articles usually tell you which other papers are good for what topic in the introduction). You then get locate the referenced papers and sometimes you look at the references in those. I usually need to read the intros to about 5-6 articles for every good foundation type paper that I come across.

Search engines in general, won't improve any time soon because every search is personal and customized. A little bit of effort and a little bit of experience is all that you need.

Finally, I'm not going to proof read this, since I'm late for work, but I hope it all makes sense.

Humourous aside...

[nick_davison]

The problem with trying to assess efectiveness for users is the old problem that everyone works in different ways - often capable of great idiocy.

I was involved in the Y2K cover for MCI WorldCom's web services. We had constant calls from scared silly customers who'd seen that support for their area was down around 50-60% and were worried they'd lose service.

The reality of what happened was we were using ultraseek, they'd type in a term like ATM, then get the search results page. Next to each search was an accuracy score - it just doesn't say it's an accuracy score and they all interpreted it as a service level score, hence the panic.

So, in conclusion, any metrics you impose are going to end up being subjective and making assumptions of a basic level of intelligence that you just can't assume.