Is The Web Becoming Unsearchable?

wayne writes: "CNN is running a story on web search engines and their inablity to keep up with the growth of the web. Web directories such as Yahoo! and the Open Directory Project can take months to add a site and the queue of unreviewed sites is growing. Most search engines are even further behind and are filled with off-topic and dead pages. The trend is toward pay for listing. Will the free, searchable web fade away?" The article gets beyond the "Wowie, so much content, engines can't keep up" typical blather and addesses some of the reason search engines have a hard time keeping up.

Comments

Google

[citizenc]

I don't know WHAT they are talking about -- I can find ANYTHING that I look for on Google -- even sites that I have just created a day or two ago have been found. These people just aren't using the right search engine, dammit! =)

------------
CitizenC

Directories are not search engines

[ichimunki]

Yahoo and DMOZ are web directories. This is a very human labor intensive way to categorize the web. Google is actually a search engine. It spiders out and runs an indexing algorithm of some sort to help it respond to queries. These are very different approaches.

Yahoo and the like are doomed to failure until someone implements something like the Dewey Decimal System for web pages and then convinces a large number of webmasters to correctly classify their pages using it. That way a machine can do the hard work and only the person designing the page need do the actual work of making sure the page is classified correctly.

Obviously this is fraught with problems similar to those of keyword spamming, but it's either that or build something like DMOZ on a decentralized basis, so that any individual maintainer builds a set of links that are tailored to his/her interests and either uploads them to a central sever or provides them as an XML document for an engine to work with.

The Sky Is Falling, The Sky Is Falling!

[Eoli]

You said the same thing two years ago!