Search Engine Results Relatively Fair

perkr writes "The Economist and PhysicsWeb report on a study from Indiana University claiming that search engines have an egalitarian effect that gives new pages a greater chance to be discovered, compared to what would be the case in the absence of search engines. Based on an analysis of Web traffic and topology, this result contradicts the widely held 'Googlearchy' hypothesis according to which search engines amplify the rich-get-richer dynamics of the Web."

Comments

google good

[seanadams.com]

First of all any time you want to analyze Google, you have to realize that they've had ten PhDs crunching the problem already for years. Google is designed to give the best results for whatever its users are searching, thus any apparent bent towards egalitarianism, monopolism, antidisestablishmentarianism, or what-have-you, is purely incidental.

If you're searching for something obscure, Google will instantly tell you the one startup company building it. On the other hand, if you want something mainstream, they'll give you a prioritized list of the best sources. There's no alterior motive it seems - they just give you what you searched for... imagine that! I've seen a business through from obscure geek hack to the mainstream consumer, and Google has been there at every step of the way, working exactly as users expect. To accuse them of favoring any particular stratum of that chain is awfully unfouned IMHO unless there are some specific examples. Indeed, answering users' needs instead of pandering to the status quo seems to be he most valuable bit of what google does.

My experience bears this out also

[Analogue Kid]

I've made sites with fairly mainstream content before, which were totally ignored by google. But then, I put an article on my blog about the history of a certain group of elite English schools in Taiwan. Previously, this information had not been on the internet anywhere. Now, if you type the name of the original school of that group (Modawei) into google, my article comes up #1.

Re:My experience bears this out also

[DeadSea]

I've made sites with fairly mainstream content before, which were totally ignored by Google

That is precisely what the "rich get richer" effect is about. This study seems to be measuring the wrong thing. Of course your mainstream site is going to get a few hits from Google because your site mentions something in some quirky way that other sites don't. However, because there are already 10,000 sites about what you have written, you will never get into the top ten search results. Google puts sites near the front of the SERPs because they have lots of incoming links. Sites that are in the top will get a lot more traffic and some percent of that traffic links to them. Sites at the bottom, get few new incoming links.

Yes those few visitors that you are getting from Google are more visitors than you would get if Google did not exists, but that says nothing about the relative number of visitors that your competitors are getting.

Let's try a thought experiment...

[NickFortune]

Suppose there were no search engines.

Most web newbies would form their impressions of the web from their ISP's portal site. That would give a lot of power to corps like AOL, who for a long time tried to persuade their subscribers that there was no web outside of AOL hosted content.

There might still be blogs and social networking sites, but the take up would be slowed since fewer people wold have heard of them, and both might have failed to ignite into the movement we see today.

Which would probably mean that if you wanted something outside of the main ISP channels, you'd be reduced to digging through the spam on USENET to find it.

Google as an egalitarian influence on the web? I think it's a bit of a no-brainer, personally.

Impact of Search Engines on Page Popularity

[Omar El-Domeiri]

There is another paper out of UCLA that is similar to this one except with somewhat opposing results. In which, the authors show analytically that the rich-get-richer phenomenon does exist. http://oak.cs.ucla.edu/~cho/papers/cho-bias.pdf

It seems tough to reconcile these two sets of findings, and this new paper even makes mention of this:

"The connection between the popularity of a page and its acquisition of new links has led to the well-known rich-get-richer growth paradigm that explains many of the observed topological features of the Web. The present findings, however, show that several non-linear mechanisms involving search engine algorithms and user behavior regulate the popularity of pages. This calls for a new theoretical framework that considers more of the various behavioral and semantic issues that shape the evolution of the Web. How such a framework may yield coherent models that still agree with the Web's observed topological properties is a difficult and important theoretical
challenge."

Re:An example of poor Google performance

[baadger]

Too true. How about some serious search innovation from Google?

- Effective (but switchable) web spam filtering, as parent mentions.
- The ability to search for strings like "-x flags" (note the quotes) and actually get meaningful results.
- More complex patterns (mathematical expressions, anyone?)
- Sort search results by the date pages were modified, they were discovered by Google? (useful in circumstances when you're looking for the latest information on a topic).
- Semantic sensitive search bots.
- Better results for filetype: operator. Why can't Google index all major filetypes even if it can't make them searcheable?

Anyone got any others?

Google could be working constantly behind the scenes on their engine but perhaps they should start making more noise about it. When was the last time Google's web search engine trod some new ground? Or any search engine for that matter (I refer to Google because they are 'innovating' so much).