Slashdot Mirror

← Back to Stories (view on slashdot.org)

Google Outlines the Role of Its Human Evaluators

Posted by timothy on from the humans-are-dead-they're-probably-dead dept.

An anonymous reader writes "For many years, Google, on its Explanation of Our Search Results page, claimed that 'a site's ranking in Google's search results is automatically determined by computer algorithms using thousands of factors to calculate a page's relevance to a given query.' Then in May of 2007, that statement changed: 'A site's ranking in Google's search results relies heavily on computer algorithms using thousands of factors to calculate a page's relevance to a given query.' What happened? Google's core search team explain."

8 of 62 comments (clear)

  1. Google is PEOPLE by Kotoku · · Score: 5, Insightful

    In reality this is why search engines like Wolfram Alpha without the broad research and knowledge of Google in the industry don't stand much of a chance unless Google drops the ball.

    --
    AnimePapers.org: Anime Wallpapers Handled With Care
  2. Summary, missing from TFS by Useful+Wheat · · Score: 5, Informative

    Because the summary wasn't kind enough to give you the answer to the question, here it is.

    Human evaluators (mostly college students) are trained in the art of validating a search engine result. They examine the results of their searches, and determine which ones are the most highly relevant. For example, searching for the Olympics should yield information about the 2008 Olympics (or any current one) instead of the 1996 Olympics. The reviewers frequently work on the same query results, that way they can see how consistently the reviewers are rating websites.

    The vast upshot of this, is that it helps weed out those websites that are cheating the system, and trying to get their website as the #1 google hit, so they can show you ads. So the large part of what they are doing is tracking spam websites, not real ones.

    1. Re:Summary, missing from TFS by Ethanol-fueled · · Score: 5, Insightful

      I think having an indefinite human element would be a good thing for Google. College students are reasonably smart and many of them would enjoy doing such a simple thing to make a few bucks on the side for beer or textbook money. It's a lot like Slashdot's mod system. Hopefully it will drastically reduce spam pages being in the top results.

    2. Re:Summary, missing from TFS by morgan_greywolf · · Score: 5, Funny

      It's a lot like Slashdot's mod system.

      Oh, God, I hope not! Searching for jokes, you'll wind up with completely humorless results. Completely irrelevant pages will end up getting to the top of search results because they were "insightful" or "informative", despite being completely offtopic. All anti-Microsoft, anti-Apple and anti-Linux pages will be completely buried in the search results. And searching for "Apple" or "Linux" will result and you'll wind up with in a bunch of fanboy pages....wait a minute (searches Google)....ARRRGGHHHH! The Slashdot Mods have TAKEN OVER!!!! Ummm....ahem....

      I, for one, welcome our new humorless Apple, Linux and Microsoft fanboy overlords!

      --
      My blog
  3. College students? Hmmmm... by Bearhouse · · Score: 5, Funny

    Google, for example, employs a vast team of human search "Quality Raters" ... Spread out around the world, these evaluators, mostly college students, review search returns ....

    Well, THAT explains a lot of what happens when you set 'safe search' to 'off'...

  4. But for how long? by wdsci · · Score: 5, Interesting

    This reminds me of a comment from a friend of mine who works at Google - he says that he's gotten the sense of a company philosophy (unofficial of course) that advocates doing things automatically, without human intervention as much as possible. Basically, they work as though there's an algorithm for everything and it's just a matter of how long it takes us (well, how long it takes them) to produce it and properly refine it. So I wouldn't be surprised if the reliance on human evaluators decreases over time. I bet Google would really like for the original language of their search result explanation to be true, but they've had to make concessions to reality...

  5. Article is somewhat inventive by jpallas · · Score: 5, Insightful
    That part about "mostly college students" comes from the interviewer, not from Google:

    JP: So are these raters college students or random folks responding to a job post? What are the requirements?

    SH: It's a pretty wide range of folks. The job requirements are not super-specific. Essentially, we require a basic level of education, mainly because we need them to be able to communicate back and forth with us, give us comments and things like that in writing.

    Funny how the introduction restates the interviewer's preconception even though the actual interview implies otherwise.

  6. Fuzzy logic is killing Google by Anonymous Coward · · Score: 5, Interesting

    Slightly off-topic: Am I the only one who finds Google web search less and less useful? There's no way to really force literal search anymore. Everything I enter gets auto-"corrected". Plus signs, quotation marks or that misleading field "this exact wording or phrase" in Advanced Search used to help, but that stopped working a while ago. Everything is fuzzified now. Is there an alternative or some trick I haven't heard of?