Slashdot Mirror

← Back to Stories (view on slashdot.org)

"Wisdom of Crowds" Works For Individuals Too

Posted by kdawson on from the I-am-large-I-contain-multitudes dept.

ideonexus writes "Take a crowd of people and have them guess how many jelly beans are in a jar, and the average of their answers will be remarkably accurate. Now researchers have found the same goes for asking one person to guess about the same thing several times. Accuracy improved when the individual was given longer periods of time between guesses." The anonymous author of the Economist piece, not quoting the researchers, says the finding bolsters the "generate and test" model of creative thinking.

37 of 158 comments (clear)

  1. In related news... by morgan_greywolf · · Score: 4, Funny

    In related news, students were found to do far better on multiple choice tests when given an unlimited number of guesses at each question. Even students that didn't study eventually got As.

    --
    My blog
    1. Re:In related news... by smallfries · · Score: 4, Insightful

      Not quite... but you are close. It sounds like you're pointing out that anyone will get lucky if given enough chances. These guys are claiming that the average will converge to the ground truth over time. This would need to have guesses with some Gaussian distribution about the correct answer.

      If the guesses were uniformly distributed then the average wouldn't tend to the correct answer over time. Of course what is described in the summary has nothing to do with the wisdom of crowds as it is commonly thought of (i.e in markets) where shared information is vital. Instead it is simply an artifact of sampling (which is why the longer gaps are necessary for better accuracy)

      --
      Slashdot: where don knuth is an idiot because he cant grasp the awesome power of php
    2. Re:In related news... by bunratty · · Score: 3, Informative

      I think you need another class in statistics. It doesn't matter whether the guesses are normally or uniformly distributed. If the guesses are distributed around the correct value, the average over many guesses will converge to the correct value. All this shows is that when someone makes an estimate, they are usually close, and they overestimate about as much as they underestimate. The average of those guesses will then be more accurate than any one guess selected at random. The guesses probably are normally distributed, but that the fact that the average of the guesses converges to the correct result in itself does not prove that they are.

      --
      What a fool believes, he sees, no wise man has the power to reason away.
    3. Re:In related news... by Falkkin · · Score: 2, Interesting

      Good observation about the Gaussian distribution being necessary. Thought experiment: I am thinking of a number between one and a million. What's the likelihood that the average of a bunch of people's guesses are anywhere near the number I am thinking of?

    4. Re:In related news... by Free+the+Cowards · · Score: 3, Insightful

      You state the really cool thing about this but somehow completely miss it!

      You say, "If the guesses are distributed around the correct value...." Well, why would they be? They're guesses! There's no reason to expect one person's guesses to be centered on the correct value if they don't know the correct value. But this study shows that they are centered near the correct value, even though the person doesn't know what that value is.

      --
      If you mod me Overrated, you are admitting that you have no penis.
    5. Re:In related news... by jibster · · Score: 2, Insightful

      The Gaussian distribution is completely unnecessary. The only necessity for the law of increasing averages to hold is that the distribution is centered on the average.

    6. Re:In related news... by Free+the+Cowards · · Score: 3, Insightful

      That would be a flaw if I ever discussed "a bunch of people", but I never did.

      The interesting thing here is not that the individual can guess a number close to the true value. What's interesting is that if he guesses more than once, the average is closer to the true value than his initial guess. This is unexpected and a little weird.

      --
      If you mod me Overrated, you are admitting that you have no penis.
  2. So how long is the emperor of China's nose? by Watson+Ladd · · Score: 5, Insightful

    Wisdom of crowds only works when the crowd has some information about the situation. Look at polls about Iraqi weapons of mass destruction for more details.

    --
    Inventions have long since reached their limit, and I see no hope for further development.-- Frontinus, 1st cent. AD
    1. Re:So how long is the emperor of China's nose? by zappepcs · · Score: 4, Insightful

      Exactly. Penn and Teller asked a group of people if the chemical Dihydrous Monoxide should be banned. Nearly every one of them said yes. The wisdom of crowds is not in and of itself some sort of magic. It is merely an interesting observation.

      That your own guesses seem to exhibit the same 'average' correctness as a crowd is bad science IMO. Once you guess at a problem, you're subconsciously directed to think of that problem, thus getting more than a knee jerk reactionary guess. The longer you have to think about it, the longer you have to assimilate information pertaining to the answer.

      --
      Support NYCountryLawyer RIAA vs People
    2. Re:So how long is the emperor of China's nose? by rsborg · · Score: 2, Informative

      HUH???? Most polls said people expected that we would find WMD and WMDs were found.

      How is this informative? This is wrong. Read the wikipedia article. Money quote:

      "No one was more surprised than I that we didn't find (WMDs)." General Tommy Franks December 2, 2005.[67]

      --
      Make sure everyone's vote counts: Verified Voting
    3. Re:So how long is the emperor of China's nose? by juuri · · Score: 2, Interesting

      Nearly every one of them said yes.

      The problem with this particular example from P&T (who are awesome despite doing this from time to time) is that this isn't an appeal to find a concrete value or fact, it is instead an appeal to a person's knowledge. Just like the questions asking about weapons of mass destruction were framed in a manner which directly appealed to the information people were being fed by the administration and in turn the media.

      Had the questions been framed more like "If Iraq possesses weapons of mass destruction that could reach Iran? Should we invade?" (Which is a false and leading appeal like most poll questions) you can bet most of the Americans who agreed with the invasion would have said, "No."

      The polling system in the US is greatly flawed, yet many people quote the stats from polls without actually reading the questions. Ever wonder why so many Americans believe in god? Because the typical poll question asks about a higher power rather than any sort of identifiable god. The responder is then primed for the followups.

      Before quoting a poll, make sure you know what the questions actually asked, you may be surprised.

      --
      --- I do not moderate.
  3. A little biased by InvisblePinkUnicorn · · Score: 5, Funny

    Granted, the tests were done on the Price is Right.

    "600 jelly beans?"

    "Higher"

    "900?"

    "Looower...."

  4. The Delphi Method by Illbay · · Score: 4, Interesting
    http://en.wikipedia.org/wiki/Delphi_method

    Another product of the RAND Corporation.

    --
    Any technology distinguishable from magic is insufficiently advanced.
  5. From the I-am-large-I-contain-multitudes dept.? by lilomar · · Score: 4, Funny

    Do I contradict myself?
    Very well then! I contradict myself!

    --
    The creator of this post (Jacob Smith) hereby releases it, and all of his other posts, into the public domain.
    1. Re:From the I-am-large-I-contain-multitudes dept.? by JasterBobaMereel · · Score: 2, Funny

      My name is Legion, for we are many

      --
      Puteulanus fenestra mortis
  6. Wisdom of the Crowds" by phoenixwade · · Score: 5, Insightful

    The idea that a group guessing is more accurate than an individual guess, and if you make more than one guess the mean or average of the guesses is more accurate than a single guess?

    So, in real world terms, 1000 rednecks are going to be more accurate than one Harvard graduate? (assuming the graduate in question isn't our current President) (if we were guessing the number of pickled eggs in a pickle jar, I'd have to agree... Otherwise, I'm somewhat skeptical of how well this translates beyond the estimation of things.

    --
    A positive attitude may not solve all your problems, but it will annoy enough people to make it worth the effort.
    1. Re:Wisdom of the Crowds" by thethibs · · Score: 4, Funny

      More like it takes a thousand Harvard graduates in conference to show the common sense of one redneck. But who's counting?

      --
      I'm a Programmer. That's one level above Software Engineer and one level below Engineer.
    2. Re:Wisdom of the Crowds" by DaoudaW · · Score: 2, Interesting

      Clearly if someone _knows_ the answer, then the wisdom of crowds doesn't work. The 1000 rednecks are clearly not going to out-guess the guy that packed the pickles in the jar if he counted them. But that's totally not the point. Experimental data shows that a group of rednecks beats any individual redneck, that a group of Harvard graduates will beat any individual Harvard graduate, and sometimes a large group of rednecks will beat a small group of Harvard graduates. I did an experiment in the high school math classes I teach during the NCAA basketball tournament. On the day of the final game, I asked each student to predict the score. I was shocked at the results. In each class (5), the class average beat the best individual guess and the aggregate average beat each of the class averages. The final score was predicted to be 75-69 with Kansas winning, in reality they won 75-68. Try it yourself. It definitely works.

  7. Explains by Paranatural · · Score: 4, Funny

    This explains why there's so much informative discussion here at slashdot. N o one knows much of anything, but if you throw enough wild assed guesses at something, one of them is bound to be right, right?

    1. Re:Explains by HairyCanary · · Score: 4, Insightful

      No, I don't think so. It wouldn't be "one of them is bound to be right" -- it would be something more along the lines of "with enough posts, the consensus is likely to be close to reality."

      This assumes, of course, that everything in life is like a jar of jellybeans.

    2. Re:Explains by virgil_disgr4ce · · Score: 3, Insightful

      Har har, but that's not the idea. If only one of them is right it's not the average. I interpret it more like this: intuition is a product of subconscious information processing. The brain is a pattern-recognition machine, and is generally very good at that. I would hazard a guess that if you average out everybody's intuitions ("first guesses"), some of the people are "overthinking" things, but many are just going with their gut, and the pattern recognition and extrapolation that's going on constantly anyway in your brain is often onto something.

      The "generate and test" idea is something I've made great effort to more consciously embrace in my creative endeavors. People decry "quantity over quality," but what I've found is that you simply can't just brood over an idea and "work on" the idea until it's "perfect" and then execute it--you have to create prototypes and test them, and the more you do this, the better you get at creating good prototypes in the first place. Still, it's remarkable how difficult it can be to convince yourself of this.

      --
      Limina.Log
  8. Ah duh! by mspohr · · Score: 5, Funny

    The amazing discovery they made is that when people had time to think about a question, they gave better answers. This is profound.

    --
    I don't read your sig. Why are you reading mine?
    1. Re:Ah duh! by Yvanhoe · · Score: 3, Funny

      And it took a study to prove that. Now let's have a control group that will be base on faith...

      --
      The Wise adapts himself to the world. The Fool adapts the world to himself. Therefore, all progress depends on the Fool.
    2. Re:Ah duh! by JustinOpinion · · Score: 4, Insightful

      In fact there is some research that suggests for certain kinds of decisions, more thought is actually counter-productive. That is, initial "gut" decisions are sometimes more optimal than carefully-considered ones (where "optimal" is measured by longer-term happiness/regret of decision). (For instance, check this writeup of this paper, or the associated Slashdot submission.)

      The point is that while thinking long and hard about some problems can be helpful (e.g. designing something complex and technical), for other kinds of problems, added thought can hinder (e.g. when there are many confounding unknowns).

    3. Re:Ah duh! by John+Hasler · · Score: 3, Insightful

      They didn't say that the second answer was better. They said that the average was better. It would be interesting to know if the second answer was, on average, better than the first.

      --
      Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
    4. Re:Ah duh! by John+Hasler · · Score: 2, Interesting

      I predict that if you ask the same person the same question over and over again even at wide intervals the answers will converge but not necessarily to the correct value (of course, that value may often be "You asked me that before. Bugger off!")

      --
      Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
    5. Re:Ah duh! by Talderas · · Score: 5, Funny

      The point is that while thinking long and hard about some problems can be helpful (e.g. designing something complex and technical), for other kinds of problems, added thought can hinder (e.g. when there are many confounding unknowns).

      So that explains why most /.ers are single.

      --
      "Lack of speed can be overcome. In the worst case by patience." --Znork
  9. yea, no by epfreed · · Score: 2, Funny

    Yea, that seems right. But maybe not.
    Yes, a little right. No, not at all. Total bullshit. Yet also 100% right. Doorknob. Right about 30% of the time. Wait, what was the question?

  10. Durr by Anonymous Coward · · Score: 4, Insightful

    Um, three weeks is plenty of time to look up such an intriguing factoid on the Internet.

  11. Should this be a surprise? by Kohath · · Score: 4, Insightful

    I thought this was understood.

    This is how you are able to catch a ball. Your brain doesn't do a physics calculation and determine where the ball will land. It guesses, watches, refines the guess, repeats, and eventually the guess is close enough so your hand is in the right spot to catch it.

    1. Re:Should this be a surprise? by D+Ninja · · Score: 2, Funny

      This is how you are able to catch a ball. Your brain doesn't do a physics calculation and determine where the ball will land. It guesses, watches, refines the guess, repeats, and eventually the guess is close enough so your hand is in the right spot to catch it.

      Apparently that feature of the brain is broken for the players of the [Insert Name of Hated Sports Team].

      (Hey...I try to make my put downs fun for everybody!)

  12. I call bs on the concept.. by Anonymous Coward · · Score: 2, Insightful

    I read through the first few chapters of James Surowiecki's book in the bookstore. The only thing I found was a small (statistically speaking) number of anecdotes. Nothing really well researched (perhaps there were actual studies done later on).

    I would say my main gripe is that the idea is often presented in an extremely poor manner. Like the author above does with the jelly beans.

    It implies that the "popular mean" can express knowledge that isn't strongly represented in the group already. I.e. Clearly people voting on what medical procedure should be done for a given set of symptoms is radically different than people voting on what they would like to be fed for breakfast and likely puts the patient in a worse position rather than a better one. Now I get that with the idea of jelly beans is the belief that more people with overestimate or underestimate than guess right and that these two sides "balance" but, to my knowledge anyway that hasn't been actually demonstrated in a statistically valid way or for that matter in a way where proper bias control was done (the first example in the book IIRC was about the weight of cattle - clearly that could be biased by the sample used - especially since it was self-selected)

    This brings us to the question: "How is this useful?" It doesn't introduce us to a new concept. We already believe that the "popular mean" is a better judge of some things but not others. It doesn't give us any better idea HOW to judge which things are better judged by crowds and which do not.

  13. Sorry, can't resist... by bhunachchicken · · Score: 2, Funny

    http://www.thinkgeek.com/images/products/zoom/despair-poster-idiocy.jpg

    --
    THE HONOUR OF THE KNIGHTS - CC Licensed Sci-Fi Novel
  14. Short vid about Wisdom of the Crowds by cplusplus · · Score: 2, Interesting

    Here's an interesting little bit that was on Nova Science Now the other night explaining (in a fun way) about the Wisdom of the Crowds: http://www.youtube.com/watch?v=r-FonWBEb0o

    --
    "False hope is why we'll never run out of natural resources!" - Lewis Black
  15. The wisdom of averages by thethibs · · Score: 3, Interesting

    So what we've always thought was the wisdom of crowds turns out to be the wisdom of averages. That does make more sense.

    --
    I'm a Programmer. That's one level above Software Engineer and one level below Engineer.
  16. That's What The Mice Wanted by morgauo · · Score: 2, Funny

    Ah Ha!! Proof! The world and the people in it are a gigantic computer, built by the greatest mice scientists working on the meaning of life! Collectively our minds hold the answer to Life, the Universe, and Everything... even if so many individual minds hold so little.... Where's my improbability engine?!?!? The Vogons are coming!

  17. Minsky's Emotion Machine by br00tus · · Score: 2, Interesting

    In Minsky's book "The Emotion Machine" he describes what we know about the human brain from observation and such. When one encounters a tough problem, one turns different parts of the brain on and off in an attempt to solve it. First might be a trial-and-error brain agent, then an analogy brain agent that searches memory for some similar situation and so forth. That is why there is a difference between blitz chess and tournament chess - in tournament chess, where you have several minutes to make a decision for each move, you can draw on memory, make tactical and strategical decisions and the like quicker than the snap decisions made in blitz chess. It's also why we often go to sleep working on a tough (programming etc.) problem and wake up with the answer - our "unconscious" brain put the answer together while we slept.