Slashdot Mirror
"Wisdom of Crowds" Works For Individuals Too
ideonexus writes "Take a crowd of people and have them guess how many jelly beans are in a jar, and the average of their answers will be remarkably accurate. Now researchers have found the same goes for asking one person to guess about the same thing several times. Accuracy improved when the individual was given longer periods of time between guesses." The anonymous author of the Economist piece, not quoting the researchers, says the finding bolsters the "generate and test" model of creative thinking.
In related news, students were found to do far better on multiple choice tests when given an unlimited number of guesses at each question. Even students that didn't study eventually got As.
My blog
I'm going to guess there will be 3 "first posts" on this...
There is no -1 Disagree mod. Slashdot.org/faq defines mod options. USE IT.
Wisdom of crowds only works when the crowd has some information about the situation. Look at polls about Iraqi weapons of mass destruction for more details.
Inventions have long since reached their limit, and I see no hope for further development.-- Frontinus, 1st cent. AD
Granted, the tests were done on the Price is Right.
"600 jelly beans?"
"Higher"
"900?"
"Looower...."
And if they run several of these tests over time, their result maybe will also get closer to the truth .. That it's bu#sh#t.
http://revj.sourceforge.net
Another product of the RAND Corporation.
Any technology distinguishable from magic is insufficiently advanced.
Think long think wrong.
Do I contradict myself?
Very well then! I contradict myself!
The creator of this post (Jacob Smith) hereby releases it, and all of his other posts, into the public domain.
The idea that a group guessing is more accurate than an individual guess, and if you make more than one guess the mean or average of the guesses is more accurate than a single guess?
So, in real world terms, 1000 rednecks are going to be more accurate than one Harvard graduate? (assuming the graduate in question isn't our current President) (if we were guessing the number of pickled eggs in a pickle jar, I'd have to agree... Otherwise, I'm somewhat skeptical of how well this translates beyond the estimation of things.
A positive attitude may not solve all your problems, but it will annoy enough people to make it worth the effort.
This explains why there's so much informative discussion here at slashdot. N o one knows much of anything, but if you throw enough wild assed guesses at something, one of them is bound to be right, right?
N.B. -- this does not apply to politics. In fact, the phrase "Political Science" may be turning into the biggest oxymoron of all time.
Rule #1 -- Politics always trumps technology.
I use this all the time... I just never thought about it that way!
Never ask for directions from a two-headed tourist! -Big Bird
The amazing discovery they made is that when people had time to think about a question, they gave better answers. This is profound.
I don't read your sig. Why are you reading mine?
Yea, that seems right. But maybe not.
Yes, a little right. No, not at all. Total bullshit. Yet also 100% right. Doorknob. Right about 30% of the time. Wait, what was the question?
Usually what you get out of crowds is some form of mob rule, not wisdom.
If you've never been modded as "flamebait" or "troll," you've never tried to argue a minority viewpoint here!
So what they're saying is when someone has a long period of time to think about an answer, or the trial-and-error option they're answers are better then just guessing? Astounding! This new information changes everything...wait no no it doesn't this has been known for thousands of years. Good Job at rediscovering what was already known though, really I mean it.
Um, three weeks is plenty of time to look up such an intriguing factoid on the Internet.
I was thinking about a jellybean in the jar scenario just the other day, as we had someone guess correctly for a prize at work. Examples of fast calculation by certain autistics (think toothpicks in the movie Rainman) suggests to me that the mind counts and analyses all sorts of information, and with certain individuals is able to be called more easily. What does the mind do behind the curtain? Can these feats be learned and a person be trained to do these things? I'd like to think so. Of course, this study could just be evidence that humans are capable of abstract calculation and rounding. I am not a statistician (ianas?), but wouldn't you expect that several guesses would average out this way? Humans have carried berries in baskets for millennia sure, but aside from an "evolutionary" trait, I would think humans have ample brain power to make very educated guesses quite easily.
Like a city whose walls are broken down is a man who lacks self-control.
I thought this was understood.
This is how you are able to catch a ball. Your brain doesn't do a physics calculation and determine where the ball will land. It guesses, watches, refines the guess, repeats, and eventually the guess is close enough so your hand is in the right spot to catch it.
I read through the first few chapters of James Surowiecki's book in the bookstore. The only thing I found was a small (statistically speaking) number of anecdotes. Nothing really well researched (perhaps there were actual studies done later on).
I would say my main gripe is that the idea is often presented in an extremely poor manner. Like the author above does with the jelly beans.
It implies that the "popular mean" can express knowledge that isn't strongly represented in the group already. I.e. Clearly people voting on what medical procedure should be done for a given set of symptoms is radically different than people voting on what they would like to be fed for breakfast and likely puts the patient in a worse position rather than a better one. Now I get that with the idea of jelly beans is the belief that more people with overestimate or underestimate than guess right and that these two sides "balance" but, to my knowledge anyway that hasn't been actually demonstrated in a statistically valid way or for that matter in a way where proper bias control was done (the first example in the book IIRC was about the weight of cattle - clearly that could be biased by the sample used - especially since it was self-selected)
This brings us to the question: "How is this useful?" It doesn't introduce us to a new concept. We already believe that the "popular mean" is a better judge of some things but not others. It doesn't give us any better idea HOW to judge which things are better judged by crowds and which do not.
http://www.thinkgeek.com/images/products/zoom/despair-poster-idiocy.jpg
THE HONOUR OF THE KNIGHTS - CC Licensed Sci-Fi Novel
I was deeply involved in sports betting for a while. One of the first things any serious sports bettor learns is that if everyone else likes the same team as you, then start to worry.
There are several websites dedicated to come up with a public consensus on wagers. They were always a must-see for me but only as another piece of information (oddly, it's just as dumb to bet along with the crowd as it is to bet against it.)
Yes, this is gambling, but it's not like betting red or black in roulette. There here has never been so much information available: stats, trends, computers, and tons of message boards to share it all.
In the end, very few manage to win that elusive 52.x percent of the time needed to make a profit.
As for me, I did win for a while but proved to be too undisciplined to stick to my own rules :)
Here's an interesting little bit that was on Nova Science Now the other night explaining (in a fun way) about the Wisdom of the Crowds: http://www.youtube.com/watch?v=r-FonWBEb0o
"False hope is why we'll never run out of natural resources!" - Lewis Black
It sounds like this is just an example of someone making a confidence interval; "I know the answer is in a certain range, so I'll make two guesses and it should probably be in there."
If each guess is made using a different model, then you're adding more "information" to the guess. Then there's more total information in the average, than in each guess on it's own.
But what do I know, I'm not a psychologist. I could just be making stuff up.
So what we've always thought was the wisdom of crowds turns out to be the wisdom of averages. That does make more sense.
I'm a Programmer. That's one level above Software Engineer and one level below Engineer.
I guess in voting for him they picked the winner (insane laughter in my head right now) which would verify what this is saying.. but wisdom ???
waiting for ad.doubleclick.net
some numerical methods concepts.
Invenio via vel creo
Things get awkward when people talk about harnessing the power of crowds to improve complex predictions. A question like, "What will the price of oil be in five years?" is damn near impossible to answer without a time machine because it completely fails to factor in unpredictable economic & political disruptions (9/11, some guy inventing a portable fusion generator in his basement, alien invasion, global war, or pandemic). In fact, the financial markets are a great example of "the wisdom of crowds" looking more like "the wisdom of sheep."
Humans are forced to take analytical shortcuts when observing the world, and the process involves heavy socialization. We can spot a crazy (and potentially dangerous) street person in mere moments. We can judge the social standing of a person within seconds. It's one reason that branding works so well, because we can be trained to think, "Mmm, BMW good. Macintosh good" without pausing for hours of tedious analysis. The trouble is that socialization plays a massive role in our views of the world. Any "educated guess" we make about the possibility of a future pandemic or going to war with the Vogons will be colored by our social network - what our friends think, what our favorite bloggers think, what the media tells us. It is a rare soul who can step out from the crowd to see the forest for the trees.
But what if the first guess is correct?
Ah Ha!! Proof! The world and the people in it are a gigantic computer, built by the greatest mice scientists working on the meaning of life! Collectively our minds hold the answer to Life, the Universe, and Everything... even if so many individual minds hold so little.... Where's my improbability engine?!?!? The Vogons are coming!
Earth-shattering stuff.
Gotta love Pop Science
"Ignorance more frequently begets confidence than does knowledge." Darwin
Has the approval rating gone down because so many more americans are learning from the crowd?
Or..
Are we all just weekend hippies following the crowd to a doomed Haight Ashbury?
I want to be retired when I grow up.
Yes wisdom not intelligence. That's why there are two stats on the D&D character sheet.
Guess what people will think is insightful/informative/positive.
Get modded troll.
Repeat. :)
In Minsky's book "The Emotion Machine" he describes what we know about the human brain from observation and such. When one encounters a tough problem, one turns different parts of the brain on and off in an attempt to solve it. First might be a trial-and-error brain agent, then an analogy brain agent that searches memory for some similar situation and so forth. That is why there is a difference between blitz chess and tournament chess - in tournament chess, where you have several minutes to make a decision for each move, you can draw on memory, make tactical and strategical decisions and the like quicker than the snap decisions made in blitz chess. It's also why we often go to sleep working on a tough (programming etc.) problem and wake up with the answer - our "unconscious" brain put the answer together while we slept.
Have a look at this Derron Brown episode!
As pointed out in "The Black Swan", this is HIGHLY dependent on the scenario in question. For problems where the bounds are easily determined (say, the number of jelly beans in a certain size jar), then "crowd wisdom" might apply. For example, you would be unlikely to guess to that a 1 gallon jar holds 3 million normal sized jelly beans.
But take a different scenario. Let's have the crowd guess the income of some random individual. Depending on the individual the crowd could be very wrong. If the individual is a homeless man with no income, they will be wrong. If the individual is James Simons with $1.5B (yes, that's a B) of income, they will also be wrong.
Only when the scenario in question is already "average", then a crowd may be able to make a reasonable estimate. If the scenario is not "average", the crowd will continue to make an average estimate.
So, while it appears that there may be wisdom in a crowd, really there's only mediocrity. If you apply the mediocre crowd to the mediocre problem, you should expect to get reasonable results.
Of course, the same dependencies on the problem also exist for the crowd. A crowd of A-list movie stars guessing the income for the random individual may be very different from a crowd on wellfare.
This is just like noise reduction. You either shoot with more sensors at the same time, and do an average between them to eliminate sensor noise, or shoot with the same sensor repeatedly and average with itself.
The noise and other sensor defects are reduced.
Apparently, it's the same with people.
Many people are taking this out of context.
Obviously if you presented a complex mathematical PHD level math problem to a crowd, the crowd have absolutely no concept of what the answer could be and the answer would never converge.
However, when looking at an example like counting the number of jellybeans in a jar, most of us are guessing within REASON. We know it's > 10 and we know it's 1 000 000. So we're testing our ability to approximate.
What this study says is our ability to judge increases with the number of people. Lots of people judging means the group's average is closer to the real answer than most individuals.
It also says that an individual judging multiple times with greater breaks between each judgment gets closer to the answer.
I don't think this should be too surprising. By judging multiple times, your subconscious is able to work on the problem. It also eliminates variations of things you don't think about. Maybe when you first judged the number of jellybeans in the jar, it was morning and you were hungry and your subconscious made you judge higher. Now its afternoon after you had lunch, and that apsect is not being played out. I think that given multiple judgments, we also take that into account. If last time I judged 900, but I had a 'feeling' it was too high, this time I might judge 800.
http://www.psychologicalscience.org/journals/ps/19_7_inpress/vul.pdf
It depends... are you thinking of the number 500000 :D
Beware of bugs in the above code; I have only proved it correct, not tried it.
Please reply with your time estimates so we can average them and produce a more accurate answer.
In other news, I prefer cold drinks, hot women, and slow dances. Let's average everyone's opinion on those too.
Reduce, reuse, cycle
I regularly observe on this site that a "redneck" is stereotypically utterly stupid, and it reminds me of the now universally unacceptable stereotype that black people were dumb. So is "redneck" the new black, in that it has become our new 'generic group of intellectually inferior people' with the advantage of not being deemed racist because "redneck" is conveniently not regarded as a race? Doesn't it make us bigots anyways to pick regularly on this group of people? Am I the only one who feels offended despite not being part of this "redneck" group? Is it a sort of food chain of intelligence in which black <- redneck <- non-redneck American <- European?
You just got troll'd!
So what is the answer to the article's question: "What percentage of the world's airports are in the USA?" ? Anyone? Hive-mind?
The trick to guessing how many jelly beans or whatever are in a jar is to estimate how many beans tall wide and long the jar is and then multiply it out as if the jar was rectangular. You can then take off a little bit to adjust for a round rather than rectangular jar, but your estimate will be so inaccurate anyway that a little bit like that won't matter much anyway. You'll still be way off, but much closer than anyone that just looks at it and makes a guess.
If the items in the container are very small like grains of sand, then it may help to first estimate the width of a group of the items and then estimate how many groups wide the container is. For example it might take three grains of sand to make a millimeter and therefore 30 grains of sand to a centimeter. If a barrel is 60 centimeters wide then the barrel is about 1800 grains of sand wide. Round that off to 2000. A cube 2000 grains across would have about 2000*2000*2000 = 8,000,000,000 grains of sand.
The thing that makes estimating volumes so hard is that when the estimate of the width is cubed the error in the estimate of the width is also cubed. For example if you estimate a cube is four centimeters on a side and it is really five centimeters on a side then it will be DOUBLE the volume of your estimate.
Assuming you accept that sporting events are non-linear enough to ensure that there is NO practical way to determine the outcome beforehand, sports betting markets are actually one of the best demonstrations of the accuracy of the 'Wisdom of Crowds'.
Weight of money makes the market, especially on betting exchanges like Betfair, and any statistical analysis you care to do will show a very close relationship between prices and long-term frequency. As the only commodity being traded here is information, these markets are frighteningly efficient.
[ ]Half Empty [ ]Half Full [x]Twice as big as it needs to be
This sounds like an experiment a coworker once relayed to me: Given a object you want to measure, drop a ruler onto it from a height of a few feet. Do this repeatedly and each time record the value on the ruler at the edge of the object. The average of a sufficient number of such values will approach the actual measurement.
Just applied to humans. Go figure, the world is self-similar like that.
All hail chaos.
Terrorists can attack freedom, but only Congress can destroy it.
GP said:
and you said
You are both right. However, GP is more correct than you.
You speak of a distribution where the expected value is the correct solution, but you also said that it was a "symmetric" distribution (and thus it would be "centered" around the correct solution). This is sufficient but not necessary. Amazingly enough, no matter how asymmetric, skewed or weird-looking a distribution is, if you take the average of enough independent guesses, the distribution turns into a Gaussian bell curve.
For example, if you have a fair die, with faces numbered 1, 1, 1, 1, 1, and 7, then most of the time you will roll a 1, but once in a while you will roll a 7. The average value rolled is 2. If you roll the die, say, 4 times, the average will be 2 --with a somewhat bell-shaped curve. If you roll the die 20 times, it will be pretty much bell-shaped (and symmetric). If you roll it 100 times, it's a near-perfect bell.
Take a look at the Central Limit Theorem for more info. When I first learned this in stochastics, I couldn't wrap my mind around it. Math can be stunningly beautiful sometimes.
404555974007725459910684486621289147856453481154 in hex is "You sank my Battleship?"
[GPG key in journal]
Well spotted, and a very good explanation.
*bows*
Slashdot: where don knuth is an idiot because he cant grasp the awesome power of php
Now researchers have found the same goes for asking one person to guess about the same thing several times. Accuracy improved when the individual was given longer periods of time between guesses
If you have more time to guess and consider, your answers are more precise. Truly amazing discoveries...
Wisdom of crowds may work only with Altruists!
Slashdot = Sarcasm