Slashdot Mirror
Crowd Wisdom Better At Predictions Than Top CIA Analysts
First time accepted submitter tkalfigo (1448133) writes "The Good Judgment Project is an experiment put together by three well-known psychologists and some people inside the intelligence community. What they aim to prove is that average, ordinary people in large groups and access just to Google search can predict far more accurately events of geopolitical importance than smart intelligence analysts with access to actual classified information. In fact there is a clearly identified top 1 percent of the 3000 predictors group, who have been identified as super-forecasters: people whose predictions are reportedly 30 percent better than intelligence officers."
People ahead in guessing games such as these are probably more likely to regress to the mean than to continue defying probability.
Happiness in intelligent people is the rarest thing I know.
Ernest Hemingway
CIA cannot believe a wisdom based output, they have to believe that their actions will change the outcome.
There was a project affiliated with Google that aimed to predict disease outbreaks using the search engine, but that didn't turn out that well. In fact, it barely succeeded in any of its predictions.
This is why its better to have elections than let the CIA select the government. AFAIK, anyway.
http://michaelsmith.id.au
I wonder if they properly controlled for luck. Take three thousand people and get them to make predictions and some of them are going to appear unusually accurate than others even if all of them are just making completely random guesses. You'd be surprised how many people don't correctly account for that. Every paper proposing clinical diagnostic criteria I've ever read, for example.
Back in 2003, there was a similar system called the Policy Analysis Market (PAM) that was close to being implemented. It got deep-sixed by some world-class idiots from Congress (see my opinion then). It's too bad that we have to go to a somewhat contrived surveying/polling system rather than use something that we know works.
For example, I think a PAM system would have given us (and I mean everyone not just US policy makers) insight into how the events of the Arab Spring revolutions would evolve even if it couldn't have predicted the original flash point.
With enough people, there will be someone with insightful information, and probably a balance of opinions. Searching for bugs in open source works a little like that.
But in theory if a professional intelligence service had hard evidence that, for example, a politician is bluffing about something, then a policy can be adopted even if it goes against some conventional wisdom.
For example, the information that Saddam Hussein's WMD programme was a hoax prevented a rash invasion...., um, never mind.
http://en.wikipedia.org/wiki/D...
This has been known since the 60s. Only reason it keeps cropping up is the ego of the people involved in analysis, and the organizational inertia of the agencies involved.
We're already doing that. ;-) (Unless you believe this world is so awful that it's better to never get conceived in the first place!)
Ezekiel 23:20
My drunken pet vole makes better predictions than those idiots at the CIA. News at 11, Captain Obvious.
I doubt there is any field where one percent of laymen aren't vastly superior to the majority of professionals. The same applies to art, engineering, science, and any other field of study. This is statistically normal.
Maybe you missed the reference.
Teela Brown is a character from Larry Niven's Ringworld series.
Her defining characteristic is that she's a 6th generation of Birthright Lottery winners and thus, uniquely lucky.
[Fuck Beta]
o0t!
While I am sure there are occasional situations where it might be advantageous to be thought foolish and incompetent, in general this is likely a bad thing.
It's like being thought *weak* in military terms. There in tactical situations you'd like the enemy to underestimate your strength, strategically it's better to be thought stronger than you actually are. If a hostile country is considering violating some treaty they have with us, we'd want them to think our intelligence agencies will catch them red-handed. Once they actually go down that road, we'd want them to think our agencies are completely incompetent.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
I did indeed, but my point is that all of us are winners of one such lottery. :-)
Ezekiel 23:20
"What is the chance that North Korea will launch a multi-stage missile before June 2015?" People enter a guessed % of probability. They get 3000 random people to respond. People's guesses are wildly all over the place. However. . .
When you average out all those responses, the resulting number is spooky accurate. So-called, "Wisdom of the crowd."
Luck has both nothing, and everything to do with it.
How can a probability be spooky accurate when it is in reference to a singular event that can't be repeated over and over again?
You should read the article. There's a little section about "wisdom of crowds" and then the balance of it is about particular people they've selected as being super accurate, such as the pharmacist they use as an example. If you take enough people and ask them to guess randomly, some of their guesses will line up very nicely with the answers to any questions. Purely by luck. If you cherry pick these randomly lucky guessers and don't properly allow for your cherry picking in your calculation of expected performance, you will be misled badly.
A different version of the same phenomenon confuses people who try to write classifiers. I have a friend who was trying to classify patients who did or did not have a disease. He put a bunch of measurements into the classifier, trained it, and look, it was 100% accurate! He was suspicious, so he put in a bunch of randomly generated numbers, trained it, and look, it was 100% accurate! Of course, neither version did any better than chance on data it hadn't been trained on.
Two words: Delphi Pool. This is an idea from 1975's The Shockwave Rider.
EYYYEEEEE ENNN TEEEE Ps?
That's where it's at the...
EYYYEEEEE ENNN TEEEE Pee... eeeeeees
I don't exactly get it. Is it the group as a whole that predicts accurately or its "best predictors"? Because clearly the first hypothesis favors direct democracy as a decision-making process. My intuitive guess is that when you pick a large enough group, some people within that group are clearly going to do better than specialists, because, in a certain way, they are themselves specialists.
I'm not particularly smart, but doesn't EVERYBODY already know those CIA fools are incompetent?
They feared that it could be used to suppress protest or support unpopular rule.
Alas, no.
The Birthright Lotteries in question were a literal lottery held annually to allow someone to have a child.
Everyone in the world was authorized to have one child.
Then, special people were authorized more than one (geniuses, that sort of thing).
And rich people were allowed to buy extra birthrights (the theory being that getting rich was a skill - note that taxation in place pretty much prevented inherited wealth).
And finally, if there were not enough children being born to replace losses in any particular year, there was the Birthright Lottery - winners got a license to have a child.
The odds of winning were on the orders of millions to one against. All of Teela's ancestors for the last six generations were winners of the Birthright Lottery. And she was lucky. Alas for the people who chose to send her to the Ringoworld, her luck did NOT extend to the people near her - odd things happened that were good for HER - if they were good for nearby people, cool, if not, tough....
"I do not agree with what you say, but I will defend to the death your right to say it"
This story actually really interested me - On its face, the idea of a website that does these things: Poses user-submitted predictive questions, with user profiles so you can track the most successful predictors, and probably some sort of range voting system for the actual voting process, seems like a really swell idea.
Unfortunately, I've not nearly the technical skills or capability to jump into making a website that aggregates questions, votes, user statistics, graphs, profiles and so on. I went ahead and did the next best thing I could think of: http://psychohis7ory.blogspot....
There's a big damned difference between predicting that there might or will be a bombing and naming an individual culprit.
If you compare it to the study, the CIA actually looks reasonably good.
Out of the 3k people in the study, 3% (10 people) were able to make 30% better predictions. This sounds to me like the analysts would come down far ahead of most of a normal curve.
Ask three thousand people to predict what sequence of heads and tails will come up when you flip a coin 30 times. A few of them will appear to have the ability to predict it correctly, at least in that sample.
Whoops, sorry. Math fail. 90 people.
It's much easier than that actually.
Go tools-options-advanced (by fonts and colors.) Find 'allow sites to choose' and make sure that evil box is NOT checked. Then for each category of font make sure that the one selected is clean and easy to read. If not, change it. Hit ok, done.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Friends don't let friends enable ecmascript.
By someone else: http://www.amazon.com/The-Diff...
By me on the need for better intelligence tools for the public: http://pcast.ideascale.com/a/d...
http://www.phibetaiota.net/201...
By me on the security clearance process reduces cognitive diversity in three letter agencies: http://www.phibetaiota.net/201...
"This essay discusses how the USA's security clearance process (mainly related to ensuring secrecy) may [ironically] have a counter-productive negative effect on the USA's national security by reducing "cognitive diversity" among security professionals. "
A 21st century issue: the irony of technologies of abundance in the hands of those still thinking in terms of scarcity.
The US Government came out with an idea to do this for terrorist attacks after 9/11 where people with knowledge in the areas of the middle east, various groups and other specialties would be invited to a program they could make forecasts. Those that were correct would have money donated to their university or another organization of their choice.
Liberals came out and called it the idea of a bunch of crazies.
People speak of the wisdom of crowds but the madness of crowds create things such as financial crashes or stupid wars.
Shall we discuss and research this?
putting the 'B' in LGBTQ+
Robert Anton Wilson summed it up nicely many years ago: "National security is the number one cause of national insecurity."
=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Friends don't let friends enable ecmascript.
Even to suggesting a "basic income": http://en.wikipedia.org/wiki/R...
http://www.rawilson.com/home.h...
http://www.rawilson.com/though...
http://deoxy.org/raw.htm
Thanks for the pointer. I'd be curious where specifically (which book or other writing) wherein he says that, if you know off-hand.
A 21st century issue: the irony of technologies of abundance in the hands of those still thinking in terms of scarcity.