Slashdot Mirror
Crowd Wisdom Better At Predictions Than Top CIA Analysts
First time accepted submitter tkalfigo (1448133) writes "The Good Judgment Project is an experiment put together by three well-known psychologists and some people inside the intelligence community. What they aim to prove is that average, ordinary people in large groups and access just to Google search can predict far more accurately events of geopolitical importance than smart intelligence analysts with access to actual classified information. In fact there is a clearly identified top 1 percent of the 3000 predictors group, who have been identified as super-forecasters: people whose predictions are reportedly 30 percent better than intelligence officers."
People ahead in guessing games such as these are probably more likely to regress to the mean than to continue defying probability.
Happiness in intelligent people is the rarest thing I know.
Ernest Hemingway
CIA cannot believe a wisdom based output, they have to believe that their actions will change the outcome.
Wouldn't the smart thing be that we think they're barely competent?
There was a project affiliated with Google that aimed to predict disease outbreaks using the search engine, but that didn't turn out that well. In fact, it barely succeeded in any of its predictions.
This is why its better to have elections than let the CIA select the government. AFAIK, anyway.
http://michaelsmith.id.au
I wonder if they properly controlled for luck. Take three thousand people and get them to make predictions and some of them are going to appear unusually accurate than others even if all of them are just making completely random guesses. You'd be surprised how many people don't correctly account for that. Every paper proposing clinical diagnostic criteria I've ever read, for example.
Back in 2003, there was a similar system called the Policy Analysis Market (PAM) that was close to being implemented. It got deep-sixed by some world-class idiots from Congress (see my opinion then). It's too bad that we have to go to a somewhat contrived surveying/polling system rather than use something that we know works.
For example, I think a PAM system would have given us (and I mean everyone not just US policy makers) insight into how the events of the Arab Spring revolutions would evolve even if it couldn't have predicted the original flash point.
Cynics don't see the world the way they think it ought to be. Cynics see the world the way it is, and are hence capable forecasters of events.
People are herd animals, and the stuff they start thinking about in groups can become a self-fulfilling prophecy.
With enough people, there will be someone with insightful information, and probably a balance of opinions. Searching for bugs in open source works a little like that.
But in theory if a professional intelligence service had hard evidence that, for example, a politician is bluffing about something, then a policy can be adopted even if it goes against some conventional wisdom.
For example, the information that Saddam Hussein's WMD programme was a hoax prevented a rash invasion...., um, never mind.
I would say the CIA agents between ~1947-2007/2008/2009 are smarter then 2010-2014 CIA agents since the ones pink sliped work in the private sector...
http://en.wikipedia.org/wiki/D...
This has been known since the 60s. Only reason it keeps cropping up is the ego of the people involved in analysis, and the organizational inertia of the agencies involved.
This sounds like the Delphi Method, developed by RAND in the 1950s.
Let's start breeding lucky people!
What they set out to prove has nothing to do with Google vs. classified stuff, but lone analyst vs. crowd.
According to one report, the predictions made by the Good Judgment Project are often better even than intelligence analysts with access to classified information
That is much different than the summary's claim.
The project is about testing a method of making predictions by using a large pool of people, wisdom of the crowd. That pool of people could very well be entirely composed of CIA analysts with access to classified information.
Anyone who specializes in a given field will become myopic. Having worked in the intelligence community for decades, I've seen it first hand hundreds of times. Great analysts become worse at their jobs over time as they become more convinced of their 'rightness' and less apt to all doubt or introspection to impact their judgement. Human nature. Next story.
Alex Jones is probably somewhere very near the top of that group. Woe is we! :)
You should read the article, as the summery is confusing.
The project asks questions like, "What is the chance that North Korea will launch a multi-stage missile before June 2015?"
People enter a guessed % of probability. They get 3000 random people to respond. People's guesses are wildly all over the place. However. . .
When you average out all those responses, the resulting number is spooky accurate. So-called, "Wisdom of the crowd."
Luck has both nothing, and everything to do with it.
My drunken pet vole makes better predictions than those idiots at the CIA. News at 11, Captain Obvious.
I doubt there is any field where one percent of laymen aren't vastly superior to the majority of professionals. The same applies to art, engineering, science, and any other field of study. This is statistically normal.
Seriously who could even argument against the fact that a democracy composed of fucking idiots wont like to vote for you ???
It has been shown that judges make just as good of decisions in family court than hired gun psychologists. The "psychology" industry as a whole wastes a lot of taxpayer money.
While I am sure there are occasional situations where it might be advantageous to be thought foolish and incompetent, in general this is likely a bad thing.
It's like being thought *weak* in military terms. There in tactical situations you'd like the enemy to underestimate your strength, strategically it's better to be thought stronger than you actually are. If a hostile country is considering violating some treaty they have with us, we'd want them to think our intelligence agencies will catch them red-handed. Once they actually go down that road, we'd want them to think our agencies are completely incompetent.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
http://en.wikipedia.org/wiki/Ouija
Two words: Delphi Pool. This is an idea from 1975's The Shockwave Rider.
EYYYEEEEE ENNN TEEEE Ps?
That's where it's at the...
EYYYEEEEE ENNN TEEEE Pee... eeeeeees
The CIA pursues a self serving agenda, and their analysis is coloured by that. Although the CIA has a history of involvement in some of the worst crimes against humanity ever committed, and subverting democracy across the world, I do actually have a respect for some of their analysts. Listen to what they say. They are generally not stupid.
I don't exactly get it. Is it the group as a whole that predicts accurately or its "best predictors"? Because clearly the first hypothesis favors direct democracy as a decision-making process. My intuitive guess is that when you pick a large enough group, some people within that group are clearly going to do better than specialists, because, in a certain way, they are themselves specialists.
Your question implies 'laymen' have another option. They don't. The politcal process eliminates the very people it needs. It's a shortcoming of all political systems.
I'm not particularly smart, but doesn't EVERYBODY already know those CIA fools are incompetent?
They feared that it could be used to suppress protest or support unpopular rule.
I am really curious as to who makes up the "super forecasters" of these geopolitical problems. I suspect that they are merely the Libertarian contingent. 30 people out of 3000? Sounds about right.
Or, alternatively, are they spiritual people? People who partaken in psychedelic experiences? What defines this group?
This story actually really interested me - On its face, the idea of a website that does these things: Poses user-submitted predictive questions, with user profiles so you can track the most successful predictors, and probably some sort of range voting system for the actual voting process, seems like a really swell idea.
Unfortunately, I've not nearly the technical skills or capability to jump into making a website that aggregates questions, votes, user statistics, graphs, profiles and so on. I went ahead and did the next best thing I could think of: http://psychohis7ory.blogspot....
If you compare it to the study, the CIA actually looks reasonably good.
Out of the 3k people in the study, 3% (10 people) were able to make 30% better predictions. This sounds to me like the analysts would come down far ahead of most of a normal curve.
Ask three thousand people to predict what sequence of heads and tails will come up when you flip a coin 30 times. A few of them will appear to have the ability to predict it correctly, at least in that sample.
Whoops, sorry. Math fail. 90 people.
It's much easier than that actually.
Go tools-options-advanced (by fonts and colors.) Find 'allow sites to choose' and make sure that evil box is NOT checked. Then for each category of font make sure that the one selected is clean and easy to read. If not, change it. Hit ok, done.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Friends don't let friends enable ecmascript.
By someone else: http://www.amazon.com/The-Diff...
By me on the need for better intelligence tools for the public: http://pcast.ideascale.com/a/d...
http://www.phibetaiota.net/201...
By me on the security clearance process reduces cognitive diversity in three letter agencies: http://www.phibetaiota.net/201...
"This essay discusses how the USA's security clearance process (mainly related to ensuring secrecy) may [ironically] have a counter-productive negative effect on the USA's national security by reducing "cognitive diversity" among security professionals. "
A 21st century issue: the irony of technologies of abundance in the hands of those still thinking in terms of scarcity.
The US Government came out with an idea to do this for terrorist attacks after 9/11 where people with knowledge in the areas of the middle east, various groups and other specialties would be invited to a program they could make forecasts. Those that were correct would have money donated to their university or another organization of their choice.
Liberals came out and called it the idea of a bunch of crazies.
People speak of the wisdom of crowds but the madness of crowds create things such as financial crashes or stupid wars.
Shall we discuss and research this?
putting the 'B' in LGBTQ+
Robert Anton Wilson summed it up nicely many years ago: "National security is the number one cause of national insecurity."
=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Friends don't let friends enable ecmascript.
Even to suggesting a "basic income": http://en.wikipedia.org/wiki/R...
http://www.rawilson.com/home.h...
http://www.rawilson.com/though...
http://deoxy.org/raw.htm
Thanks for the pointer. I'd be curious where specifically (which book or other writing) wherein he says that, if you know off-hand.
A 21st century issue: the irony of technologies of abundance in the hands of those still thinking in terms of scarcity.