Crowd Wisdom Better At Predictions Than Top CIA Analysts

First time accepted submitter tkalfigo (1448133) writes "The Good Judgment Project is an experiment put together by three well-known psychologists and some people inside the intelligence community. What they aim to prove is that average, ordinary people in large groups and access just to Google search can predict far more accurately events of geopolitical importance than smart intelligence analysts with access to actual classified information. In fact there is a clearly identified top 1 percent of the 3000 predictors group, who have been identified as super-forecasters: people whose predictions are reportedly 30 percent better than intelligence officers."

Comments

Luck resets every time you guess.

[rmdingler]

People ahead in guessing games such as these are probably more likely to regress to the mean than to continue defying probability.

Re:Luck resets every time you guess.

[Arker]

Except that this has nothing to do with luck. It has to do with independent observers having less pressure on them to, consciously or subconsciously, produce rhetoric ostensibly concerning foreign policy but whose content is determined by domestic political needs.

Re:Luck resets every time you guess.

[fustakrakich]

Why are you being modded down? For the font? And yes, they produce the results they are told to produce. That's how they play their game.

Re:Luck resets every time you guess.

[Flentil]

I think the majority here use Slashdot's default font and never messed with it, but you did mess with yours, making your posts stand out as odd. Blaming everyone else, making them responsible to fix the 'bug' you created on their screen, isn't very helpful. It's a lot easier to just mod you down than to delve into browser font settings and possibly mess up how we view all other websites, just because you like your posts on Slashdot to look like they were typed on an old-timey typewriter. Why don't you just fix your own browser font settings and not put the burden on everyone else?

Re:Luck resets every time you guess.

[Cenan]

If you're using Firefox, you can download the Stylish plugin and make sure everybody conforms to your wishes by adding a rule for the "commentBody" class. Might also want to add one for the "tt" tag.

.commentBody {
              font-family: monospace;
              font-size: 14px;
        }

This makes everybody's posts appear to use code tags. I find it readable, you might not, but you do have a say in it.

Re:Luck resets every time you guess.

[Sique]

Not only that, but people with some distance to the situation and only superficial knowledge are not blinded by facts and details. We have the same phenomenon with predicting sport results. There, people who are not absolute fans or professionals in the sport usually fare better at predicting results as they don't give too much weight on some details, or their own preferences, which in the long run prove to have much less influence than expected. Instead they basicly tend to put teams or athletes they often heard about in front and less famous ones more to the back.

So you have three group of people: Those who don't know anything, and who indeed are the worst predictors, then those who have some basic understanding of the situation, who are the best predictors, and then the specialists, who know intricate details and have very profound knowledge, who aren't very good at predicting because they give too much weight to single aspects and have some very strong opinions or emotions about the situation which blinds their judgement.

Re:Luck resets every time you guess.

[Ardyvee]

No. Courier New is a pretty nice font for some things. Just not the /. comment section.

Re:Luck resets every time you guess.

[ultranova]

Unreadable? How sad you have a comprehensive system so primitive.

It's called optimization. "Unreadable" means that the low-level - indeed, in most people subconscious - routine was unable to comprehend the text, and kicked it up to a higher-level one. The higher level subsystem is more flexible but, as a result, needs more resources which could be put to better use; and of course the process can occur again, escalating text comprehension all the way to the conscious mind. Thus, if you write unreadable - which means nonstandard - text, you are inconveniencing everyone who reads it, and deserve to be modded down.

CIA uses Dirty Tricks Project

[Anonymous Coward]

CIA cannot believe a wisdom based output, they have to believe that their actions will change the outcome.

Well...

[ComputersKai]

There was a project affiliated with Google that aimed to predict disease outbreaks using the search engine, but that didn't turn out that well. In fact, it barely succeeded in any of its predictions.

Re:Well...

[Cenan]

The problem with Google's prediction algorithm is that it consistently overshoots. The story was on /. about a month ago, as far as I can tell they're not only not predicting cases correctly, they aren't even attempting to distinguish between strains (how could they, they're predicting from search activity - flu victims rarely know their strain).

Well yeah

[MichaelSmith]

This is why its better to have elections than let the CIA select the government. AFAIK, anyway.

Re:Well yeah

[BradMajors]

The CIA disagrees.

Seems fishy

[ceoyoyo]

I wonder if they properly controlled for luck. Take three thousand people and get them to make predictions and some of them are going to appear unusually accurate than others even if all of them are just making completely random guesses. You'd be surprised how many people don't correctly account for that. Every paper proposing clinical diagnostic criteria I've ever read, for example.

Re:Seems fishy

[mikael]

Why do you think it is purely luck? When you have these wild discussion parties - things like "is a bright blob of pixels on a Mars Rover image a cosmic ray, a high-voltage dust-devil, light contamination of a camera box, a gas geyser", you will have an incredible combination of experts - everyone from geologists, ranchers, hill-hikers, photographers, astronomers. Geologists will tell you want can and can't come from the ground, ranchers and hill-hikers will tell you things they have seen and never seen, photographers will tell you what visual artifacts can appear on a camera, and astronomers tell you what can fall from the sky and can't, and what those falling things look like.

It's like solving a giant logic problem where everyone can cross off or tick what what they know. Eventually the set of possible answers reduces down to one or two.

Re:Seems fishy

[hibiki_r]

Except this isn't how it works at all.

The wisdom of crowds works doesn't have anything to do with having experts. After all, the experts have no way of influencing the crowd. It is a well defined phenomenon that works when people's biases are pretty random, so mistakes cancel each other out. It's a lower quality estimation mechanism than a market, where people that are sure of their answer can be 'louder' than those that don't know said answer, and it lacks the feedback mechanisms of a market, but still, it is helpful to predict things based on widely available information. Ask the crowd information few of them have any idea about, and their result will suck.

So what does the average beating CIA personnel? That the CIA's biases are large enough to need quite a bit of quality control.

Now, having a 1% of the respondents be far better than the CIA experts probably means nothing. If I invite 3000 people over to guess how 10 coin flips will turn out, chances are one or two of them will guess all of them correctly, but that would not make them seers capable of seeing the future. how many people were worse than 30% worse than those same CIA experts?

Re:Seems fishy

[Attila Dimedici]

The wisdom of crowds works doesn't have anything to do with having experts.

You are right that the wisdom of crowds does not come from having experts. The wisdom of crowds comes from having a lot of people who all have a little bit of knowledge relevant to the subject. Some of that knowledge might be something that you would not necessarily think was relevant, but when applied as a filter on the other knowledge present produces a result much more accurate than an expert on the subject would ever produce.

The results of this study are not new. Back in the lat 70s, early 80s, there was a study which showed that a group of people with no particular expertise on the subject will reach a better decision than an individual expert on the subject as long as certain criteria are met in the group discussion. The most important criteria that needs to be met is that the groups deliberations must be such that the individual charisma of each person must not be allowed to influence the group discussion. My understanding is that they accomplished this by having all of the discussion occur in anonymous text (such as if all of the comments on slashdot were from Anonymous Coward).

Reminds me of the Policy Analysis Market

[khallow]

Back in 2003, there was a similar system called the Policy Analysis Market (PAM) that was close to being implemented. It got deep-sixed by some world-class idiots from Congress (see my opinion then). It's too bad that we have to go to a somewhat contrived surveying/polling system rather than use something that we know works.

For example, I think a PAM system would have given us (and I mean everyone not just US policy makers) insight into how the events of the Arab Spring revolutions would evolve even if it couldn't have predicted the original flash point.

Re:Reminds me of the Policy Analysis Market

[TapeCutter]

even if it couldn't have predicted the original flash point.

Funny you should say that, the diplomatic cable leaks showed that high level western diplomats in Syria were concerned about a civil war erupting due to the severe "fertile crescent" drought fuelling internal migration from rural areas to the cities (10% of Syria's total population simply abandoned their farms due to lack of water). The drought caused food prices to rise sharply and food riots became a regular occurrence in cities across the middle east and North Africa.

"flash point" - Have a look at why that protester set fire to himself in the public square and why it resonated so strongly across the Arab world, it wasn't because they all logged on to FB and suddenly realised their governments were tyrannical. Predicting this sort of social unrest is like predicting an earthquake in LA, you can be pretty confident that your prediction will come to pass but have no idea when.

Re:Reminds me of the Policy Analysis Market

[nut]

Back in 2003, there was a similar system called the Policy Analysis Market (PAM) that was close to being implemented. It got deep-sixed by some world-class idiots from Congress ...

Maybe they weren't idiots. Maybe the were protecting a lucrative after-Congress job market...

Re:Reminds me of the Policy Analysis Market

[gringer]

New Zealand has iPredict, where you can make money off correct guesses about the future:

https://www.ipredict.co.nz/

Question of scale

[Livius]

With enough people, there will be someone with insightful information, and probably a balance of opinions. Searching for bugs in open source works a little like that.

But in theory if a professional intelligence service had hard evidence that, for example, a politician is bluffing about something, then a policy can be adopted even if it goes against some conventional wisdom.

For example, the information that Saddam Hussein's WMD programme was a hoax prevented a rash invasion...., um, never mind.

Re:Question of scale

[mikael]

Both the British and Americans used the same government contact for their information, but they didn't tell each other who that contact was. In fact, they had different codenames for the person. When they cross-referenced each others information, they got two confirmations.

Old news

[Crashmarik]

http://en.wikipedia.org/wiki/D...

This has been known since the 60s. Only reason it keeps cropping up is the ego of the people involved in analysis, and the organizational inertia of the agencies involved.

Re:Bell Curve

[TapeCutter]

Interesting, so why do "laymen" in the US keep electing zealots, crackpots, and "entrepreneurs" who are clearly lying to their face for fun and profit? Do you guys enjoy being treated with contempt?

Re:But why would the CIA release their best result

[hey!]

While I am sure there are occasional situations where it might be advantageous to be thought foolish and incompetent, in general this is likely a bad thing.

It's like being thought *weak* in military terms. There in tactical situations you'd like the enemy to underestimate your strength, strategically it's better to be thought stronger than you actually are. If a hostile country is considering violating some treaty they have with us, we'd want them to think our intelligence agencies will catch them red-handed. Once they actually go down that road, we'd want them to think our agencies are completely incompetent.

Re:Bell Curve

[Attila Dimedici]

That is because elections happen in a way that eliminates the most important aspect of crowd wisdom. In order for crowd wisdom to be reliable it must be insulated from being influenced by the charisma of individuals. There is no way to set up elections to do this. Actually, I wonder if the secret ballot may in a way actually exacerbate this problem.

This thought just came to me now, so I do not think I can explain the reasoning as to why that might be so. I will try any way. It seems possible that the necessity of having to explain to one's peers the reason one made a choice which was unpopular with them might offset the amount which the individual charisma of the candidate (or the candidate's representatives) influenced the decision. The other way in which eliminating the secret ballot would influence the outcome is that it would make it harder to manufacture votes for a particular candidate, since everyone would have some idea how everyone else voted.

So true; diversity & better tools may help too

[Paul Fernhout]

By someone else: http://www.amazon.com/The-Diff...

By me on the need for better intelligence tools for the public: http://pcast.ideascale.com/a/d...
http://www.phibetaiota.net/201...

By me on the security clearance process reduces cognitive diversity in three letter agencies: http://www.phibetaiota.net/201...
"This essay discusses how the USA's security clearance process (mainly related to ensuring secrecy) may [ironically] have a counter-productive negative effect on the USA's national security by reducing "cognitive diversity" among security professionals. "