Gambling Could Reveal Which Scientific Studies Are Worth Their Salt (sciencemag.org)

sciencehabit writes with this interesting story about using prediction markets to weed out false positives in scientific experiments. From the article: "Gambling is frowned upon in many circles. But what if the gamblers are researchers betting on how each other's experiments will turn out, and the results are used to improve science itself? A group of psychologists has found that their collective gambling—with real money—predicted the outcome of attempts at replicating experimental results better than their own expert guesses. They propose that this type of gambling setup, known as a prediction market, could become part of how science gets done."

Comments

the interesting part

[gtall]

The interesting part seems to be that at least scientists use different weights between asking their opinion (such as a poll) and gambling on outcomes with real money.

Re:the interesting part

[Anonymous Coward]

I wonder if... these psychologists can replicate experimental results of *this* experiment... or are their experiments as good as gambling?

Re:the interesting part

[Some nick or other]

Surprisingly and counter to intuition, it doesn't seem like artificial markets need to use real money to work. Play money produces comparable results in many cases.

It might be that all that is needed is some kind of limited resource so that people who make consistently bad predictions have their impact weighted down until they no longer affect the combined prediction, and that the wisdom of (thus suitably selected) crowds does the rest.

I've used this for IT project analysis

[bfwebster]

Estimating confidence in the completion date for an IT project can be hard. When I've been asked to review a troubled IT project, I've found that a very useful technique to determine how far along the project actually is -- and when it's actually likely to be done -- is to ask those involved at different levels what their "$1,000 confidence level" is -- that is, what date they would be willing to bet $1,000 out of their own pocket that the project will actually be done by. I find that even the most optimistic engineer suddenly turns cautious and starts thinking of all the things that could go wrong.

I then ask what their $10,000 confidence level is. Those answers tend to be the most accurate. ..bruce..