Gambling Could Reveal Which Scientific Studies Are Worth Their Salt (sciencemag.org)

sciencehabit writes with this interesting story about using prediction markets to weed out false positives in scientific experiments. From the article: "Gambling is frowned upon in many circles. But what if the gamblers are researchers betting on how each other's experiments will turn out, and the results are used to improve science itself? A group of psychologists has found that their collective gambling—with real money—predicted the outcome of attempts at replicating experimental results better than their own expert guesses. They propose that this type of gambling setup, known as a prediction market, could become part of how science gets done."

Comments

the interesting part

[gtall]

The interesting part seems to be that at least scientists use different weights between asking their opinion (such as a poll) and gambling on outcomes with real money.

Re:the interesting part

[Anonymous Coward]

I wonder if... these psychologists can replicate experimental results of *this* experiment... or are their experiments as good as gambling?

Re:the interesting part

[Some nick or other]

Surprisingly and counter to intuition, it doesn't seem like artificial markets need to use real money to work. Play money produces comparable results in many cases.

It might be that all that is needed is some kind of limited resource so that people who make consistently bad predictions have their impact weighted down until they no longer affect the combined prediction, and that the wisdom of (thus suitably selected) crowds does the rest.

Re:the interesting part

[Anonymous Coward]

Not really. Gambling is seldom limited by if the gamblers can afford it or not.
The big difference is that usually when it comes to making predictions people have a situation where they one outcome is beneficial for them even if it was just a case of "It would be nice if this was the case."
There simply is nothing to gain from having a correct prediction and nothing to lose from having a wrong prediction so people pick the outcome based on what they would like to happen.
Bringing in money, real or fake, is sufficient for people to override this mental block and go with what they actually think, not what they want to think.

Well if it works for business...

[ArsenneLupin]

... then why not also for science?

It does no such thing...

[the_skywise]

It's just enforcing consensus - It doesn't mean the science will be right or wrong anymore than which horse wins the race.

Yeah, the odds increase the probability of a success but that's not a guarantee.

Geez, is this what's become of "science" in the world today?!

I've used this for IT project analysis

[bfwebster]

Estimating confidence in the completion date for an IT project can be hard. When I've been asked to review a troubled IT project, I've found that a very useful technique to determine how far along the project actually is -- and when it's actually likely to be done -- is to ask those involved at different levels what their "$1,000 confidence level" is -- that is, what date they would be willing to bet $1,000 out of their own pocket that the project will actually be done by. I find that even the most optimistic engineer suddenly turns cautious and starts thinking of all the things that could go wrong.

I then ask what their $10,000 confidence level is. Those answers tend to be the most accurate. ..bruce..

Peer review commitees

[goombah99]

Proposals are gambling. When you write a white paper at a company it goes through layers of managerial review if it would be worth the companies time to have you complete a full proposal. Then the Government review panel looks at and decides if they want to invest in that. Peer review informs on the odds foliding in many predictors of success, such as past performance.

Finally the reviews are parimutual gambling. Your odds of winning are not strictly the value of your proposal but the relative odds amongst a group of competitors.