Journal Article On Precognition Sparks Outrage

thomst writes "The New York Times has an article (cookies and free subscription required) about the protests generated by The Journal of Personality and Social Psychology's decision to accept for publication later this year an article (PDF format) on precognition (the Times erroneously calls it ESP). Complaints center around the peer reviewers, none of whom is an expert in statistical analysis."

Comments

Peer review only provides weak prescreening

[Anonymous Coward]

I'm a Ph.D. candidate in EE, and I'm sometimes invited to review papers for IEEE journals.

I always read the paper carefully at least 3 times, read the important parts of references that are new to me, check all the math and sometimes even reproduce some simpler simulations.

Most reviewers aren't this careful. They either don't have the time or don't have the expertise to find some flaws. Keep in mind that reviewers aren't paid, and are anonymous. Also, the best reviewers are the best researchers, who are usually busy with their own projects.

I often find serious flaws that my fellow reviewers completely overlook. Fortunately in these cases, the editors have always used my reviews to override the other two (the review decision is not a majority vote). Given the quality of the reviewers out there, some papers are accepted simply because the editor invited 3 incompetent reviewers, which is not very unusual. And we're talking about IEEE journals, which should be the best in the field.

So in practice, peer review is only a weak pre-screening process that often rejects good papers and accepts bad work. Science progresses because once something is published, other people attempt to reproduce it. If the idea works, then it's incorporated into other work and becomes famous. Otherwise, people just ignore it.

Re:Peer review only provides weak prescreening

[gweihir]

I have a PhD in CS.

You can also have good science rejected by getting three incompetent reviewers. Happened to me several times, the worst one when the program committee attached a note that showed they had not read or understood their own call for papers. I suspect a direct lie to keep me out. Published it later unchanged somewhere else and those people were surprised it got rejected earlier.

In addition to incompetent reviewers, there are also those that are envious or want to steal your ideas. Peer-review is fundamentally broken. One friend who has a PhD in a different CS area thinks 70% of researchers are corrupt, reviewing things positively when they know the authors, no matter the quality and negatively otherwise. Lying in application to research grants is also quite common. The final result is that good researchers have trouble working and often leave research altogether, which may be an explanation for how glacially slow some fields move.

maybe...

[TheCarp]

Maybe this is a hack. They say he has a sense of humor.... think of this... he did design his studies well, at least the ones that I have read about. The effects of this "time leaking" are fairly small. Perhaps the entire point...is to make a point about statistics.

Added bonus? Put the ESP issue to bed. Him doing this, and specifically doing it so publicly and getting it passed peer review and publication, ENSURES that these studies are going to be replicated by numerous people, for the next several years. That, in and of itself, could produce enough evidence against ESP to really put the issue to bed :)

Say what you want about his paper, the effects reported are as large as many "well accepted" study results. Which may be the scariest part of all.

That said, I am no ESP believer (that may be obvious) but, some of the statements that are made against it are ridiculous too. "Why aren't people winning the lottory with their perfect precognition". The effects he is talking about here are on the order of a few percentage points better than random... which is more than the house advantage at many casino games (assuming optimal play)

-Steve

Erotic Pictures a Necessary Element

[eldavojohn]

Why settle for just 1M? Play the lottery for a few weeks and you don't even have to bother writing an article to be rich.

So I realize a lot of people aren't going to read the article but here's the meaty parts for you statistics snobs (and really, the Bayes folks are going to be all over this one):

Across all 100 sessions, participants correctly identified the future position of the erotic pictures significantly more frequently than the 50% hit rate expected by chance: 53.1%, t(99) = 2.51, p = .01, d = 0.25.3 In contrast, their hit rate on the nonerotic pictures did not differ significantly from chance: 49.8%, t(99) = -0.15, p = .56. This was true across all types of nonerotic pictures: neutral pictures, 49.6%; negative pictures, 51.3%; positive pictures, 49.4%; and romantic but nonerotic pictures, 50.2%. (All t values < 1.) The difference between erotic and nonerotic trials was itself significant, tdiff(99) = 1.85, p = .031, d = 0.19.

There's a lot more about eliminating random number generators (by using this little guy) leading to prediction as well as running more tests where they are asked to pick a preference of two identical images. The most interesting part is that these results seemed to hinge on pornography. The individuals only exhibited this "precognition or premonition" when they were picking erotic images or rewarded with erotic images (albeit from the International Affective Picture System).

The skeptic in me is very pleased and excited about this part of the paper:

Accordingly, the experiments have been designed to be as simple and transparent as possible, drawing participants from the general population, requiring no instrumentation beyond a desktop computer, taking fewer than 30 minutes per session, and requiring statistical analyses no more complex than a t test across sessions or participants.

Grad students across the country: get to work!

But you would have to have the lottery involve some sort of erotic pictures containing the known numbers in order for this edge to be garnered. Which would be impossible unless the lotteries changed how they worked. Maybe play blackjack with a set of playboy cards? :-)

Re:Reminds me of this hoax...

[SoVeryTired]

http://en.wikipedia.org/wiki/Bogdanov_Affair

The hard sciences aren't immune to this kind of thing either. The Bogdanov affair wasn't as serious as the Sokal affair, but it's still in the same ball park.

Re:Great response paper

[wonkavader]

Dabbling is fine when the results are good. He had 53%. If he'd had 65%, dabbling would have worked. But dabbling is just the start, and that's not just the nature of peer review, it's the nature of collaboration and a University setting. You find something neat by dabbling, and you walk down the hall to visit someone with more stats experience to get some clarity before you publish.

He had 53%. He knew that if he walked down the hall, he'd get told he had squat. So he didn't walk down the hall.

There's dabble initially, and that's fine. And there's dabbling (ONLY) and calling it done. That's not.

Seems like the paper was written by a dabbler, then reviewed by a respected team of dabblers. And not one of them looked at 53% and walked down the hall. Bubbleheads.