Journal Article On Precognition Sparks Outrage

thomst writes "The New York Times has an article (cookies and free subscription required) about the protests generated by The Journal of Personality and Social Psychology's decision to accept for publication later this year an article (PDF format) on precognition (the Times erroneously calls it ESP). Complaints center around the peer reviewers, none of whom is an expert in statistical analysis."

Comments

Prediction

[Anonymous Coward]

I predicted this would happen.

Why Is It Wrong to Call This ESP?

[eldavojohn]

on precognition (the Times erroneously calls it ESP).

Why is that erroneous? Precognition and premonition are two facets of Extrasensory Perception. From its wikipedia article:

Extrasensory perception (ESP) involves reception of information not gained through the recognized physical senses but sensed with the mind. The term was coined by Sir Richard Burton,[citation needed] and adopted by Duke University psychologist J. B. Rhine to denote psychic abilities such as telepathy and clairvoyance, and their trans-temporal operation as precognition or retrocognition.

So if you were dealing with anything of the above or anything external to our normal senses, I think that qualifies as ESP and calling it ESP. Sure that acronym has a lot of baggage but from the study itself:

... this is an experiment that tests for ESP (Extrasensory Perception).

That's what the tests subjects were told and I don't think the article is erroneous.

Re:Why Is It Wrong to Call This ESP?

[prionic6]

I'd say it is more a case of earth having 4 corners in some kind of a simultaneous 4-day... In only 24 hours of rotation, there are 4 corner days, cubes for a quad earth. No 1 Day God.

Re:Why Is It Wrong to Call This ESP?

[AlecC]

From the summary, the implication is that the data analysis was not proper - or at least, not shown to be proper. Since the claimed effect is a fairly small artifact only detectable by sophistcated statistics, it seems reasonable that the reviewers should include those who have a deep understanding of such statistics - which, it is claimed, they did not.

Re:Why Is It Wrong to Call This ESP?

[jason.sweet]

FTA:

In one sense, it is a historically familiar pattern. For more than a century, researchers have conducted hundreds of tests to detect ESP, telekinesis and other such things, and when such studies have surfaced, skeptics have been quick to shoot holes in them.

I always thought the hole-shooting was an essential step in the scientific process. If you can't patch up the holes, you aren't really doing "science."

In science, you tell a story. Then everyone says, "no, that's wrong because..." Then you say, "I'm afraid I'm right, because..." It is an imperfect process because people have biases that are hard to overcome. But, if the empirical evidence is strong enough, you will overcome these imperfections. In the end, you build a consensus by presenting enough evidence that no one can argue with. I'm not sure what is more democratic than that.

Re:Why Is It Wrong to Call This ESP?

[Black Parrot]

I don't understand this. If the researchers did a proper experiment, respected the rules, followed proper procedures, and did a proper analysis of the data they collected, in a scientific way. Why is it a problem to publish ?

Shouldn't be any. However, this case isn't relevant to your question, since they didn't do a proper analysis. (Or at least that's what the rebuttals say; I haven't read the paper.)

One of the most glaring problems is that they (reportedly) went fishing for statistical significance in their results, without making the correction that is required for rigor when you do that. When you find significance at the traditonal 95% confidence level, there's a 5% chance that you're finding meaning in noise. If you test for 20 different effects and then go fishing to see whether *any* of them show significance at the 95% confidence level, you have 1 - .95^20 = 64% chance of finding "significance" in noise.

There are simple ways to fix that problem, e.g. the Tukey HSD "honestly statistically different" test. Apparently the authors were either ignorant or dishonest, and the reviewers were either ignorant, dishonest, are careless.

Re:Why Is It Wrong to Call This ESP?

[ae1294]

Wait, you've lost me here... what about human beings can't be explained with classical physics?

2 girls 1 cup

Great response paper

[246o1]

For those who haven't seen it, here's a pretty sharp takedown of this paper, as well as some notes on statistical significance in social sciences in general: www.ruudwetzels.com/articles/Wagenmakersetal_subm.pdf

Re:Great response paper

[Anonymous Coward]

And for those who still haven't seen it, here's a proper link.

Re:Great response paper

[ledow]

Ouch. Taken down by two Bayesian tests on whether it's more likely that the paper is true or not. They didn't even need to get out of bed or dig out a big maths book to basically disprove the entire premise of the original paper using its own data.

As they hint at in that rebuttal - As a mathematician and someone of a scientific mind, I would just like to see *ONE* good test that conclusively shuts people up. Trouble is, no good test will report a false result and thus you'll never get the psychic / UFO / religion factions to even participate, let alone agree on the method of testing because they would have to accept its findings.

Never dabble in statistics - the experts will roundly berate you and correct you even if you *THINK* you're doing everything right. When PhD's can't even work out things like the Monty Hall Problem properly, you just know it's not something you can throw an amateur towards.

Re:Great response paper

[Burnhard]

But isn't that simply applying the maxim: extraordinary claims require extraordinary evidence? I.e. it is not asserting the evidence doesn't exist, only that the evidence isn't strong enough to make the assertion that the hypothesis is true with any confidence. There's a subtle difference.

Re:Great response paper

[wonkavader]

Dabbling is fine when the results are good. He had 53%. If he'd had 65%, dabbling would have worked. But dabbling is just the start, and that's not just the nature of peer review, it's the nature of collaboration and a University setting. You find something neat by dabbling, and you walk down the hall to visit someone with more stats experience to get some clarity before you publish.

He had 53%. He knew that if he walked down the hall, he'd get told he had squat. So he didn't walk down the hall.

There's dabble initially, and that's fine. And there's dabbling (ONLY) and calling it done. That's not.

Seems like the paper was written by a dabbler, then reviewed by a respected team of dabblers. And not one of them looked at 53% and walked down the hall. Bubbleheads.

Research Funding

[MoonBuggy]

I haven't yet had a chance to read the paper fully (it's 50 or so pages), but if they are actually that confident in their evidence that precognition has been found, the James Randi Foundation has a million dollars waiting for them.

Re:Research Funding

[Rob Kaper]

Any evidence for precognition instantly takes precognition into the scientific realm and out of the paranormal, supernatural and occult.

Re:Research Funding

[elrous0]

If they exist, I want them arrested immediately for aiding and abetting terrorists on September 10, 2001. Obviously, they all willfully stayed silent.

Erotic Pictures a Necessary Element

[eldavojohn]

Why settle for just 1M? Play the lottery for a few weeks and you don't even have to bother writing an article to be rich.

So I realize a lot of people aren't going to read the article but here's the meaty parts for you statistics snobs (and really, the Bayes folks are going to be all over this one):

Across all 100 sessions, participants correctly identified the future position of the erotic pictures significantly more frequently than the 50% hit rate expected by chance: 53.1%, t(99) = 2.51, p = .01, d = 0.25.3 In contrast, their hit rate on the nonerotic pictures did not differ significantly from chance: 49.8%, t(99) = -0.15, p = .56. This was true across all types of nonerotic pictures: neutral pictures, 49.6%; negative pictures, 51.3%; positive pictures, 49.4%; and romantic but nonerotic pictures, 50.2%. (All t values < 1.) The difference between erotic and nonerotic trials was itself significant, tdiff(99) = 1.85, p = .031, d = 0.19.

There's a lot more about eliminating random number generators (by using this little guy) leading to prediction as well as running more tests where they are asked to pick a preference of two identical images. The most interesting part is that these results seemed to hinge on pornography. The individuals only exhibited this "precognition or premonition" when they were picking erotic images or rewarded with erotic images (albeit from the International Affective Picture System).

The skeptic in me is very pleased and excited about this part of the paper:

Accordingly, the experiments have been designed to be as simple and transparent as possible, drawing participants from the general population, requiring no instrumentation beyond a desktop computer, taking fewer than 30 minutes per session, and requiring statistical analyses no more complex than a t test across sessions or participants.

Grad students across the country: get to work!

But you would have to have the lottery involve some sort of erotic pictures containing the known numbers in order for this edge to be garnered. Which would be impossible unless the lotteries changed how they worked. Maybe play blackjack with a set of playboy cards? :-)

Journal Article On Precognition Sparks Outrage...

[knewter]

The headline SO should have been:

Journal Article On Precognition Sparks Outrage BEFORE IT'S PUBLISHED.

That is all. I expect more out of an editor.

Re:Peer review only provides weak prescreening

[gweihir]

I have a PhD in CS.

You can also have good science rejected by getting three incompetent reviewers. Happened to me several times, the worst one when the program committee attached a note that showed they had not read or understood their own call for papers. I suspect a direct lie to keep me out. Published it later unchanged somewhere else and those people were surprised it got rejected earlier.

In addition to incompetent reviewers, there are also those that are envious or want to steal your ideas. Peer-review is fundamentally broken. One friend who has a PhD in a different CS area thinks 70% of researchers are corrupt, reviewing things positively when they know the authors, no matter the quality and negatively otherwise. Lying in application to research grants is also quite common. The final result is that good researchers have trouble working and often leave research altogether, which may be an explanation for how glacially slow some fields move.