powrogers - We are indeed submitting it as a letter/commentary at one of the major neuroimaging journals. We feel that is the proper way to address the topic, not as though we have discovered something new. The poster was a little more sarcastic in that regard, but the paper/commentary is very straightforward.
I would prefer not to name journal names at this time, since we are just now finishing up our complete review of all 2008 articles in seven major journals. Suffice it to say that if you are in the field of neuroimaging you have probably read a paper from these sources. You are right that the trend has been very good in terms of requiring new papers to have correction. Our end goal is to make it required unless there is a justifiable reason not to.
venicebeach - Again, good points. The trouble is that multiple comparisons correction is not the de dacto standard in any neuroimaging journal. Some journals, like NeuroImage and HBM, have become quite good about requiring correction in the results. Still, even they are not 100%. Other journals with a lower impact factor are quite a bit worse, with uncorrected statistics used in almost 50% of the studies. So, either people know about the problem and are willingly choosing to ignore it when they publish or they are unaware of the seriousness of the problem and need a salient reason to begin correcting. We believe it is the latter, which is why we published the Salmon.
As for the argument about it being counter-productive, I fully agree. We presented the poster at the Organization for Human Brain Mapping meeting last June, which was our target audience. I then uploaded the poster to my website so those researchers could grab a copy. The poster got picked up by a few weblogs and eventually spiraled into what you see on Slashdot. We were quite content to publish the paper in a sleepy corner of neuroimaging and wanted it to remain as a discussion piece among scientists.
powrogers - You are right that the conclusions were made many years ago. So, why does a sizable percentage (up to 50% in some journals) of imaging results still report only uncorrected statistics? That is our motivation with the Salmon poster - to get all fMRI researchers on board in using multiple comparisons correction in their work. I would agree that the poster has little in terms of scientific novelty, but its significance to the field lies in helping to set proper standards publishing fMRI results. Correction should be mandatory, unless you have a seriously good reason not to.
Also, no, we didn't cover autocorrelation. We thought we would take it one statistical-issue-that-people-don't-seem-to-correct-for at a time.:)
AC - You are incorrect when you state that nobody in fMRI would publish without FDR or FWER correction. The percentage of articles published using uncorrected statistics is still quite high, which is the entire reason why we published the Salmon results. The big fMRI journals like NeuroImage and HBM are pretty good these days, but I would still challenge you to look through one edition and not see some uncorrected statistics. The problem is worse depending on what journal you read. The whole point we are trying to convey is that uncorrected thresholds and minimum cluster sizes are an inappropriate control for the multiple comparisons problem and that all researchers should be doing it with their data.
AC - Our poster/paper is not about proving definitively the necessity of multiple comparisons correction. You are correct that this has already been done by folks like Benjamini, Hochberg, Friston, and Worsley (to name a few) - they all tackled this issue back in the 90s. Our commentary is targeted at the sizable fraction of individuals who do not use multiple comparisons correction for their fMRI results. You are right that we don't add a lot that is new to the technical discussion of why correction is necessary. However, we are of the opinion that the Salmon poster adds a great deal to the debate regarding why everyone should be using correction on their own results. Hopefully you see the distinction.
daenris - Great clarification between the multiple comparisons problem and the non-independence error. The only thing I would add is that while the majority of published fMRI papers do correction there is still a sizable minority that do not. That is why we wrote up the salmon results as a poster.
ardeaem - At face value you are absolutely right. The majority of cognitive neuroscientists do use multiple comparisons correction in their research. Our commentary is targeted at the remainder of researchers who continue to use uncorrected statistics. The percentage is larger than you might believe, and my co-authors and I are of the opinion that we need to get our statistical house in order for the field to mature.
venicebeach - It is good to see some other imagers commenting on the poster. The entire point of our commentary is that you should be using FDR and FWER in your research. These methods address the multiple comparisons problem in fMRI and allow you to report what the probability of a false positive is across the whole brain. Simply having a high threshold (p-value) and a minimum cluster size (8 voxels) is an unknown control for multiple comparisons that may, or may not, be appropriate for your data.
You point about 'when you do your thresholding wrong you get meaningless results' is spot on. A sizable fraction of reported studies do not use multiple comparisons correction. This poster, and our forthcoming paper, argue that they should.
owlstead - I hear you - I have been a fellow/. reader for years and have observed firsthand the waxing and waning of articles. The above post was mostly a courtesy if anyone was genuinely curious about some aspect of the poster. That and I felt somewhat compelled to post a comment - as a longtime reader it is quite an honor to see some aspect of your own work on the Slashdot main page, even if it was for a dead fish.
powrogers - Thanks for stopping by our poster last June. I like your comment quite a bit but would add one point. While the correction of multiple comparisons in fMRI has been well understood for quite some time (you mention 15 years) the current problem is that not everyone does it while conducting their research. Having a high p-value and a minimum cluster threshold is an unknown, soft control to the problem. Our argument is that true correction methods that control for the FDR or FWER should be employed in standard fMRI experiments.
joepa - You have a lot of very good points. Most neuroscientists are aware of the multiple comparisons problem and, at minimum, try to control for it using increased statistical thresholds (high p-values) and minimum clustering values (have to have several contiguous voxels). The trouble with this approach is that it is a soft control of the multiple comparisons problem. You still have no idea of what the false positive rate will be across the whole brain, only on a quasi voxel-to-voxel basis. Using techniques like false discovery rate (FDR) or Gaussian random field familywise error correction (FWER) you are able to have a much stronger case regarding what degree of your results are true or false.
You are also correct that a majority of neuroscience results are corrected using FDR, FWE, or another correction method like permutation. The trouble is that a sizable fraction of articles still report values that are uncorrected. The Salmon paper is our argument that most, if not all, fMRI research needs strong multiple comparisons correction.
gweihir - We are on our second round of reviews at a major neuroscience journal and things are looking good for getting it published. A lot of our trouble has come from individuals who don't want multiple comparisons correction to become a mandatory practice in functional imaging.
AC - The paper has been rejected once so far. I won't mention the journal, but it was rejected on an editorial basis before it reached the peer review stage. I can only conjecture regarding why the editor decided to pass on the paper, but it was not (to my knowledge) rejected for any methodological deficiencies. We are currently in the review stage at a second journal and the reviewers had no trouble with our methods, only how we argue for multiple comparisons correction without stepping on too many toes.
As an interesting aside, the poster was also rejected at first. All the peer reviewers thought is was a joke and voted to exclude it from the conference. Once it went before the program committee they realized that, even though we had an odd approach, the conclusions of our data were sound and that we had a very good point to make.
Noundi - It most likely will be published soon. The paper is just working its way through peer review right now. The last set of journal reviewers were quite kind and had very good feedback for us to improve the Salmon story.
Yali - It is a different statistical issue than the Vul et al. non-independence error. While a great many papers have been written on how to complete multiple comparisons correction in fMRI there is still a problem in that not everyone is doing it. This leaves the door open to false positives, the number of which remain unknown.
Hey guys - I am the first author of the Salmon poster. If you have any questions that you would like us to answer then post it as a reply below and I will do my best to respond as soon as I can.
I think that we might have different definitions of how good or bad EndNote is. As an individual neuroscience researcher who has his entire PDF library referenced in EndNote it does work well enough, not that I like it. You seem to be approaching it from a more technical standpoint of usability across many disciplines and citation types. There EndNote is indeed trash if you routinely require citations not found in EndNote's templates.
As for the technical rigor of the programming, 99% of the citations I use in my manuscripts are books and journal articles. There might be a few conference presentations, personal communications, and whatnot in there as well, but that is about it. This reduces the scope of the problem to something that IS relatively easy to implement software-wise. The goal of CSL seems to be the creation of a catch-all language that can represent any citation. This is a far different problem in terms of difficulty.
EndNote does one thing [citation management] well. The problem is that citation management isn't a difficult thing to accomplish in software. You get some information in one format, store it however you want, and then spit it out according to another format when you are done.
I am sure that EndNote is a cash cow for Thompson, but the gravy train can't last forever. Other free (Zotero) and non-free (Papers) alternatives are becoming increasingly available - and they are far better than EndNote. Suing the competition won't make that problem go away.
You know, the idea of investigating a younger group of people with adult-level responsibilities is the number one suggestion we have gotten since the press release went out. Many have cited the exact situation you describe - moving to the workforce or having other adult responsibilities as a teen. One individual who wrote to us cited historical cases, such as Alexander the Great, Napoleon, Constantine, Cleopatra, Marc Anthony, and Julius Caesar, as other prime examples of people who had great environmental demands placed on them early on.
You have a very astute observation that just because a brain is still maturing doesn't mean that the behavior of an individual will be immature. Speaking in general, the brain has a remarkable ability to rise to the challenges placed before it. Still, we don't have a firm grasp on what effect such challenges have on the developing brain. More work is needed.
As it stands we are continuing to follow the participants who were scanned as part of the Human Brain Mapping paper. They are all juniors now and will be graduating in a little over a year! We are also looking at beginning anew with a fresh group of high school seniors who have different aspirations after graduating. We want to bring folks in who are going to college, heading to the workforce, going to tech school, going into the military, or just staying at home and living with mom and dad. In short, we want to determine if having environmental challenges play any role in ongoing brain maturation.
I appreciate your comment Frank. Good to hear from you!
The Schwartz et al.* paper investigating functional changes in the brain arising from the use of cognitive behavioral therapy (CBT) is a classic, so I can imagine that his book would be quite good.
In the last 15 years there have been a number of similar studies looking at CBT effects in OCD and other disorders, such as phobias. The results of all of these studies underscore the idea that the brain is in a constant state of change. The way your brain operates today will not be entirely similar to how it operates a year from now, or a decade from now. The especially cool part? Research shows that you can play some part in shaping that change. It gives a whole new definition to "meta-cognition".
The nature our results varies somewhat from the above-mentioned effects. We believe that the changes observed in our study are due to both normal development and environmental provocation. While the brain is plastic to a degree throughout the lifetime, it is most plastic and malleable while it is still developing. What our results indicate is that this development, and consequent increased malleability, continues well past the age of 18.
* Schwartz et al 1996. J.M. Schwartz, P.W. Stoessel, L.R. Baxter, K.M. Martin and M.E. Phelps , Systematic changes in cerebral glucose metabolic rate after successful behavior modification treatment of obsessive-compulsive disorder. Arch. Gen. Psychiatry 53 (1996), pp. 109-113.
You couldn't be more correct about the press release.
The sample size of ~20 is small when compared to many other studies. Oddly enough though, it is above average for many MRI and fMRI experiments. Several of the studies we built off of have even smaller sample sizes. Gogtay et al., 2004*, a very good paper, had a sample size of 13! This doesn't justify the low number, but does give you an idea regarding normal study sizes.
We had wanted to end up with usuable data for 40 subjects. We scanned 50 subjects for our time1 acquisition, which we thought would be plenty to keep our numbers high after our time2 scan. In the end subject dropout, scanner breakdowns, and just plain old bad data conspired to limit the sample to 19.
* Gogtay, N. et al. Dynamic mapping of human cortical development during childhood through early adulthood. Proc Natl Acad Sci U S A 101, 8174-9 (2004).
Slashdot Mirror
powrogers - We are indeed submitting it as a letter/commentary at one of the major neuroimaging journals. We feel that is the proper way to address the topic, not as though we have discovered something new. The poster was a little more sarcastic in that regard, but the paper/commentary is very straightforward.
I would prefer not to name journal names at this time, since we are just now finishing up our complete review of all 2008 articles in seven major journals. Suffice it to say that if you are in the field of neuroimaging you have probably read a paper from these sources. You are right that the trend has been very good in terms of requiring new papers to have correction. Our end goal is to make it required unless there is a justifiable reason not to.
venicebeach - Again, good points. The trouble is that multiple comparisons correction is not the de dacto standard in any neuroimaging journal. Some journals, like NeuroImage and HBM, have become quite good about requiring correction in the results. Still, even they are not 100%. Other journals with a lower impact factor are quite a bit worse, with uncorrected statistics used in almost 50% of the studies. So, either people know about the problem and are willingly choosing to ignore it when they publish or they are unaware of the seriousness of the problem and need a salient reason to begin correcting. We believe it is the latter, which is why we published the Salmon.
As for the argument about it being counter-productive, I fully agree. We presented the poster at the Organization for Human Brain Mapping meeting last June, which was our target audience. I then uploaded the poster to my website so those researchers could grab a copy. The poster got picked up by a few weblogs and eventually spiraled into what you see on Slashdot. We were quite content to publish the paper in a sleepy corner of neuroimaging and wanted it to remain as a discussion piece among scientists.
Yeah, it is making the job of scanning through the comments a bit difficult. It is what it is though - doesn't make me love Slashdot any less...
powrogers - You are right that the conclusions were made many years ago. So, why does a sizable percentage (up to 50% in some journals) of imaging results still report only uncorrected statistics? That is our motivation with the Salmon poster - to get all fMRI researchers on board in using multiple comparisons correction in their work. I would agree that the poster has little in terms of scientific novelty, but its significance to the field lies in helping to set proper standards publishing fMRI results. Correction should be mandatory, unless you have a seriously good reason not to.
Also, no, we didn't cover autocorrelation. We thought we would take it one statistical-issue-that-people-don't-seem-to-correct-for at a time. :)
AC - You are incorrect when you state that nobody in fMRI would publish without FDR or FWER correction. The percentage of articles published using uncorrected statistics is still quite high, which is the entire reason why we published the Salmon results. The big fMRI journals like NeuroImage and HBM are pretty good these days, but I would still challenge you to look through one edition and not see some uncorrected statistics. The problem is worse depending on what journal you read. The whole point we are trying to convey is that uncorrected thresholds and minimum cluster sizes are an inappropriate control for the multiple comparisons problem and that all researchers should be doing it with their data.
AC - Our poster/paper is not about proving definitively the necessity of multiple comparisons correction. You are correct that this has already been done by folks like Benjamini, Hochberg, Friston, and Worsley (to name a few) - they all tackled this issue back in the 90s. Our commentary is targeted at the sizable fraction of individuals who do not use multiple comparisons correction for their fMRI results. You are right that we don't add a lot that is new to the technical discussion of why correction is necessary. However, we are of the opinion that the Salmon poster adds a great deal to the debate regarding why everyone should be using correction on their own results. Hopefully you see the distinction.
daenris - Great clarification between the multiple comparisons problem and the non-independence error. The only thing I would add is that while the majority of published fMRI papers do correction there is still a sizable minority that do not. That is why we wrote up the salmon results as a poster.
kozar - That is almost exactly how we prepared the salmon that we scanned. It was delicious.
ardeaem - At face value you are absolutely right. The majority of cognitive neuroscientists do use multiple comparisons correction in their research. Our commentary is targeted at the remainder of researchers who continue to use uncorrected statistics. The percentage is larger than you might believe, and my co-authors and I are of the opinion that we need to get our statistical house in order for the field to mature.
venicebeach - It is good to see some other imagers commenting on the poster. The entire point of our commentary is that you should be using FDR and FWER in your research. These methods address the multiple comparisons problem in fMRI and allow you to report what the probability of a false positive is across the whole brain. Simply having a high threshold (p-value) and a minimum cluster size (8 voxels) is an unknown control for multiple comparisons that may, or may not, be appropriate for your data.
You point about 'when you do your thresholding wrong you get meaningless results' is spot on. A sizable fraction of reported studies do not use multiple comparisons correction. This poster, and our forthcoming paper, argue that they should.
grcumb - I am the author of the Salmon poster, and I wish I had some mod points for your comment. Awesome.
owlstead - I hear you - I have been a fellow /. reader for years and have observed firsthand the waxing and waning of articles. The above post was mostly a courtesy if anyone was genuinely curious about some aspect of the poster. That and I felt somewhat compelled to post a comment - as a longtime reader it is quite an honor to see some aspect of your own work on the Slashdot main page, even if it was for a dead fish.
Thanks.
powrogers - Thanks for stopping by our poster last June. I like your comment quite a bit but would add one point. While the correction of multiple comparisons in fMRI has been well understood for quite some time (you mention 15 years) the current problem is that not everyone does it while conducting their research. Having a high p-value and a minimum cluster threshold is an unknown, soft control to the problem. Our argument is that true correction methods that control for the FDR or FWER should be employed in standard fMRI experiments.
joepa - You have a lot of very good points. Most neuroscientists are aware of the multiple comparisons problem and, at minimum, try to control for it using increased statistical thresholds (high p-values) and minimum clustering values (have to have several contiguous voxels). The trouble with this approach is that it is a soft control of the multiple comparisons problem. You still have no idea of what the false positive rate will be across the whole brain, only on a quasi voxel-to-voxel basis. Using techniques like false discovery rate (FDR) or Gaussian random field familywise error correction (FWER) you are able to have a much stronger case regarding what degree of your results are true or false.
You are also correct that a majority of neuroscience results are corrected using FDR, FWE, or another correction method like permutation. The trouble is that a sizable fraction of articles still report values that are uncorrected. The Salmon paper is our argument that most, if not all, fMRI research needs strong multiple comparisons correction.
gweihir - We are on our second round of reviews at a major neuroscience journal and things are looking good for getting it published. A lot of our trouble has come from individuals who don't want multiple comparisons correction to become a mandatory practice in functional imaging.
Ma8thew is correct - the fish had been dead for some time. I purchased it on a Saturday morning, so it was likely from Friday's shipment of seafood.
AC - The paper has been rejected once so far. I won't mention the journal, but it was rejected on an editorial basis before it reached the peer review stage. I can only conjecture regarding why the editor decided to pass on the paper, but it was not (to my knowledge) rejected for any methodological deficiencies. We are currently in the review stage at a second journal and the reviewers had no trouble with our methods, only how we argue for multiple comparisons correction without stepping on too many toes.
As an interesting aside, the poster was also rejected at first. All the peer reviewers thought is was a joke and voted to exclude it from the conference. Once it went before the program committee they realized that, even though we had an odd approach, the conclusions of our data were sound and that we had a very good point to make.
Noundi - It most likely will be published soon. The paper is just working its way through peer review right now. The last set of journal reviewers were quite kind and had very good feedback for us to improve the Salmon story.
Yali - It is a different statistical issue than the Vul et al. non-independence error. While a great many papers have been written on how to complete multiple comparisons correction in fMRI there is still a problem in that not everyone is doing it. This leaves the door open to false positives, the number of which remain unknown.
Hey guys - I am the first author of the Salmon poster. If you have any questions that you would like us to answer then post it as a reply below and I will do my best to respond as soon as I can.
You can find some more information on the poster at the following link:
http://prefrontal.org/blog/2009/06/atlantic-salmon-index/
Best ~ Craig Bennett
I think that we might have different definitions of how good or bad EndNote is. As an individual neuroscience researcher who has his entire PDF library referenced in EndNote it does work well enough, not that I like it. You seem to be approaching it from a more technical standpoint of usability across many disciplines and citation types. There EndNote is indeed trash if you routinely require citations not found in EndNote's templates.
As for the technical rigor of the programming, 99% of the citations I use in my manuscripts are books and journal articles. There might be a few conference presentations, personal communications, and whatnot in there as well, but that is about it. This reduces the scope of the problem to something that IS relatively easy to implement software-wise. The goal of CSL seems to be the creation of a catch-all language that can represent any citation. This is a far different problem in terms of difficulty.
EndNote does one thing [citation management] well. The problem is that citation management isn't a difficult thing to accomplish in software. You get some information in one format, store it however you want, and then spit it out according to another format when you are done.
I am sure that EndNote is a cash cow for Thompson, but the gravy train can't last forever. Other free (Zotero) and non-free (Papers) alternatives are becoming increasingly available - and they are far better than EndNote. Suing the competition won't make that problem go away.
You know, the idea of investigating a younger group of people with adult-level responsibilities is the number one suggestion we have gotten since the press release went out. Many have cited the exact situation you describe - moving to the workforce or having other adult responsibilities as a teen. One individual who wrote to us cited historical cases, such as Alexander the Great, Napoleon, Constantine, Cleopatra, Marc Anthony, and Julius Caesar, as other prime examples of people who had great environmental demands placed on them early on.
You have a very astute observation that just because a brain is still maturing doesn't mean that the behavior of an individual will be immature. Speaking in general, the brain has a remarkable ability to rise to the challenges placed before it. Still, we don't have a firm grasp on what effect such challenges have on the developing brain. More work is needed.
As it stands we are continuing to follow the participants who were scanned as part of the Human Brain Mapping paper. They are all juniors now and will be graduating in a little over a year! We are also looking at beginning anew with a fresh group of high school seniors who have different aspirations after graduating. We want to bring folks in who are going to college, heading to the workforce, going to tech school, going into the military, or just staying at home and living with mom and dad. In short, we want to determine if having environmental challenges play any role in ongoing brain maturation.
I appreciate your comment Frank. Good to hear from you!
~Craig
The Schwartz et al.* paper investigating functional changes in the brain arising from the use of cognitive behavioral therapy (CBT) is a classic, so I can imagine that his book would be quite good.
In the last 15 years there have been a number of similar studies looking at CBT effects in OCD and other disorders, such as phobias. The results of all of these studies underscore the idea that the brain is in a constant state of change. The way your brain operates today will not be entirely similar to how it operates a year from now, or a decade from now. The especially cool part? Research shows that you can play some part in shaping that change. It gives a whole new definition to "meta-cognition".
The nature our results varies somewhat from the above-mentioned effects. We believe that the changes observed in our study are due to both normal development and environmental provocation. While the brain is plastic to a degree throughout the lifetime, it is most plastic and malleable while it is still developing. What our results indicate is that this development, and consequent increased malleability, continues well past the age of 18.
* Schwartz et al 1996. J.M. Schwartz, P.W. Stoessel, L.R. Baxter, K.M. Martin and M.E. Phelps , Systematic changes in cerebral glucose metabolic rate after successful behavior modification treatment of obsessive-compulsive disorder. Arch. Gen. Psychiatry 53 (1996), pp. 109-113.
You couldn't be more correct about the press release.
The sample size of ~20 is small when compared to many other studies. Oddly enough though, it is above average for many MRI and fMRI experiments. Several of the studies we built off of have even smaller sample sizes. Gogtay et al., 2004*, a very good paper, had a sample size of 13! This doesn't justify the low number, but does give you an idea regarding normal study sizes.
We had wanted to end up with usuable data for 40 subjects. We scanned 50 subjects for our time1 acquisition, which we thought would be plenty to keep our numbers high after our time2 scan. In the end subject dropout, scanner breakdowns, and just plain old bad data conspired to limit the sample to 19.
* Gogtay, N. et al. Dynamic mapping of human cortical development during childhood through early adulthood. Proc Natl Acad Sci U S A 101, 8174-9 (2004).
The linked Slashdot article is just the College Relations department press release. You can look at the full prepress PDF at the following address:
f s/2005-Bennett-VBM.pdf
http://www.theteenbrain.com/about/publications/pd