Slashdot Mirror
MRI Software Bugs Could Upend Years Of Research (theregister.co.uk)
An anonymous reader shares a report on The Register: A whole pile of "this is how your brain looks like" MRI-based science has been invalidated because someone finally got around to checking the data. The problem is simple: to get from a high-resolution magnetic resonance imaging scan of the brain to a scientific conclusion, the brain is divided into tiny "voxels". Software, rather than humans, then scans the voxels looking for clusters. When you see a claim that "scientists know when you're about to move an arm: these images prove it", they're interpreting what they're told by the statistical software. Now, boffins from Sweden and the UK have cast doubt on the quality of the science, because of problems with the statistical software: it produces way too many false positives. In this paper at PNAS, they write: "the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%. These results question the validity of some 40,000 fMRI studies and may have a large impact on the interpretation of neuroimaging results."
The research is on fMRI - the F stands for Functional. As it mentions later in the summary this is used to try to associate regions of the brain with specific functions. This is not the same as the structure of the brain itself. What we see in terms of actual brain structures - folds, regions, etc, is still very much valid. We're just not so sure about the functional assignments that we've held on to for a while now.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
Wouldn't the brain itself produce a large number of false positives?
Like... yes, I am about to move my arm... I'm about to... aaaand,
Nope! Hah, stupid scientists!
It's not exactly new this issue. Through the link is a study of the active regions of the brain of a dead salmon....
http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf
Is not it great, that such Scientific scepticism remains legal and the raw data remains available for anyone trying to replicate the earlier findings of others?
In Soviet Washington the swamp drains you.
A friend of mine as worked in the social sciences (cue /. laughter, but bear with me) and they were forced by the university to use a closed source statistical package for all their data processing. So anyway, she got some really dubious results and she preferred to do her own maths, so she did, and lo! completely different results. That was the start of a research project which concluded that the closed source package contained a rounding error that basically filtered all minorities out of the data set, which is kind of sad if you're doing research on minorities.
People trust their software too much, are too lazy to do their own maths, don't really want to have got anything to do with data processing even though that's their job, and universities force bad software on their employees. This is an institutional problem that goes way beyond MRI research.
The part that is more worrisome. Is that the software ecosystem for this is so small that it seems to affect across many MRI vendors. I mean if you are are going to do a scientific study. You should make sure your results are calculated from different software.
Open Source or not probably isn't the big issue, but the fact that so many researchers were using the same software.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
On first blush this seems as bad a mistake as the guy who programmed a Mars lander in yards instead of metres and as Robin Williams put it 'buried that sucker'...given the expanse of the issue though it could be far worse. Furthermore, who will go back and reanalyze all that data to vet their applicability & conclusions? It's unlikely that researchers who may have made whole careers over the results will do it. So how long are we going to hang on to potentially incorrect conclusions? Potentially decades.
It's a matter of time before this happens with global warming, too. It's well known that the temperature record is adjusted, supposedly to remove biases. However, if you look at the unadjusted data, it fits the solar cycle perfectly, with temperatures declining over the past few decades, coinciding with solar dimming. The adjustment looks like a hockey stick, though, which can explain the entirety of the supposed warming. The National Climatic Data Center once had these figures on their website, though they've conveniently been removed. However, this is an example of how systematic errors can set an entire scientific field back many years. It's a matter of time before this happens with global warming, too.
The paper has been available as a preprint for awhile now, and my lab has discussed it internally and I've also paid attention to outside coverage. The key issue that the paper reports is that false positive rates are two high for most existing software WHEN using a specific type of test under a specific set of conditions. They show that voxelwise familywise error (FWE) correction actually seems to work reasonably or even conservatively. Cluster level FWE correction (looking for groups of voxels that are active) fails when using a very liberal cluster-defining threshold, but works reasonably well when using a more stringent cluster defining threshold. It also says nothing about the performance of another very common correction method that is frequently used in fMRI studies (false discovery rate or FDR).
I'm not really sure how extensive the group of findings that these issues actually affect is, but it's certainly not 40,000 as is claimed in the paper's significance section. Many of the earlier papers (and even more recent) likely used uncorrected statistical tests, so are suspect for entirely different reasons from this issue. Of the ones that use correction, the findings in this paper only call into question the results for those that are using FWE cluster correction with a cluster defining threshold that is too liberal (likely > 0.001, the paper's findings suggest that at 0.001 the familywise error rate is in the ballpark of the desired 5%). Those using a cluster defining threshold of p=0.001 or lower are likely fine, and those using a different correction method like FDR are unknown as to my knowledge there isn't currently any similar paper on that correction method.
You can also check out this technical report by some other big names in imaging that basically says that this result is known and expected for overly liberal cluster defining thresholds:
http://www.fil.ion.ucl.ac.uk/s...
Is why the earth is flat.
How could they possibly be wrong?
The science was SETTLED!!!
i've been wanting to learn R and thought about doing some maths on the raw data and compare it with the released results. mostly looking at trends at specific weather stations compared to official numbers
Don't get me wrong. Those people are probably doing great research but most of them have no idea about statistics. Software offering the possibility to change a few parameters so the results get closer to what they want (expect) is all too common.
All of the software packages tested in the article (AFNI, FSL, SPM) are open source, including the package the authors built to do massively parallel non-parametric permutation tests (BROCCOLI).
The researchers used published fMRI results, and along the way they swipe the fMRI community for their “lamentable archiving and data-sharing practices” that prevent most of the discipline's body of work being re-analysed.
So the raw data isn't being saved so that someone else can independently verify the results. No checking the computers math, no checking the researchers settings on the machine. Just blanket trust for the people and the machine, and purging of any way of poking holes in someones findings. Even if this wasn't caused by a software bug the lack of archiving the raw dataset so that it can be rerun when software improvements are made is just infuriating.
Meh! I already those studies (video game make you a psychopath/serial killer etc.) were crap, with an agenda.
When a story embeds the same link three times in a row (once in the mast, then twice in the article text) pretty please with sugar on top display the redundant links with "[register.com]" following the link, just like it does in my configured article view.
Or, clever idea, you could display "[repeat link]" in each case where a link is repeated.
If you're feeling extra ambitious—but you don't wish to interrupt your feverish efforts to deliver proper Unicode support one minute more than absolutely necessary—you might choose, in the short term, to combine both solutions as [repeat link; most probably The Register again]".
If you're using the word Iron and MRI in the same sentence, you're mostly probably doing it wrong. (Yes yes, I realize that's not true for very tiny amounts... but never let facts get in the way of a joke).
/most likely
//damn you slashdot, why you no have edit after submit.
And gravity. Everyone keeps using our type of matter in their experiments, where the inertial mass and the gravitational mass of everything is nearly identical if you use open source software to make the statistical comparisons! But these two masses are only the "same" if you use mathematics which can be reviewed for accuracy. If you use the correct proprietary software (you have to preserve the trade secrets), you'll see the two masses are different (because of ghosts). That's why only properly equipped scienticians, willing to pay for their math, are able to use anti-gravity coherently in their explanations for ghost behavior.
I love it when people run studies to actually verify / build-upon previous results. What I'm really seeing from this article is that there's a lot more "plug numbers into tool" research going on than I first expected. I would've hoped that the tools themselves would output confidence coefficients so that at least the researchers would have a clue as to how much magic they'd come up with...
Bye!
At least for the time being. This kind of imagery led researchers into claiming that free will is an illusion [google 'Sam Harris']. It makes me happy they apparently jumped to a conclusion, without proper validation. The universe seems more coherent right now.
One famous example of error related problems with fMRIs is the infamous brain scan of the dead salmon. I'm not sure if I can post a link but its: http://www.wired.com/images_bl...
Climate Change Software Bugs Could Upend Years Of Research!
We're Heading For An Ice Age!
Repent Sinners!
The claim is quite dubious in that it seems to suggest that scientists know someone is going to move their arm before the person does, simply from reading MRI images.
Not MRI image.
But f MRI images (where f = "functional")
In a nutshell, those image are based around the fact that hemoglobin loaded with oxygen interacts and distorts the magnetic field differently than hemoglobin which has discarded its oxygen.
By measure these signal differences, it's possible to infer where there's more oxygen consumption, and from there try to guess which parts of the brain are working more (and thus consuming more oxygen).
Spatial resolution of such image is "not so great" (blurier thant the brain anatomy visible on the brain itself), but still acceptable.
Reach is very good (you can see the whole inside of the skull).
Signal strength is very weak (very subtle variation, meaning lot's of noise).
Temporal resolution is very poor (to begin with all MRI images take a lot of time to take, and then there's the problem is that you're not measuring brain activity directly, but you're inferring it from its indirect effect on the local blood flow).
Still it's a useful tool under some circumstances.
Compare it with other tools, like measuring electric (EEG) or magnetic activity from ther outside:
Spatial resolution is absolutely shitty (you must infer what's happening from a few points scattered across the surface of the scalp)
Reach isn't deep at all (you mostly see what's happening on the surface. Deep brain structures are too deep to be visible).
Temporal resolution is amazing (you can measure the direct electrical output ms after ms)
The best tool it still open-skull surgeries (using electrodes directly on the brain to measure activities or to very precisely stimulate some area), but they are a rare commodity (= you can only find volunteers to enroll into your studies among people getting brain surgery to remove tumors).
The second best tool is the clinical description of psychiatric damage experienced by people who where victim of accident where their brain was damaged.
EEG and fMRI are coarser tools, but much easier to setup.
In addition to that anatomists and histologist have had tons of other tools to explore the anatomy and connections of the brain.
(regular MRI, dissections of cadavers, study of some virus which climb along the nerves, some freezing-/cracking- based special technique of dissection, some special type of diffusion-MRI, etc. )
I find that hard to believe except possibly in some limited cases.
The whole central nervous system works in stage, from very low level (nerves controlling muscles or nerves fed by receptors) all the way to high-level (processing complex information).
Most of the low-level (i.e.: most of the body, except the eyes, ears and a few other head organs) is connected to a region in the mid of the brain, roughly around where the head band of your headphone goes.
Except for a few preprocessing done in the spine (or in the upper layers of the retina in eyes) the signal is very close to raw (1 point of connection = 1 information about a small group of receptor. Like an edge).
Everything behind this "headband" handle signal input and perception. And the more you get away to the point where nervous tracts connects to the cortex, the more integration and convolution is done with the signal (from edges to shapes to objects like "face recognition") and combination with other signals (associative region, which aren't specific to a single sense and can't be pinpointed down to a precise simple role).
Everything in front of this "headband" handle the signal output and motor control. It has the same overall organisation: the more you move to the front away from the "headband", the more the processing is "high-level" and "multimodal" and handles high level functi
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
I had a university level Statistics "professor" once tell me that I didn't need to know how my calculator created a box plot, etc etc because I could just use someone else's statistics library instead of writing my own. While in general I agree that there is no point in reinventing the wheel, I felt like I ought to learn how such things work.
I do a *ton* of statistical work in my day job, and if I were to write a book or teach a class, I would recommend two things:
1) Always look at the data
2) Always write your own functions
The reason for this has to do with the basic nature of statistics. If you make a mistake in normal software, the error is usually patently visible or benign. Often times the software works fine and does its job and the results are correct, even if it has bugs.
In statistics however, if you make a mistake the results get closer to "random". Statistics is fundamentally an attempt to extract information from data, and if you make a misstep then you get less information, which is equivalent to the data being closer to random. There is no way to tell whether the output is correct - it doesn't crash, it doesn't show an obvious flaw, it just didn't give you any information.
The second thing is to always look at the data.
Many, many, *MANY* theories and research papers make simple assumptions about the data which simply aren't true, and if you can look at the data (in an appropriate visualization), you can avoid some of these pitfalls.
Researchers do linear regression, when a quick glimpse of the data would tell them that it's a curve. Economists assume that if a tiny piece of a function looks linear, the entire function is linear. People do Principle Component Analysis on data that has multiple loci of causes. People use Expectation Maximization and "guess" the number and position of causes. People reverse the conditional.
The list is endless.
You can use someone else's library for mundane things which can be checked. Using a library for a box plot is fine - if it crashes or if the output doesn't *look* right, then use a different library.
For doing actual statistical work, you should *first* code your own functions. You'll get a marvellous hands-on insight and a little intuition about what the results should be.
Once you've done that, you can look at (ie - plot) the data and use your human brain to make a judgement.
Then use the big library. If it doesn't look right, you can investigate further.
Is that the software ecosystem for this is so small that it seems to affect across many MRI vendors.
Nope. It's that the vendor only takes care to write the bit of software that actually controls the MRI machine.
The vendor takes care of the low-level and behind the scene work need to they point where you obtain an image - usually in a standard format like DICOM.
(think about the firmware inside a digital point and shoot camera, which is in charge of controlling the CCD, the flash and the zoom/focus, and whose purpose is to write a JPEG file on the storage media at the end).
Whatever you do with the DICOM out of the machine is up to you.
A doctor could display it using some viewing software to make some clinical conclusion.
It could be stored in archives to be referenced later to see the progression of some condition.
Or you could try to do some stats on it.
(to keep the photography metaphore: you're free to just look to your JPEG, or store it on your Drop Box/Google Drive/iCloud/the Fappen... oops. Or run it through GIMP, or even import it into Blender as texture for some even more elaborate artwork).
There are a lot of viewer software both opensource (Osirix, Aeskulap) and closed source (sometime even provided as part of a deal with the manufacturer of the MRI).
Because the market is much smaller, and because research need to pool their efforts together, there are fewer imaging research software pipelines.
Most of them are usually opensource and organized around specific project of some universities.
(e.g.: the BrainVisa pipeline, FreeSurfer, etc.) which tend to reuse similar building blocks in their pipelines (FSL is used around a lot).
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
We are entering an era of false positivies.
Slashdot coverage here.
See that "Preview" button?
Climate science has been tested and proven over and over again using numerous different methods, which all broadly lead to more or less the same ballpark of conclusion.
The only actually *REAL* controversy that exist among scientific is about the minute details of interpretation (like the exact expected decimals at the end of the predicted number), not about the broad existence of climate change.
From this perspective it's quite normal to have strong scepticism against pseudo-scientist trying to stir controversies around climate change without bringing any new data to the table.
OTOH
fMRI is a rather noisy and low resolution recent method.
Some results have been confirmed multiple time using large studies, and comparing to numerous other methods (like study of brain-accident victims, like tests done in parallel during brain-cancer surgery, like information learned from neuro-anatomy, etc.)
Other information really come from a couple of small studies with very few samples, that aren't replicated yet, nor confirmed by any other methodology. It might be too early to shout "Brain region responsible fro 'XyZ' found !!!!"
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
It will probably only take 20 years for them to come to the same conclusion about the detection of gravity waves
Ah, thanks for clarifying. So it is now Ok, in your opinion, to imprison the remaining deniers and to erase (or otherwise keep inaccessible) the raw data, that has once lead our betters to these universally-accepted conclusions?
Or do you still agree, criminal prosecution of dissenters (however unreasonable they may be themselves) is wrong and unavailability of the data — suspicious?
Please, confirm. Thank you!
In Soviet Washington the swamp drains you.
Except they aren't
https://arxiv.org/pdf/1605.043...
Reproducible and replicable CFD:
it’s harder than you think
But sure enough, they found nothing.
So it is now Ok, in your opinion, to imprison the remaining deniers and to erase (or otherwise keep inaccessible) the raw data, that has once lead our betters to these universally-accepted conclusions?
Did you stop beating your wife ?
It's a matter of time before this happens with global warming, too.
Well financed "skeptics" have been busting a gut for over 20yrs trying to prove your conspiracy theory, they have done nothing but bring the word "skeptic" into disrepute.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
I.e. people seeing what they expecting to see, not what is there. With the huge egos, (but not nearly as large skills) in people doing Neuro-"Science" these days, I am entirely unsurprised. The grand claims about what they know and how things work have been a dead giveaway for years. Things are not that simple in practice.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
I once met a guy doing biochemistry, and one of the most widely used analysis software package was developed by someone in his lab like a decade ago. He told me that he once had a look into the software code, and found a bug that arithmetic mean is used in all places where harmonic mean should be used. The tricky part is that if you use the software to find a general trend, it won't do much harm, but if you want to look for some subtle change, the result could be totally wrong. He said that there have been about 500+ papers citing the software, among them are quite a few papers published in prestigious journals like Science, Nature or Cell. I am not sure whether this is just another researcher's horrible tales or it is true.