Slashdot Mirror
How To Better Verify Scientific Research
Hugh Pickens DOT Com writes "Michael Hiltzik writes in the LA Times that you'd think the one place you can depend on for verifiable facts is science but a few years ago, scientists at Amgen set out to double-check the results of 53 landmark papers in their fields of cancer research and blood biology and found only six could be proved valid. 'The thing that should scare people is that so many of these important published studies turn out to be wrong when they're investigated further,' says Michael Eisen who adds that the drive to land a paper in a top journal encourages researchers to hype their results, especially in the life sciences. Peer review, in which a paper is checked out by eminent scientists before publication, isn't a safeguard because the unpaid reviewers seldom have the time or inclination to examine a study enough to unearth errors or flaws. 'The journals want the papers that make the sexiest claims,' Eisen says. 'And scientists believe that the way you succeed is having splashy papers in Science or Nature — it's not bad for them if a paper turns out to be wrong, if it's gotten a lot of attention.' That's why the National Institutes of Health has launched a project to remake its researchers' approach to publication. Its new PubMed Commons system allows qualified scientists to post ongoing comments about published papers. The goal is to wean scientists from the idea that a cursory, one-time peer review is enough to validate a research study, and substitute a process of continuing scrutiny, so that poor research can be identified quickly and good research can be picked out of the crowd and find a wider audience. 'The demand for sexy results, combined with indifferent follow-up, means that billions of dollars in worldwide resources devoted to finding and developing remedies for the diseases that afflict us all is being thrown down a rathole,' says Hiltzik. 'NIH and the rest of the scientific community are just now waking up to the realization that science has lost its way, and it may take years to get back on the right path.'"
Science is infallible.
More like science is always wrong. Scientists always set out to be less wrong than the last guy, though.
So basically they want to introduce a Slashdot for scientists..
Prepare for a brand new style of flame-wars!
Reminds me of Eric Raymond's 'Given enough eyeballs, all bugs are shallow' law.
I think the real problem is that scientists aren't lending any prestige to reproducing experiments so nobody bothers. Journals want to publish new results, not confirmation. Advisors discourage students from reproducing experiments, which makes sense since they won't be published.
...... and I am happy its finally being acknowledged and tackled more openly.
Follow-up studies are where the validation/replication/testing happens. This is not new. Any decent scientist knows this. Peer review is a filter, but it's a pretty basic sanity check, not a comprehensive evaluation of the work. Once published, that opens a paper and the ideas within it to critique by ALL readers, not only the reviewers. Thus, post-publication is when the real scientific review happens. Peer review merely removes the stuff that isn't formulated, measured, and organized well enough to bother reading it in the first place (i.e. it gets rejected). It's an imperfect process, so sometimes stuff slips through anyway. That's what the follow-up papers are for.
The best way of checking spurious, biased, or erroneous results is for someone else to independently do the same experiment. However there's no money or glory in replication. So nobody does it.
I wonder which will be most amusing, Fox's interpretation of this story or the tardbaggers' interpretation of that. I've already assigned "herp", "derp" and "6,000 years" to hotkeys.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
New Paper Idea: The study of light waves as they traverse the complex structure of a Brazilian volleyball team
It's important to remember that in vivo biology is not all of science. It's a lot harder to know what you're doing in biology. If you want excellent reproducible science, let's just roll balls down inclines, measure that and hope we don't get sick.
Well, it's good to see a major scientific institution waking up to a phenomenon Richard Feynman warned about in the 1970s. Yet it seems to me the proposed solution is a little ad hoc. If scientists want to restore integrity to their field(s) -- and I applaud their efforts to do so -- why aren't they using an experimental approach to do so? I think they should try several things and collect data to find out what actually works.
[Sir Garlon] is the marvellest knight that is now living, for he destroyeth many good knights, for he goeth invisible.
Whenever one of these stories is posted about inaccurate and falsified research papers, it's always a field related to biology. This doesn't seem to be nearly as much of a problem with the hard sciences (physics, chemistry). We should avoid rhetoric like "science has lost its way" since the problem is mostly isolated to one branch of science and such statements only serve as ammo for the anti-science crowd. Disclaimer: I'm a physicist.
Interestingly, the Economist's article on the same points this weeks notes that there is a group specifically devoted to doing replication- the Reproducibility Initiative from PLOS One. They've got a $1.3 million grant from the Arnold Foundation to look at 50 high profile papers in cancer research.
"Seven Deadly Sins? I thought it was a to-do list!"
Took 29 minutes to get from the story being posted to "CLIMATE SCIENTIST ARE LIREZ!!11!!1". You know there are a lot of other branches of science, many of which are far more subjective than climate science.
There's also plenty of data and models out there if you wanted to run your own experiments to confirm or disprove a particular paper or claim. I'd be very interested in reading your counter paper.
In addition to questioning "Climate Change", one could also look at the "Science" of may other fields as suggested. Especially crucial for examination are the many psychological and sociological studes that are used to "guide" public policy. I venture that many are complete loads of crap designed specifically to influence public policy.
...and all the while leaving out that their claims don't even offer any kind of falsification chance, i.e. being nonscientific in the first place.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Or "How Better To Formulate Subject Lines". But "How To Better Formulate Subject Lines"? Ugh. Have to read that three times to parse it.
Physics is not immune to parasitic and mercenary research phenomena either, especially in more exotic areas with great funding potential, such as quantum computing & crypto where exaggerations and self-puffery are common. One might say the whole field is of that kind, since their whole theorizing (which is all they got) rests on the speculative aspects of quantum measurement theory, the foundations of which are still awaiting unambiguous experimental demonstration (such as the "loophoole free" violations of Bell inequalities), for over half century already. Should the experimental failure to confirm the fundamental conjectures persist, the whole field will be recognized as fancily relabeled analog computing (such as D-Wave system).
Far too many people treat science like a religion. If a scientist says it, it must be true, which ironically is the exact opposite of science. As has already been pointed out, all science can do is tell you something DOESN'T work that way. Instead what happens is people latch on to stupid things as if its carved in stone, regardless of how many times over the years it gets proved to be untrue or not entirely correct, they'll latch on to current theory and treat it as if its a law, and won't even blink an eye when the current theory turns out to be wrong and needs modified, the next time around ... mysteriously, it can't be wrong now!!@%!
People who treat science like a religion are just as bad as religious nut jobs, arguably worse since at least the religious nut job is aware that they are basing their thoughts on faith rather than proof.
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
What is not discussed is that in science as in life it's all about incentives. All you have to is look at who is paying for these studies, directly (through research grants) or indirectly (speaking or consulting fees), and things will become much clearer. The biomedical and life sciences are most vulnerable to corruption because the incentives are very high, successful drug/treatments are worth a lot of money. Even unsuccessful ones, given the proper appearance of effectiveness are worth money.
Other sciences are less susceptible because there is no incentive to hype the results, not because those scientists are more ethical. There is two solutions for the problem. One is to remove incentives, which would mean overhauling the whole system of scientific funding. The other is to mandate raw data sharing. This would make it easier for people to reanalyze the data without actually redoing the experimental parts.
A good example of this is Reinhart-Rogoff controversy in economics, where they claimed one thing in their widely publicized 2010 paper (high debt levels impede growth), but their statistical analysis was shown to be riddled with errors, skewing the data to the desired conclusion. This was discovered the when they shared their raw data with a University of Massachusetts grad student. While data sharing would not eliminate these issues it would make is harder to perform "statistical" analysis that introduces biases.
It is amazing how much "it's settled science!" sounds like "God said so!" isn't it?
I know people who regularly drop the "settled science" quips on social media (average intellectual level: ZOMG AGW is totally fer realz!!) -- people I personally know who could not even define the word "science". It's important to remember that there's a gulf between those who understand the method, and those who mindlessly parrot whatever is popular. The latter we call religion.
That I'm right, and you don't like it, doesn't mean I'm a troll.
A scientific 'theory' is proven. A 'hypothesis' is still uncertain.
Or do you not believe in the theory of gravity either?
People in cars cause accidents....accidents in cars cause people
Really there is a simple reason for this. (Of course, it's not the only one but presumably the primary one.)
Tenure-track positions and funding are to a large extent determined on the basis of the number of publications weighted by the reputation of the journals, not by the quality of publications.
The idea is that good journals will reject bad papers, which doesn't work as well as is desirable due to the extreme amount of submissions the journals receive, which have to be reviewed by relatively small numbers of unpaid voluntary reviewers.
There are many ways this problem could be alleviated and I have no idea which would be the best one. For example, hiring comittees could be encouraged to only take a look at 10 papers chosen by the applicant and disregard all others including their total number. But it's doubtful they would follow this advice in practise. Or, "allowed" publications per average year could be limited to a minium of n and a maximum of m papers. So for example, to keep funding you need to publish (on average, over a larger period of time) at least 1 peer-reviewed article and no more than 3 per year in average. Sounds crazy and I don't know how to enforce this, but it would increase the quality of papers if m is chosen sufficiently low. Or, get more stringent peer reviewing, although it's a mistery how you'd obtain that in the current system. Perhaps open access journals with crowd reviewing/ranking and meta-moderation would work, as long as mechanisms are held in place to weed out sockpuppets and trolls - difficult, though.
Anyway, it's mostly the publication pressure, in terms of numbers, that causes bad publications.
Now back to work... I need to finish a hastily written paper.
Oh so Popper's Falsificationism is the be-all and end-all of what constitutes science? I guess I was mistaken when I thought there is far more subtlety and detail in the philosophy of science.....
...except about something like catastrophic anthropogenic global warming :)
Oh man, you are totally right! How could I have been so blind! We should be more skeptical about shit like evolution and gravity too! Down with close minded dogma!
The correct take-away from this kind of study is not that a specific field of science is "broken" (also, cancer research is not all of science), but rather that there is room for improvement.
There is no question whatsoever that cancer research has made leaps and bounds over the last few decades in terms of improving the lives of many people with cancer, both by helping them to live longer, and by helping them to live better. What this kind of study shows is that we can do even better still, if we can find ways to fix the flaws that remain in cancer research.
Every great scientist's career is built on the cold, dead corpses of his peers' and antecedents' own work.
No kidding!!! What do you say at this point?
Completely agree, I have a friend who's getting a doctor in Sociology with a concentration in women studies. Some of the crap she's made me read is ridiculous. She stopped talking to me for awhile after I said what she does isn't science. She can't even replicate an "experiment" from one group to another let alone across a generational, cultural, or geographic gap, and yet some of these "studies" are used to set employment policies that discriminate against majorities and created the "we don't care if you're qualified to do this if you don't help us meet our quota" environment.
>Remember, first and foremost, science is a way for us to prove ourselves *wrong* - it's a way of knocking down ideas, and only grudgingly giving acceptance to the ones that survive the contest. The best scientists ruthlessly try to find every possible hole in their ideas, rather than glossing over contradictory evidence or alternatives.
Yes it is, which is why I'm inclined to take seriously steadily expanding body of data that almost entirely supports catastrophic AGW. Are there occasional points of conflicting data? Of course - there always are in any experiment. When almost everything fits together consistently within the framework of a broadly accepted model, then the outliers are quite likely to be errors, or possibly things which we simply do not yet understand well enough to see how they fit within the model.
Of course it's also always possible that they are the signs of something completely unexpected that's going to turn the accepted model on it's ear - as quantum mechanics did to physics. The thing is though that even then the prior model tends to be mostly valid, there's just other factors at work that allow for additional behaviors not predicted by the original model. So yeah, it may be that there's some other way to interpret the data that makes for a much rosier climate picture, but nobody has yet managed to create another model that holds together in the face of available data. And not for lack of trying by various researchers funded by those with a vested interest in the status quo, who have mostly produced science so laughably bad that it's obviously a PR tool never intended to be plausibly presented to other scientists.
--- Most topics have many sides worth arguing, allow me to take one opposite you.
Yes, in fact, Popper was right - falsification is the bedrock of science.
Without falsification, you simply have religion, no matter how fancy the lab coat you dress up in looks like :)
Just because you use maths doesn't mean it's not religion.
Having lost the argument on GW and then AGW, denlialists have now invented catastrophic AGW as their new talking point? Good grief.
No kidding!!! What do you say at this point?
It's more like "you'll need to provide counter-evidence at least as strong as the concensus".
No kidding!!! What do you say at this point?
Evolution (or more specifically, natural selection), and gravity have necessary and sufficient falsifiable hypothesis statements.
Astrology does not have a necessary and sufficient falsifiable hypothesis statement.
Intelligent design does not have a necessary and sufficient falsifiable hypothesis statement.
AGW does not have a necessary and sufficient falsifiable hypothesis statement.
If you want to understand how to discern pseudo-science from science, look for the necessary and sufficient falsifiable hypothesis statement.
Oh so Popper's Falsificationism is the be-all and end-all of what constitutes science? I guess I was mistaken when I thought there is far more subtlety and detail in the philosophy of science.....
If there is philosophy in science, you aren't doing science correctly. As a practicing (and grumbling and struggling) scientist, the original poster is correct - real science is all about proving things wrong because it is impossible to prove something right. The best we can do is to say 'we haven't proven it wrong yet.' This is why in my eyes large portions of biology and sociology and psychology isn't science. Or to quote a famous scientist:
All science is physics, the rest is stamp collecting. - E. Rutherford
As mentioned in a thread below, Sociology. They make stuff up, but if you disagree then you're going against what's considered politically correct and you're sexists, and/or racist. Climatology is based on real numbers of things that can be measured.
Like I said it's fine if you don't agree with it, but what ends up happening is people go on message boards and start screaming and making outrageous claims against the popular literature and data, but then have absolutely nothing to back them up other than "Fox news said so!!!"
So what do I believe the lying climatologists that have reproducible facts and figures supporting their claims, or some nobody screaming that I'm an idiot because I'm not outraged that there's evidence to support climate change is real?
The same can be said about astrology.
A "consistent with" model isn't science - hell, the bible gives us plenty of "consistent with" observations...the key to science is falsifiability, period.
If your model predicts that a coin flip will be either heads or tails, 100% of the time, it's not much of a model. Heads I win, tails you lose is a sucker bet, not a scientific proposition.
Put another way, can you name or cite any catastrophic AGW studies that ever stated a necessary and sufficient falsifiable hypothesis statement? Can you quote that statement?
Wait, are you saying you deny the last 17 years of no statistically significant warming? (GW)
Or are you saying you deny the last 150 years of natural warming coming out of the little ice age? (AGW)
Or are you saying that AGW is true, but we don't need to worry about it, because on the whole increased temperatures are better for the biosphere? (CAGW)
What part of "climate always changes" don't you understand?
You understand that you're basically calling for the elimination of explorational science, right? No more observational science, no more materialistically inventive science, no more methodologically inventive science, no more science but that which can be boiled down into a child's pat hypothesis-test-result-conclusions science lesson.
You're basically saying that we should obliterate science as a creative endeavour.
No kidding!!! What do you say at this point?
He neither said that nor implied it. What he said was that any criticism of AGW is met with a defense akin to a religious fervor. This is a true statement.
As demonstrated.
I can't say that I, personally deny any of those things, as I'm unqualified. However I have the sense to side with a broad spectrum of independent, competing researchers in a wide variety of fields with decades more experience than me whose work all points in a direction contrary to the argument your furthering.
No kidding!!! What do you say at this point?
It has been said in other papers too that a lot of the literature is wrong (http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124) and that this is more likely in higher impact journals and for papers with lower sample sizes (http://www.nature.com/nrn/journal/vaop/ncurrent/full/nrn3475-c6.html). The idea is that a smaller sample size is more likely to lead to a Type I error (incorrectly finding a statistically significant result) or over-estimating the size of an effect. Consequently, these smaller sample size studies find what looks like a stunning effect but what they're really seeing is an outlier. The paper looks awesome so it gets published somewhere high impact, where it is sensationalised. This effect is exacerbated by the "publish or perish" mentality, where researchers are pressured to produce many high impact papers in order to get grants. It's also a function of the fact that a lot of research is being done, so the high volume increases the odds of this shit happening. Cancer biology is particularly prone to this sort of effect because it's very competitive, there's a lot of interest in it and so it generates high impact papers, and there are a lot of big screening studies that depend heavily on statistics to confirm effects. In some branches of biology you hardly need a stats test because variables are few in significance is obvious. However, when you're screening vast numbers of drug targets then you have all sorts of problems with multiple comparisons and the like. You need elaborate stats tests and they have to be done right. Overall, however, whether the community as a whole believes something is determined by state of the literature in general and not just a single study. What we consider true or false is influenced by the politics of science as well as the data. This is nicely reviewed in the controversial book, "The Golem", by Collins and Pinch (http://www.amazon.com/The-Golem-Should-Science-Classics/dp/1107604656).
soylentnews.org
These are not fields of science other than as repositories of anecdotal evidence. The vast bulk of each discipline's studies and experiments cannot be reproduced. You venture correctly.
The problems that plague science today are much deeper than the simple, solvable problem of peer review. If you actually listen to the critics who have been speaking out on the issue for decades now, the problems start in grad school. See Jeff Schmidt's book, Disciplined Minds, which exposes the details of how consensus actually forms in science today. The public likes to imagine that consensus is decided by individuals who are aware of alternative options for belief. The truth is that the consensus is simply manufactured in the grad schools, through an over-reliance upon memorization (as opposed to checking for actual conceptual comprehension, like with force concept inventory tests) and the weeding out of students who stray from the technical details of the problems they are assigned to. The truth is that the features we desire in professionals -- obedient thinkers who can fit into large organizations without "getting political" -- is really quite different than the values we associate with thinking like a scientist (which necessarily includes open-mindedness and skepticism). The notion of "professional scientist" is actually an idea with internal conflicts. It's a contradiction out in the open which apparently few have put any thought into. But, once you look at the way we train professionals today, it becomes apparent that we are not training them to actually think like scientists.
We actually had an incredible chance to have this debate back in May of 2000 when Noam Chomsky stood up with around 700 researchers in support of Jeff Schmidt. Schmidt even won his case against the American Institute of Physics, but the AIP's purpose has always been to obscure this debate from national discourse.
The AIP realizes that the credibility of much of science is basically on the line. If consensus is largely manufactured, then the public cannot rely upon it as a guide in the more empirically challenged domains.
left to the reader
People in cars cause accidents....accidents in cars cause people
To a point, yes. If I were engaging in a debate, I would have to provide evidence.
Most of the time, however, I'm not. I am not impressed by the consensus of large groups of people who spend half their day on Facebook proving what vapid idiots they are. Since when has consensus ever been an indication of truth, anyway? Everyone once agreed on geocentrism.
That I'm right, and you don't like it, doesn't mean I'm a troll.
I'm not for denying creativity - but calling astrology science doesn't do anyone any favors. The creativity of science lies in the novel creation of insightful necessary and sufficient falsifiable hypothesis statements. The creativity of say, using a shotgun approach to discovering new material syntheses, or exploring the bottom of the ocean can be an implementation of science, but simply because alvin may be loosely related to necessary and sufficient falsifiable hypotheses of geology doesn't mean that taking a robot submersible out and making videos is "science".
Science is science. Science requires falsifiability. It's really that simple, despite all the complexity that such a simple statement generates.
So, you've now reduced your argument to an appeal to unnamed authorities :)
Sounds like religion to me :)
It's true that the system can be gamed in the short run. And sometimes someone can be game it enough to get tenure. But without follow up and citations, they'll just end up in academic limbo of being an associate professor with no funding.
Scientists always set out to be less wrong than the last guy, though.
No they don't. TFA lists many examples of scientists choosing to advance their careers rather than trying to be "less wrong".
The Economist Magazine had a cover story on this issue just last week, that in my opinion covers the issue better than TFA.
Basically, the current system of peer review and replication is failing. Peer reviewers actually miss many errors, rarely check statistics, and almost never re-run any software. The current publishing system has little interest in printing replication, and spending time replicating experiments is a dead end career path. The existing system doesn't work well in the era of "big science" and "big data".
We need to move to a system where all publicly funded science is required to be disclosed when it is initially funded, so negative results cannot later be buried. We should also move to online publishing, with a permanently active area for comments, so if the research is later refuted, or even questioned, that is immediately visible. A portion of public science spending should be set aside for replication. There also should be negative consequences for researchers that publish papers that cannot be replicated, whether because their results are wrong, or because they failed to disclose enough information about how the experiment was conducted. Scientists accepting public funds should be required to make their data and software available.
But the biggest obstacle to reform is researchers and publishers that have prospered under the existing system. Many of them treat the current system of peer review as some sort of holy ritual, and refuse to even admit that the system is broken.
Ah, so you would have believed the sun revolved around the earth, at least until around 1514 or so. Or, that eggs were good for you, oh wait, they're bad, oops, nevermind, they're good again. Or maybe that until 1982 most believed that gastric ulcers were caused by stress and spicy foods instead of bacteria.
Personally, I choose not to believe experts without some basic evidence that can be explained in layman's terms. Not doing so is to follow those who believed the experts when we were told Iraq had weapons of mass destruction. Show me some evidence.
As for climate change, I'm not a "denier", but I've yet to see solid evidence that it's anthropomorphic. But, I'll also admit that I haven't searched for it...color me uncommitted to either side of the argument.
Just another day in Paradise
He neither said that nor implied it. What he said was that any criticism of AGW is met with a defense akin to a religious fervor. This is a true statement.
As demonstrated.
No. Scientific criticism of AGW is fine. But coming up with inane conspiracies, casting aspersions, or character assassinations are NOT valid forms of scientific criticism. Worse, the people often spouting such nonsense have little if any knowledge of the actual science and DON'T WANT TO KNOW IT.
Don't equate denialisim with legitimate skepticism. There are legitimate skeptics, but they aren't the ones claiming that the entire world's population of climate scientists is on a mission to murder Jesus and create a socialist utopia. Deniers make the real skeptics look bad, and actually serve to drown out real scientific skepticism with their idiocy.
~X~
Creationists say: "Only Biologists are immune to scrutiny. For all other branches of science you're allowed scrutiny without calls for being burned at the stake."
For what it's worth: I think people who reject certain branches of science (like Climatology or Biology) for non-scientific reasons (be they religious or political) contribute very little actual scrutiny because their objections are not based in science.
Instead they mostly contribute FUD: Fear, Uncertainty and Doubt.
Get rid of the big money and political interests funding it.
(i.e. Exxon, BP Global, Al Gore Carbon Exchanges.)
The people are not interested in climate change man made or otherwise, they are interested in profits.
To avoid this, government in the past was usually employed to carry out science. In the golden age of scientific discovery, (50-60's) gigantic paces in scientific and technological progress were made, not because it was profitable to do so, but because one country in the world, the United States decided that the knowledge gained from such an exercise was the profit. This profit was to be applied to the human conditions of food, shelter and medical.
Now, we have a fascist state, and there can be no separation of government/profit in a corporate fashion. Science isn't even possible to do any more on the scale of what was done in the 50's-60's.
Now, the only science we do largely, is commercial and it is sick and twisted.
(i.e. Like the focus on symptomatic causes of disease because that is more profitable than a cure, man made climate change with people seriously about setting up carbon exchanges to save the planet and more investment in technological trinkets and more iShit.).
It is all going to fail, and it is going to fail harshly for the species who is apply it and is heading out the door and into the fossil record.
-Hack
Got Geometrodynamics? Awe, too hard to figure out? Too bad.
...gravity [has] necessary and sufficient falsifiable hypothesis statement[s]...
Oh, yeah? Prove it.
Now that you have opened my eyes about this global "science" conspiracy, I cannot live in darkness anymore. The path you have laid for me is clear. Everything must be equally called in to question until I get personally satisfying answers.
However, social choice theory is a fairly precise discipline with a number of marvellous impossibility theorems, new voting procedures, etc., and there are also many socialogists who make fairly good statistical research. In a nutshell, your friend chose to focus on the crapiest areas of her discipline.
Not a bad idea, just be sure to post lots of pictures to keep people interested :-P
"When information is power, privacy is freedom" - Jah-Wren Ryel
Kudos, exactly what I was getting at. I couldn't have said it better.
You've got a good point about negative results, but I don't think I agree with the rest.
There's nothing wrong with peer review as such, but the current research climate doesn't help it at all. In many countries, research grants are tied to "measurable, objective results", e.g. articles published, preferably in highly-ranked journals. And so researchers want to publish as much as possible, in as highly-ranked journals as they can get into. (Leading to an explosion in research, so no one really has the time to follow all the research in their own field, or even doing thorough peer review.) Journals are ranked among other things from how often they are cited. Negative results aren't often cited. Replicated tests are only cited in systematic reviews. Setting aside money for replication would be a good idea, but journals shouldn't need to fear for their ranking for publishing less glamorous articles either (or rather: ranking shouldn't be taken seriously). Most importantly, publishing shouldn't be so strongly encouraged. Far too much is published already, and much of it just isn't very good.
As for online publishing: that has been the norm the last decade, and is absolutely dominant now. Comment areas? Like Slashdot? God forbid.
Has the conversation ever been about anything other than catastrophic AGW? Trying to claim that it wasn't is just lying to yourself. If the conversation was about whether the planet was going through a natural warming cycle, it wouldn't be a discussion at all. If the discussion were about the planet having warming and cooling trends, it wouldn't be discussion.
Have you ever met a single person ever that denied the planet had warming and cooling trends? Of course not. From the very beginning, it was understood by everyone that GW meant catastrophic global warming. Everyone understood AGW to mean catastrophic anthropomorphic global warming. Everyone understood that climate change meant catastrophic climate change. If it wasn't understood to be catastrophic, no one would care.
Any claim that there was ever a conversation that wasn't about catastrophic climate change is a very poor lie.
If you do your statistics right, the likelihood of a Type I error is unaffected by sample size. When you use an arbitrary threshold for determining a significant effect (typically 5%), the number of papers attaining that threshold "by chance" (=5%) is independent of sample size.
http://www.nature.com/nrn/journal/v14/n5/full/nrn3475.html
Maybe I phrased it badly. Also I gave the wrong link. The correct one is: http://www.nature.com/nrn/journal/v14/n5/full/nrn3475.html and is worth reading.
The point is that the error being made in research isn't purely a statistical one. It originates in various bad practices, such as "flexible" study design and an ignorance of statistical power. The smaller the sample size, the greater the standard error of the mean. Thus, studies with small sample sizes are more likely to produce an estimate of the population mean that is very different from the true value. Negative results aren't interested and don't get pursued, so we're left with a bias. The result is that under-powered studies are more likely to produce large, "interesting looking", effects which get published in top journals.
soylentnews.org
In part. But there are frequently hard to quantify systematic errors that can be corrected for by using a larger sample size with more diverse representation. For example, a study might pick up a spurious correlation due to focusing on white men of western European descent between the ages of 20-29, and that correlation may not hold when a larger study uses a more diverse group of people.
There's also the issue of smaller studies being quicker and easier, which reduces the desire to publish even if the findings are negative.
A portion of public science spending should be set aside for replication.
This was the first thing that came to my mind as well.
It seems one group does something and everyone else relies on that, until something falls down at some distant point in the future.
At the very least, someone building upon a work that was not replicated should include replication in their proposal as the first step.
Often expensive, but not nearly so expensive as finding out later that the original was wrong.
(As for re-running the software, that seems risky at best, especially if the results could have been influenced by
buggy software, more so if it is custom software, probably less so if its just off the shelf statistical packages or some such).
Sig Battery depleted. Reverting to safe mode.
Okay, Newton's law of universal gravitation states that every point mass in the universe attracts every other point mass with a force that is directly proportional to the product of their masses and inversely proportional to the square of the distance between them.
This statement is clearly falsifiable - all we have to do is find two point masses (say, two planets for sake of argument), and show that their pattern of motion violates this law.
The "if the surface temperature gets warmer or if the surface temperature gets colder, catastrophic anthropogenic global warming is real" has no falsification, on the other hand.
It is clearly the duty of anyone who wishes to learn scientifically to insist on a necessary and sufficient falsifiable hypothesis statement for any significant proposition.
So my necessary and sufficient falsifiable hypothesis statement is "there is no necessary and sufficient falsifiable statement of catastrophic anthropogenic global warming". You can falsify me simply by quoting directly some expert's necessary and sufficient falsifiable hypothesis statement of CAGW.
Good luck! :)
Basically, the current system of peer review and replication is failing. Peer reviewers actually miss many errors, rarely check statistics, and almost never re-run any software. The current publishing system has little interest in printing replication, and spending time replicating experiments is a dead end career path. The existing system doesn't work well in the era of "big science" and "big data".
No, it's not. Peer review isn't meant to catch all errors, just errors in logic. You're assuming that the review process that goes on in mathematics is the same as the review process in all other fields of science, which just isn't true.
Replication is given to grad students -- if they succeed, you know they learned the method, and if they fail, it goes into a chapter of their thesis -- if they're a crappy grad student and they can't get anything to work, they wash out, and if they're good grad students that have other good results, then you go around for the next twenty years saying "so-and-so tried to redo that guys work and couldn't figure it out, it's probably crap, so don't work for that guy". And that's how replication is done.
Sure, I don't want to spend my time replicating experiments, but why should I? And what is the point to publishing it? If I'm any good you don't want me wasting my time with that. I'll have students and then they can do it, and in the process they'll learn, and so life goes on. Science is a community process... you can't break it down into something so cut and dried as "it must all be on paper, published and formatted nicely" -- nobody has time for that crap.
You verify research by reproducing the results. If you can get them, it's science. If you can't get them, it's bullshit. It's as easy as that.
Seven puppies were harmed during the making of this post.
Oh man, shit shit shit. They got to you! How did they get to you? This is bad, very bad...
Think man! THINK! Shake it off. Gravity was already falsified by Einstein, those predictions fail on relativistic scale! Of course, instead of admitting that gravity was wrong, the left-wing "Gravitationist" establishment simply came up with a bunch of excuses as to why their predictions failed, and then changed their theory to match the data!
It's the same thing they're doing with AGW and astrology. Don't buy it man, snap out of it! They don't give a crap about truth, just pushing the "Gravitationist" agenda.
Hold tight man, you've opened my eyes, so I owe you one, I won't sit by and see you brainwashed by "them". I'll find you, I don't know how - but I will, and we'll get you the help you need, I just hope it's not too late.
Oh, crap! There's someone at the door! I think they've found me too! How did they get here so quick? Just hang on, don't give up, we're coming for you brother, we'll get you out of there! Gotta disappear now...
Um, no - you've misunderstanding the cite you're trying to make. http://en.wikipedia.org/wiki/Tests_of_general_relativity
"In general relativity, this remaining precession, or change of orientation of the orbital ellipse within its orbital plane, is explained by gravitation being mediated by the curvature of spacetime. Einstein showed that general relativity[1] agrees closely with the observed amount of perihelion shift."
Furthermore, pay attention to just how *small* the difference was:
"His re-analysis of available timed observations of transits of Mercury over the Sun's disk from 1697 to 1848 showed that the actual rate of the precession disagreed from that predicted from Newton's theory by 38" (arc seconds) per tropical century (later re-estimated at 43").[3]"
Climate scientists would *love* to have that kind of error bar :)
The fact of the matter is that the gravity hypothesis has a falsification - a climate scientist looking at 38 arc seconds of divergence from prediction would've simply claimed that it was within error bars, or they would've hard coded in a ad hoc special pleading to their model to account for it.
But hey, you live in a world where science only requires lab coats, vice presidents, and movie stars, right? :)
Ok, I give. You've clearly got this all figured out, climatology is obviously pseudoscience.
I'm curious though, since you've got better reasoning skills and insight than all the climatologists, how did they do it?
How did they get all the climatologists to think they're doing science when they're not, and to come to the consensus that AGW is real? I mean manipulating an entire field of academia is some pretty scary, nex level shit.
Are all climatologists just idiots? Is it some kind of conspiracy? Is it because "Big Green" is simply out-spending "Big Oil" to buy scientists' favour?
You have to help me out here, without your brilliant insights and superior reasoning, I don't think I'll be able to figure it out on my own - and I need to know! The uncertainty is frightening.
I'd argue two points - one, the odd need for some humans to have *some* sort of faith, when raised as rational atheists, end up replacing it with something else apocalyptic...in this case the CAGW fraud.
Second, as you point out, money. Money money money. The amount of money that has been poured down the drain of big wind and big solar, and all the rent seekers out there is *phenomenal*. Want to talk about billions of propaganda pushing the CAGW line? A wave that big is going to chum up the waters pretty fierce...it certainly seems to have gotten you on board :)
But the real question is this - not whether or not a specific climatologist is an idiot, but whether or not you're going to be the kind of person that outsources their thinking to someone else. Blindly trusting men in lab coats is as silly as blindly trusting men in priestly robes - they're both an abdication of the responsibility to think for oneself...which, the scientific method, and its foundational premise of falsifiability, allows us to do for ourselves.
So do yourself a favor, and figure it out on your own without my help :) Look for the falsifiable hypothesis statement of CAGW. Look *real* hard. If after a significant effort in this fruitless search you find nothing, perhaps you'll have actually learned something :)
It's all THREE???
Wow. Just... wow. I had no idea.
ALL climatologists are idiots who don't know about the scientific method, the whole thing is being covered up by some pseudo-religious apocalyptic cult AND "Big Green" has more money than "Big Oil"?
Jesus H. Christ! We have to do something! We have to tell someone! This is scary HUGE!!!
Why hasn't this gotten out yet? Who's got a stranglehold on this? We need to stop them ASAP.
Wait, what was that click, are they listening to this?
Nevermind, I have NO idea what you're talking about, this AGW thing is *totes* real! (*wink* *wink*)
In the big picture, we can know that this can only happen by fraud or sloppy science, because all these papers claim 90% of more confidence that the null hypothesis is false. Out of a sample size of 53 the odds of only 6 being verified if they're individually 90% likely to be right is vanishing small.
Like, 10^-40 small. It fucking can't happen unless the scientists are either liars or incompetents.
In fact, it's more likely that all of those papers, including the 6 with verified results, were complete fabrications than that more than a third of the authors weren't making shit up.
Seriously, if science is this bad generally, why even bother paying the least bit of attention to it?
Obviously that can't be concluded from the evidence. What can be concluded is that biomedical researchers are liars. It makes me feel real confident about this new drug and the safety of my genetically engineered food.
Yeah, but it could stand some serious improvement if only 11% of the major findings are replicable. Admittedly, that's 10.979% better than religion, 10.4% better than tradition, 6.7% better than polling and 8% better than pulling numbers out of my ass.
Boy, you've got just one sarcasm spewing level, "tsunami", don't you? :)
And frankly, it's a typical defense when you've got no real argument to make - unable to reconcile the fact that science requires falsifiability, and that the CAGW hypothesis lacks falsifiability, with your deep seated belief that CAGW must be science because people in lab coats told you so...I can totally get why that would send one running to the warm, safe Fortress of Sarcasm :)
But you know what, at least your defense is humor - that's a good start! By showing an ability to make a joke, you've already shown more mental flexibility than 99.99% of your typical warmist foot soldiers :) It might take another 10 years of stagnant temperatures and rising CO2 levels to convince you, but there's at least *hope* in your case.
In the meantime, I know that itch in the back of your mind still lingers, "where, or where, is the necessary and sufficient falsifiable hypothesis statement of CAGW?" :)
The double helix was not a "suffient falsifiable hypothesis statement". The Higgs model was not a "sufficient falsifiable hypothesis statement". The atomic picture of matter was not a "sufficient falsifiable hypothesis statement". Science is about more than making neat little yes/no questions. Sometimes it's about making observations and throwing a model out there and seeing if it works. In Higgs' case, that might take decades.
Science must be falsifiable. That does not mean that every scientific statement must be originally presented as a falsifiable hypothesis.
No kidding!!! What do you say at this point?
I could list the papers that constitute the "authority" on the issue, but I wouldn't know where to start, and we'd be here for years. In much the same way that you can use simple arithmetic without having to demonstrate its validity from axioms (and those axioms' validity for the problem at hand), I can quite reasonably defer to a body of scientific knowledge that would take man-centuries to recreate, unless presented evidence to the contrary.
No kidding!!! What do you say at this point?
Yes, I defer to the best available evidence rather than picking an unsupported answer and hoping that dumb luck proves me right in the future.
No kidding!!! What do you say at this point?
That is only acceptable as long as there are only two possible outcomes and it is impossible that NEITHER is correct.
My guess is that it usually fails at that last part. Just because A is false doesn't mean B can't be false as well.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
So your excuse for not listing specific authorities is that you wouldn't know where to start, and the list would be larger than human capacity to enumerate?
Really?
Astrology charts would take man-centuries to recreate...do you consider astrology science?
Look, if you want to outsource your rational thought processes to specific authorities you can't even *start* to enumerate, well, I suppose you have no quarrel with someone who has a hard time picking a favorite patron saint out of the catholic church :)
The double helix, the higgs model, and even our atomic picture of matter have very specific, necessary, and sufficient falsification criteria.
Intelligent design and catastrophic anthropogenic global warming do not.
Now, you're correct, science often starts simply as inspiration from unconnected observations, but it cannot proceed without falsifiability. Thus far, no warmist has ever presented any sort of falsification criteria whose absence can only mean that human CO2 emissions have been the cause of recent warming, and that this warming will be catastrophic at a given point in the future.
Like any good scientist, I've presented you with a falsifiable hypothesis - simply quote someone's necessary and sufficient falsification criteria, and I'll be proven wrong.
Good luck! :)
Far too much is published already, and much of it just isn't very good.
But who decides what is "good"? The temptation is for scientists to publish the results that support their theory and reject the rest of the results as "bad" data, leading to massive selection bias.
Surely it is better to publish all the data so that others can check the conclusions that the author has drawn from it? And in the case of publicly-funded research it seems right that all the data should be made publicly available (with the obvious exception of sensitive information such as personal medical details).
Assume for argument's sake that 80% of science is crap. Big deal; 80% of all human endeavour is crap.
Star Trek transporters are just 3d printers.
I tell you what, if you want to provide a list of evidence for the fact that gravity is attractive at all distances and that air is breathable, then you can make that argument. I'll see you when you're done. It's gonna be a long list.
No kidding!!! What do you say at this point?
The falsifiability criteria for the double helix, higgs model, and atomic picture of matter were not determined until decades after publication. By your reasoning, they should never have been published.
Resolve that.
No kidding!!! What do you say at this point?
You've missed the point - we don't prove that there are only white swans by enumerating every single white swan. We prove that there are only white swans by looking *real hard* for non-white swans, and failing to find them.
Both gravity and the "breathability" of air have necessary and sufficient falsifications. The novel and dubious premise of catastrophic anthropogenic global warming, on the other hand, does not have necessary and sufficient falsifications to exclude all other possibilities and leave CAGW as the only remaining alternative.
You really don't understand the scientific method, do you?
Hogwash. The inherent *design* of these models included clear and necessary falsification criteria. Pics or it didn't happen.
As for publishing, anyone is allowed to publish speculation and even fiction, but it doesn't become science until it has the property of falsifiability.
Not quite. We're seeing an exponential growth in publishing without a similar increase in quality. That means the crap to quality ratio is going up.