Slashdot Mirror
Bloggers Put Scientific Method To the Test
ananyo writes "Scrounging chemicals and equipment in their spare time, a team of chemistry bloggers is trying to replicate published protocols for making molecules. The researchers want to check how easy it is to repeat the recipes that scientists report in papers — and are inviting fellow chemists to join them. Blogger See Arr Oh, chemistry graduate student Matt Katcher from Princeton, New Jersey, and two bloggers called Organometallica and BRSM, have together launched Blog Syn, in which they report their progress online. Among the frustrations that led the team to set up Blog Syn are claims that reactions yield products in greater amounts than seems reasonable, and scanty detail about specific conditions in which to run reactions. In some cases, reactions are reported which seem too good to be true — such as a 2009 paper which was corrected within 24 hours by web-savvy chemists live-blogging the experiment; an episode which partially inspired Blog Syn. According to chemist Peter Scott of the University of Warwick in Coventry, UK, synthetic chemists spend most of their time getting published reactions to work. 'That is the elephant in the room of synthetic chemistry.'"
The bloggers are not testing the scientific method, they are testing methods that are scientific. Those are two vastly different concepts. Their work is important, but not epic.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
If you try to repeat an experiment and fail then it is almost impossible to get published. Failed experiments, though critical for advancing science, aren't sexy and editors prefer their journals to be full of positives. So scientists don't even bother trying anymore. This is a problem in medicine and probably all sciences. There is a movement in medicine to report all trials so they can be found by researchers doing meta-studies.
It's not a secret that about half of published synthesis methods are garbage and yield values are wildly creative. Reviewers don't have the means to verify these, so anything that seems plausible gets published. Then researchers are left to sort out the best methods based on which ones get the most citations.
The bloggers are not testing the scientific method, they are testing methods that are scientific. Those are two vastly different concepts. Their work is important, but not epic.
I'm not so sure about that.
We believe in a scientific method founded on observation and reproducible results, but for a great number of papers the results are not reproduced.
Taking soft sciences into consideration (psychology, social sciences, medical), most papers hinge on a 95% confidence level. This means that 1 out of every 20 results arise from chance, and no one bothers to check.
Recent reports tell us depression meds are no better than chance and scientists can only replicate 11% of cancer studies, so perhaps the ratio is higher than 1 in 20. And no one bothers to check.
I've read many follow-on studies in behavioral psychology where the researchers didn't bother to check the original results, and it all seems 'kinda fishy to me. Perhaps wide swaths of behavioral psychology have no foundation; or not, we can't really tell because the studies haven't been reproduced.
And finally, each of us has an "ontology" (ie - a representation of knowledge) which is used to convey information. If I tell you a recipe, I'm actually calling out bits of your ontology by name: add 3 cups of flour, mix, bake at 400 degrees, &c.
This assumes that your ontology is the same as mine, or similar enough that the differences are not relevant. If I say "mix", I assume that your mental image of "mix" is the same as mine. ...but people screw up recipes, don't understand assembly instructions, and are confused by small nuanced differences in documentation.
Does this happen in chemistry?
(Ignoring the view that reactions can depend on aspects that the researchers were unaware of, or didn't think were relevant. One researcher told me that one of her assistants could always make the reaction work but no one else could. Turns out that the assistant didn't rinse the glassware very well after washing, leaving behind a tiny bit of soap.)
It's good that people are reproducing studies. Undergrads and post-grads should reproduce results as part of their training, and successful attempts should be published - if only as a footnote to the original paper ("this result was reproduced by the following 5 teams..."). It's good practice for them, it will hold the original research to a higher standard, and eliminate the 1 out of 20 irreproducible results.
Also, reproducing the results might add insight into descriptive weaknesses, and might inform better descriptions. Perhaps results should be kept "Wikipedia" style, where people can annotate and comment on the descriptions for better clarity.
But then again, that's a lot of work. What was the goal, again?
Frankly, Chemistry is among the easiest of the physical sciences. I say this as the physicist who was tasked by the chemists to fix their gear when it broke down.
Organic chemistry is quite difficult. The purpose of synthesis is not as you suppose, just mix A and B, see what happens and publish. Most organic chemists are trying to make specific transformations on certain parts of molecules in high conversion and trying to control the variables of time, temperature, concentration, reagent reactivity with substrate functional groups, etc.
Physics is just a block on an inclined plane and variations.
If Slashdot were chemistry it would look like this:Cadaverine
The sad truth is that this happens everywhere, also in computer science. I once implemented a software pipelining algorithm based on a scientific paper, but when implemented, the algorithm appeared to be broken and had to be fixed first. In this case, it probably wasn't malice either. But the requirements to get something published are different from getting it to actually run. Brevity and readability may get in the way of correctness, which may easily go unnoted.
I am a physicist. I do believe that a large percentage (less say 50%) of scientific publications do not meet basic quality standards, let alone the scientific method. It depends also on the level of the journal. But even in the best journals you can find articles and results that can not be reproduced. The pressure to publish is too strong. Anyway, 70% or more of the articles are only academic exercises, or do not have robust statistics or do not receive more than one or two citations (including a couple of my own). Only the best of the articles (should) prevail with time (Darwinism).