Bloggers Put Scientific Method To the Test

ananyo writes "Scrounging chemicals and equipment in their spare time, a team of chemistry bloggers is trying to replicate published protocols for making molecules. The researchers want to check how easy it is to repeat the recipes that scientists report in papers — and are inviting fellow chemists to join them. Blogger See Arr Oh, chemistry graduate student Matt Katcher from Princeton, New Jersey, and two bloggers called Organometallica and BRSM, have together launched Blog Syn, in which they report their progress online. Among the frustrations that led the team to set up Blog Syn are claims that reactions yield products in greater amounts than seems reasonable, and scanty detail about specific conditions in which to run reactions. In some cases, reactions are reported which seem too good to be true — such as a 2009 paper which was corrected within 24 hours by web-savvy chemists live-blogging the experiment; an episode which partially inspired Blog Syn. According to chemist Peter Scott of the University of Warwick in Coventry, UK, synthetic chemists spend most of their time getting published reactions to work. 'That is the elephant in the room of synthetic chemistry.'"

Comments

Terrible, Terrible, Headline

[damn_registrars]

The bloggers are not testing the scientific method, they are testing methods that are scientific. Those are two vastly different concepts. Their work is important, but not epic.

Re:Terrible, Terrible, Headline

[girlintraining]

The bloggers are not testing the scientific method, they are testing methods that are scientific

Putting your ignorance in boldface type is amusing. The most basic promise of the scientific method is that results can be replicated by anyone with the proper equipment repeatedly and reliably. This is accomplished by describing an experiment to the detail level necessary to reproduce the result. If the result cannot be reproduced from a description of the experiment, it has failed this test.

They are testing the scientific method insofar as asking whether professional and peer-reviewed scientific work actually meets this basic test. And in many cases it doesn't. Science only works if it is built on a firm foundation: Their work isn't just important, it's critical. It may not be fun, but explorations like this prevent us from assembling a body of knowledge and understanding based on flawed experiments... and that has happened many times in the history of science, especially in medicine.

Re:Terrible, Terrible, Headline

[phantomfive]

they are testing methods that are scientific.

If a lot of those experiments can't be reproduced, then those methods weren't scientific to begin with.

Re:Terrible, Terrible, Headline

[gman003]

They are testing whether scientific papers meet the scientific method (ie. the results are reproducible). They are not testing the validity of the scientific method itself (myself, I cannot see how one could test the scientific method without using it, thus bringing the results into question).

That is the point GP was attempting to make.

Re:Terrible, Terrible, Headline

[flink]

Right, but they are utilizing the scientific method to test the quality of published papers, not attempting to verify the utility of the scientific method itself.

The headline should read "Bloggers apply scientific method to validate published findings".

Re:Terrible, Terrible, Headline

[gandhi_2]

This speaks to the failings of the participants implementation of the scientific method, not to the failing of The Scientific Method.

We have discussed here before the problem of too much content being generated and not enough people to peer review it all. Still not a failing of The Scientific Method.

And I prefer my ignorance in courier font.

Re:Terrible, Terrible, Headline

[gandhi_2]

Oh snap!

Re:Terrible, Terrible, Headline

[damn_registrars]

they are testing methods that are scientific.

If a lot of those experiments can't be reproduced, then those methods weren't scientific to begin with.

Not necessarily. Sure, irreproducible results can be the result of shoddy, missing, or fabricated work, and unfortunately often are. There are also times when those results can come about through no fault the experimenter. If the scientist was running an overnight synthesis and was not notified that the HVAC failed for two hours starting at 3am is that his fault? Sure, he should check afterwards to make sure that the environmental conditions were properly controlled but that isn't always immediately apparent to the scientist. Similar with something like a pressure regulator or any of a number of other laboratory instruments which should be reliable though have a nasty habit of failing in interesting ways at inopportune times.

There are times when good science is done, and bad (or badly reproducible) results come out.

Re:Terrible, Terrible, Headline

[phantomfive]

"Bloggers apply scientific method to validate published findings".

A much better headline.

Re:Terrible, Terrible, Headline

[c0lo]

myself, I cannot see how one could test the scientific method without using it, thus bringing the results into question

So little faith you have...

(large grin)

Re:Terrible, Terrible, Headline

[damn_registrars]

The bloggers are not testing the scientific method, they are testing methods that are scientific

Putting your ignorance in boldface type is amusing. The most basic promise of the scientific method is that results can be replicated by anyone with the proper equipment repeatedly and reliably

And I am sorry that you struggle so greatly to understand what I have written.

They are testing the scientific method insofar as asking whether professional and peer-reviewed scientific work actually meets this basic test.

Do you not understand the purpose of peer-review? If results that were peer-reviewed are not reproducible, that is not a failing of the scientific method itself, nor is it a failing of peer review. Peer review does not exist to validate methods as that would be quite nearly an impossible task for the majority of all scientific papers that are published currently unless the journal sent an editor to the lab that submitted said paper to rerun the work themselves - which would be so absurdly expensive that nobody would ever pay to publish. Peer review is intended to make sure that work published is scientifically rigorous and well written.

Hell, if you go back and actually read my comment - I would say re-read but it does not appear you read it successfully for a first time yet - you will find that I did say this work is important. I also said that it is not testing the scientific method itself, which is correct.

Re:Terrible, Terrible, Headline

[Zontar The Mindless]

Right, I tagged the story "crapheadline" as soon as I RTFS.

Re:Terrible, Terrible, Headline

[RatherBeAnonymous]

A scientist should also run experiments multiple times to see if the results are repeatable before publishing those results. If you can't repeat your results you can't possibly give others instructions on how they can repeat them. Not knowing that the HVAC failed for a couple hours during one run out of a dozen should result in outlier results that can be investigated or discarded.

Re:Terrible, Terrible, Headline

[girlintraining]

They are testing whether scientific papers meet the scientific method (ie. the results are reproducible). They are not testing the validity of the scientific method itself (myself, I cannot see how one could test the scientific method without using it, thus bringing the results into question).

Epistemology would have something to say about your sight. Yes, the scientific method can be evaluated and tested. Anyway, if that's what you say the GP is making as a point, then it's a really stupid and pedantic one. The scientific method isn't just an abstract concept, it's also a means to an end. If we find that many people are making the same mistakes in the process itself, then yes, it is a statement about the scientific method.

It's like saying a lake is a body of water and ignoring the fact that you also need something to hold it in.

Re:Terrible, Terrible, Headline

[Anonymous Coward]

E-specially, forgive me I couldn't help myself. I am one of your devoted and humble fans girlintraining.

Re:Terrible, Terrible, Headline

[fluffy99]

Do you not understand the purpose of peer-review? If results that were peer-reviewed are not reproducible, that is not a failing of the scientific method itself, nor is it a failing of peer review. Peer review does not exist to validate methods as that would be quite nearly an impossible task for the majority of all scientific papers that are published currently unless the journal sent an editor to the lab that submitted said paper to rerun the work themselves - which would be so absurdly expensive that nobody would ever pay to publish. Peer review is intended to make sure that work published is scientifically rigorous and well written.
 

Many of the published results and methods being verified are ones that have questionable results - such as producing too much output chemicals or reactions don't appear they should work at all. Those are papers for which peer-review has failed to provide adequate review. If the paper was truly read in-depth by other equally qualified scientists these issues would have been noticed and the paper (published or not) would have been called into question.

The caveat to this would be papers that are published with the sole purpose of seeking peer review and inviting other to validate the results, for example many of the cold fusion papers and the experiment which implied neutrinos were traveling faster than light.

I also recognize that peer review happens both before and after publication, and in fact the bloggers are part of the peer-review process.

Re:Terrible, Terrible, Headline

[fluffy99]

A scientist should also run experiments multiple times to see if the results are repeatable before publishing those results. If you can't repeat your results you can't possibly give others instructions on how they can repeat them. Not knowing that the HVAC failed for a couple hours during one run out of a dozen should result in outlier results that can be investigated or discarded.

Unfortunately, sometimes the experiments do get run multiple times and the data that didn't fit the expected results was thrown out and not included in the final data. I've seen far too many scientists who automatically throw out the highest and lowest samples and only average the data that grouped nicely, without making much effort to find out why some samples deviated.

Re:Terrible, Terrible, Headline

[jedidiah]

...or alternately: The Half Blood Prince tweaks the results.

Re:Terrible, Terrible, Headline

[countach]

Since testing IS the scientific method, if there is something wrong with the scientific method, this test will fail!

Re:Terrible, Terrible, Headline

[paiute]

A scientist should also run experiments multiple times to see if the results are repeatable before publishing those results.

No time. Many if not most synthetic methodology papers will test a new reaction on a range of homologous compounds, say 20. Few of those are repeated twice, let alone multiple times. A lot of this work comes out of graduate school groups where the emphasis is on publishing rapidly. There is one Nobel Prize winner in particular whose publications in Tetrahedron Letters (usually a two page paper) are notorious for sacrificing accuracy for rapidity.

Re:Terrible, Terrible, Headline

[Anonymous Coward]

Oh good lord. Testing other's methods is not testing the scientific method. They are using the Scientific Method. What is wrong with Slashdot? The Scientific Method is not merely doing stuff in science. Part of the method is testing other's work. This is what they are doing. Showing that you can repeat an experiment is part of the Scientific Method. They are not testing the Scientific Method itself. Please do tell. How would you test the Scientific Method without using the Scientific Method?

Re:Terrible, Terrible, Headline

[Anonymous Coward]

Science seems much more brute force now rather than driven by curiosity. Curiosity might lead someone to make a new discovery by exploring those outliers. Both methods yield results though. Progress is made. One has the mindset of industry and the other of enlightenment.

Re:Terrible, Terrible, Headline

[Anonymous Coward]

Thank you for making your ignorance more verbose, and ignorant

They are not testing the method, they're testing the documentation and practices of those who claim to be using this method.

IF an experiment is not repeatable given the information published by peers, due to vague or ambiguous documentation, Then the method hasn't failed, those publishing and peer reviewing said documentation have failed.

Re:Terrible, Terrible, Headline

[nedlohs]

That's not testing the scientific method. Testing the scientific method would involve some test as to whether the scientific method itself works. Determining if some published experiment is actually described in a way that is reproducible says nothing about whether the scientific method itself works or is useful.

Asking whether professional and peer-reviewed scientific work is actually using the scientific method is also not asking whether the scientific method itself works or is useful.

I agree it's a useful thing to do though. In fact it seems like something that should be a routine part of graduate studies, heck even honors level studies. Give them some experience with real world experiments and check something to boot.

Re:Terrible, Terrible, Headline

[__aaltlg1547]

No, it means that the original experimenters didn't describe their experiment correctly. Or worse, may have never done it at all...

Re:Terrible, Terrible, Headline

[Anonymous Coward]

To find that a researcher falsified results, or can't write a paper if his life depended on it, does not reflect on the validity of the scientific method in the least. They are using the scientific method to test published works produced by others.

The subject of testing is the work, not the method. I don't know how many other ways I can put this to explain it to you, but the end result is that damn_registrars is correct in his statement.

Re:Terrible, Terrible, Headline

[Anonymous Coward]

If the result cannot be reproduced from a description of the experiment, it has failed this test.

You might as well say that the laws of reality have failed this test. What actually fails, of course, is not the scientific method, but the scientists who published the irreproducible results in the first place, and the scientists who allowed them to pass peer review.

Re:Terrible, Terrible, Headline

[mysidia]

We have discussed here before the problem of too much content being generated and not enough people to peer review it all. Still not a failing of The Scientific Method.

The article's headline text is a demonstration of that issue of too much content and not enough peer review :)

Re:Terrible, Terrible, Headline

[mysidia]

Right, but they are utilizing the scientific method to test the quality of published papers

This could turn into a study on... what is the quality of published papers/ ? :)

If studies show, that studies are meaningless -- eg if it becomes shown that very often, the scientific method has been ignored....

Then the scientific method is effectively shot, not because it's invalid, but because it's not being followed, and it becomes no longer reasonable to believe researchers are following it

Re:Terrible, Terrible, Headline

[RespekMyAthorati]

Then the scientific method is effectively shot, not because it's invalid, but because it's not being followed, and it becomes no longer reasonable to believe researchers are following it

That is identical to:

People who don't do up their seatbelt buckles die.
Therefore, seat belts fail to protect people.

The fact that people are failing to apply the scientific method does not mean that the scientific method has failed, only that some people who call themselves "scientists" shouldn't.

Re:Terrible, Terrible, Headline

[Anonymous Coward]

Yeah, but he has the benefit of being interested in science and technology. The editors as Slashdot don't care about that nerdy shit - they're just nerd chic. (See Timothy's glasses.)

Re:Terrible, Terrible, Headline

[Anonymous Coward]

yeah, based on other posts as well, girlintraining has a serious stick up her butt.

Re:Terrible, Terrible, Headline

[sFurbo]

I thought Tetrahedron Letters in general was notorious for allowing very low quality articles to be printed. There must be a reason my professor in organic chemistry insisted on calling it "tetrahedron liars".

Re:Terrible, Terrible, Headline

[Mindcontrolled]

Could as well say "Bloggers do what tens of thousands of grad students do every day."

Re:Terrible, Terrible, Headline

[Sique]

The claims, the experiments of the climate science could not be reproduced, could not be reproduced. In fact, the Berkeley Earth Surface Temperature project was a successful reproduction of the experiments of climate science.

Re:Terrible, Terrible, Headline

[paiute]

I thought Tetrahedron Letters in general was notorious for allowing very low quality articles to be printed. There must be a reason my professor in organic chemistry insisted on calling it "tetrahedron liars".

TettLet is for rapid publications. The methods are not always fully worked out and there is no detailed experimental. The function of TetLett is to give the chemical world a heads up of a new way of doing something. Where I expect a paper from the Journal of Organic Chemistry to be detailed, I don't expect the same of TettLet.

Re:Terrible, Terrible, Headline

[Anonymous Coward]

More probably, run the experiment 20 times, dismissed the 19 times it didn't work as manipulation error and only reported the one time it worked. Since peer review doesn't (most of the time) select on validity but on interest, honest scientists who'd have reported the 19 times it didn't work would have seen their paper rejected for "lack of interest".

Re:Terrible, Terrible, Headline

[vlm]

A scientist should also run experiments multiple times to see if the results are repeatable before publishing those results.

Won't help. I studied to be a chemist (admittedly a long time ago) and by far the biggest non-ethical problem out there is contamination.

So it turns out that your peculiar reaction you're studying is iron catalyzed, in fact its incredibly sensitive to iron, but no one in the world knows that yet. And your reagents are contaminated. Or your glassware, which you thought was brand new and/or well cleaned, is contaminated. Or your lab is downwind of a hematite ore processor and the room dust is contaminated.

Sure, you say, test everything. Well there isn't time/money for that, but for the sake of argument we'll assume there is. What if dust from the hematite ore processor is far larger than the filter paper pores in filtration stage of your overall process? Test the reagents and product all you want but you'll never find iron anywhere except room dust (which you already knew about) and the debris in the filter paper (which you assume was contaminated by room dust AFTER removal from the apparatus)

The most important thing is this is the norm in chemistry, not an outlier. Chemistry is not math or CS, sometimes stuff just doesn't work or just works for no apparent reason. Unlike some technologies, detailed modeling of "why" "how" often doesn't happen for years, decades, centuries after the ChemEng team has been selling product / papers have been written.

A very important lesson is analysis paralysis. So you live downwind of a hematite ore crusher. And you know it. And periodic tests of your lab show iron enriched house dust. But you can't go around testing everything, because you're surrounded by millions of things to test for. You're a gardener, god only knows whats on your hands. Skin oil of certain blood types is a contaminant? Your breath has a tinge of ethanol in it from last night? Maybe its your perfume / cologne / antiperspirant / nail polish? The point of discovery is its literally unknown... maybe wearing nitrile gloves instead of old fashioned latex "does something" good or bad to the reaction.

I think the main thing "slashdotter IT people" need to understand is most chemistry and most chemical engineering runs somewhat less than 6 sig figs. This is incomprehensible to IT people... if your T1 or whatever LAN had a BER worse than 1e5 you'd call it in for repair... If you got one thousand read errors when you read a 1 gig DVD, you'd throw out that DVD. If your processor runs at 1500 mips then at a six sig fig error rate it would crash about 1500 times per second. The bad news is six sig figs is actually pretty good work for a chemistry lab. Certainly undergrads could never aspire it that level, both skill and experience and specialized equipment....

Please spare me the details of one peculiar quantitative analysis technique that in one weird anecdote measured once in tiny fractions of a ppt. The overall system cannot be "cleaner" than the filthiest link in the entire system. Is the hand soap in your lab spectrographically pure? The unused toilet paper in the bathroom? All of your hoods and benches and storage cabinets are in a verified and tested cleanroom environment? Seriously? The drinking cooler and lab fridge also only hold spectrographically pure substances? Please no anecdotes.

Re:Terrible, Terrible, Headline

[jo_ham]

Could as well say "Bloggers do what tens of thousands of grad students do every day."

That's about the face of it. The only difference is that they're blogging their progress rather than simply trying to get an intermediate step to work because you wanted to make something that is on the route to what you're really studying and you just want a decent prep for it.

I think the general rule of thumb is "subtract at least 15% from the expected yield and assume at least one crucial step has been inadequately documented" and "ignore esoteric and exotic purification method and just recrystalise it".

Re:Terrible, Terrible, Headline

[rickb928]

No, I have not studied philosophy with any serious effort, but if the scientific method cannot be used to prove itself, then perhaps it is lacking?

And I don't reject the 'scientific method'. I think it is self-evident and self-revealing.

The real question of these papers is not can the results be shown to be reproducible, but did the papers sufficiently disclose the methods? And if not, why not? there is, naturally, the possibility that the results were not even reproducible by the authors, and so they either falsified data/results or failed to disclose their success rate, but that's to be expected. Complaints about yields or consistency are par for the course, I suspect, in many branches of science, and predictable, which explains why so many revolutionary ideas are not yet in widespread use.

Re:Terrible, Terrible, Headline

[rickb928]

If 'Peer Reviewed' doesn't include replicating results (which would delay publication and we can't have THAT!), then it is reduced to 'nice penmanship'.

Which, of course, we know is true. Peer review is just self-ratification. No one else could possibly understand, much less reproduce, serious and groundbreaking work, so just step back and watch the show. Pay at the door.

Re:Terrible, Terrible, Headline

[rickb928]

Not describing the experiment correctly will not bother their corporate sponsors, who will either employ them directly or get the full description upon payment. Never doing it at all? No problem if they get more grants to build upon the nonexistent foundation they fabricated - unless they change direction with every grant and never evolve their previous work, which is nice work if you can get it.

Re:Terrible, Terrible, Headline

[Anonymous Coward]

That is because a lot of the people paying for science expect results and a return on their investment, which precludes messing around with probable dead ends; running everything like a business has a lot of downsides, but since the only people who need to be convinced of that are the ones with money, and since they got that money by ignoring the fact, it seems unlikely to change anytime soon.

Re:Terrible, Terrible, Headline

[Electricity Likes Me]

Do you not understand the purpose of peer-review? If results that were peer-reviewed are not reproducible, that is not a failing of the scientific method itself, nor is it a failing of peer review. Peer review does not exist to validate methods as that would be quite nearly an impossible task for the majority of all scientific papers that are published currently unless the journal sent an editor to the lab that submitted said paper to rerun the work themselves - which would be so absurdly expensive that nobody would ever pay to publish. Peer review is intended to make sure that work published is scientifically rigorous and well written.

 

Many of the published results and methods being verified are ones that have questionable results - such as producing too much output chemicals or reactions don't appear they should work at all. Those are papers for which peer-review has failed to provide adequate review. If the paper was truly read in-depth by other equally qualified scientists these issues would have been noticed and the paper (published or not) would have been called into question.

The caveat to this would be papers that are published with the sole purpose of seeking peer review and inviting other to validate the results, for example many of the cold fusion papers and the experiment which implied neutrinos were traveling faster than light.

I also recognize that peer review happens both before and after publication, and in fact the bloggers are part of the peer-review process.

Also frankly, there's a ton of papers out there which do work but which omit crucial details of how they work. Queue 2 years of my life discovering that someone's "simple robust synthesis" can't possibly have worked the way they said it did, and gradually reverse engineering that they were doing something they didn't report in any papers which had an important effect (they didn't quite seal up the reaction vial, but didn't leave it open, so coupled with water and condensation they were setting up a highly unreliable oxygen headspace control system - rig up a reliable way to fix the mol % and hey, anyone can do it!).

I've very much become convinced that the terse writing style which gets encouraged in a lot of chemistry papers is just not useful at all when it comes to replicating them (and protocol papers are worse - people write still write what they were trying to do, not necessarily what actually gets done more often then not).

Re:Terrible, Terrible, Headline

[Artifakt]

If a large proportion of researchers claiming to use the scientific method either can't or won't actually use it, the method has failed in some way. How is saying it would work if only human nature changed any different from making the same claim about anything else, say Communism? How do you prove the method itself is correct if most of your test subjects actually fail to understand how to use it, yet in this case, your test subjects are supposedly all well trained expert users? How do you prove the method is correct if your test subjects are mostly crooks trying to cheat the overall larger human system by manipulating the data they are providing you?

      And, just because you claim that two things that are not identical are identical doesn't make it so. Try diagramming your two sets of sentences in symbolic logic and they do come out just about a full 180 degrees out of phase.

Re:Terrible, Terrible, Headline

[Anonymous Coward]

Right - so you agree with ghandi_2 then that it is the implementation. Not describing your experiment correctly is a failure of implementation.

Re:Terrible, Terrible, Headline

[gman003]

No, I have not studied philosophy with any serious effort, but if the scientific method cannot be used to prove itself, then perhaps it is lacking?

It's more a matter of logic.

Assume, for the sake of argument, that the scientific method is flawed. Trying to test the scientific method could produce myriad results. Tests could show that it always fails. Tests could give random results. Or tests could correlate with some random thing - imagine if the scientific method only works with a waxing gibbous moon is ascendant in Leo. Or, and this case is significant, tests could always show that it does work (but it then fails when you try to test other theories).

So if the scientific method is flawed, any experiments to determine its validity rely on flawed methodology, and are thus untrustworthy.

Now let us assume that the scientific method works. Attempting to test this leads to only one conclusion: that the scientific method works.

Now let us live in the real world for a moment. We do not "know" if the scientific method works or not. And when we run an experiment to test the scientific method, it gives us a result "true".

However, how can we distinguish the case of "the experiment shows 'true' because the scientific method works" from the case of "the experiment shows 'true' because the scientific method does not work, but it does not work in such a way that it produces false results when asked to test itself"? You cannot, at least when you only use the scientific method.

That is not too dissimilar to other fields. You cannot prove that parallel lines do not intersect purely by using geometry. You cannot prove 1+1=2 using math. The former is treated as an axiom, a statement that is intuitively assumed to be true in further proofs. The latter is a definition of terms - given what 1, 2, addition and equality are defined as, the statement is true.

So, even though the scientific method cannot prove itself, it can still be completely correct. Or, at the very least, statistically correct.

Re:Terrible, Terrible, Headline

[rickb928]

Lines that intersect are not, by definition, parallel. Lines either intersect or they do not. Lines that do not intersect are parallel.

We define non-intersecting lines as parallel. We don't prove they are parallel, we describe them as such.

It has been a long time since I've employed geometry. How that relates to proving the Scientific Method is beyond me.

Re:Terrible, Terrible, Headline

[__aaltlg1547]

Not describing the experiment correctly will not bother their corporate sponsors, who will either employ them directly or get the full description upon payment. Never doing it at all? No problem if they get more grants to build upon the nonexistent foundation they fabricated - unless they change direction with every grant and never evolve their previous work, which is nice work if you can get it.

Honestly, I think the most probable reason is poor record taking and poor writing.

Re:Terrible, Terrible, Headline

[rickb928]

Sadly, you are quite possibly correct.

So much for the scientific method. Even the experts suck at it.

Re:Terrible, Terrible, Headline

[mysidia]

People who don't do up their seatbelt buckles die.
Therefore, seat belts fail to protect people.

If researchers show people on average say they do up their seat belts but are actually fibbing, lying, or mistaken, and don't actually do up their seatbelts properly, then, yes, it does show seat belts fail to protect people.

But more importantly: it says, you can't trust people when they say they do up a seatbealt.

Which is where you and your analogy fall flat on the face.

It doesn't physically effect other people (much), if you fail to put on your seatbelt, and you lie about it -- you're hurting yourself.

If nobody follows the scientific method, or implements appropriate statistical procedures -- then they are generating garbage, that other people might accidentally rely upon.

Showing people don't actually follow the scientific method, is pretty much as good as showing the scientific method doesn't work.

Not the method is not valid, BUT: because people can't (or don't) actually follow the method, even when they say they do: AND, there's no method available of keeping people honest.

At least with the proper doing-up of seatbelts, there are police officers, to monitor, and enforce it in some cases (people still often don't wear their seatbelts) -- with scientific method implementation - not only is there no enforcement, but there are practical barriers that make enforcement, or reliable fraud detection, essentially impossible.

Re:Terrible, Terrible, Headline

[monkeykoder]

You cannot prove that parallel lines do not intersect purely by using geometry. You cannot prove 1+1=2 using math. The former is treated as an axiom, a statement that is intuitively assumed to be true in further proofs. The latter is a definition of terms - given what 1, 2, addition and equality are defined as, the statement is true.

The former is not treated as an axiom it is an axiom and as such it cannot be proven in any system. The latter can indeed be proven if you step outside of the bounds of algebra (which is a field of mathematics that covers addition) and into the bounds of set theory where we have a definition for 1 and a definition for 2 and a definition for the process of addition and using those three definitions we can indeed prove that 1 + 1 = 2. In algebra we would define the addition of our system such that 1 + 1 = 2 at which point it is treated as an axiom. To be clear there are systems where 1 + 1 != 2 (boolean algebra) but we can assume you knew that and were specifically discussing integer arithmetic.

Re:Terrible, Terrible, Headline

[__aaltlg1547]

Sadly, you are quite possibly correct.

So much for the scientific method. Even the experts suck at it.

Nobody said science was easy.

Not just synth chemistry

[Anonymous Coward]

If you try to repeat an experiment and fail then it is almost impossible to get published. Failed experiments, though critical for advancing science, aren't sexy and editors prefer their journals to be full of positives. So scientists don't even bother trying anymore. This is a problem in medicine and probably all sciences. There is a movement in medicine to report all trials so they can be found by researchers doing meta-studies.

Re:Not just synth chemistry

[ChrisMaple]

If you try to repeat an experiment and fail then it is almost impossible to get published

Tell that to Fleischmann and Pons.

Re:Not just synth chemistry

[WrongMonkey]

As a research chemist, I've published a couple of papers that were motivated because I didn't believe a paper's results to be true. The trick to get it past reviewers is to not only prove that they are wrong, but to come up with an alternative that is demonstrably superior.

Re:Not just synth chemistry

[phantomfive]

That doesn't sound like a very good system.

Watching scientists work should be like hearing one guy say, "Hey, check this out!" and another guy saying, "wow, that's cool, I can't get it." and another saying, "oh, you need to do it like this!"

If someone can't say, "that doesn't work like you said it did," then scientific progress is going to be hampered.

Re:Not just synth chemistry

[BrokenHalo]

That doesn't sound like a very good system.

It's one that works, and is another part of the scientific method, albeit a step along from establishing reproducibility of an experiment. If prior research appears to set a hypothesis, then that should be tested - i.e. someone should make an attempt to knock it down by whatever means are available. You never really "prove" something to be the case, but if nothing can be done to disprove it, that's pretty nearly as good.

Re:Not just synth chemistry

[phantomfive]

It's not a problem with the scientific method, it's a problem of communicating results. Clearly it isn't working optimally if this article is correct. If people can only tell each other when something works, but can't discuss when things don't work, then the communication channels are severely broken.

Re:Not just synth chemistry

[DiamondGeezer]

These bloggers are really denialists who refuse to accept published scientific findings in peer-reviewed journals by credible scientists. An overwhelming scientific consensus is against them.

Re:Not just synth chemistry

[jo_ham]

That doesn't sound like a very good system.

Watching scientists work should be like hearing one guy say, "Hey, check this out!" and another guy saying, "wow, that's cool, I can't get it." and another saying, "oh, you need to do it like this!"

If someone can't say, "that doesn't work like you said it did," then scientific progress is going to be hampered.

It's how it works though. Usually it's "hey look at this, and this happens because of x reason" and you come back with "I don't think that works the way you think because of y systematic error or z error in your technique". I've personally done this before with published papers, and it doesn't necessarily invalidate them so much as add more information to the table.

Re:Not just synth chemistry

[Electricity Likes Me]

It's not a problem with the scientific method, it's a problem of communicating results. Clearly it isn't working optimally if this article is correct. If people can only tell each other when something works, but can't discuss when things don't work, then the communication channels are severely broken.

This is actually a huge problem.

Science really needs something like Freenode for different disciplines. A lot of the time you simply want to be able to talk to some of the people dealing with the work in the original group, since they're likely to be able to give you the small details which turn out to matter.

Mythbusters: Chem Edition

[gmuslera]

They got explosions at least?

Seriously? IDIOTS

[BobGod8]

Okay, how is this NOT common knowledge? Of course chemistry is hard, of course it's hard to replicate results--if it weren't, why in the hell do you think it took this long in the first place? If chemistry were easy, there'd be hundreds of reports of planes downed by terrorists, instead of the 3-4 ATTEMPTED explosions there have been. Chemistry is HARD, good luck doing it with limited ingreditents and improvised equipment.

Re:Seriously? IDIOTS

[Anonymous Coward]

Replicating chemistry isn't hard with good instructions. Having had to turn a simple experiment that was more simple physics than anything else from a Doctorate thesis into a production process, I can tell you these things are sometimes very poorly documented. The basic assumption should be that they can be repeated from the instruction provided, and that simply isn't true. I'm not saying there's fraud involved (we were able to replicate the results, but even with access over the phone to the person who did it it wasn't easy as his notes were crap) I'm just saying these people sometimes get away with doing an extremely poor job at documentation. And that goes against one of the basic premises of the scientific method.

Re:Seriously? IDIOTS

[Anonymous Coward]

They have access to GC/LC-MS and NMR equipment. They are not using improvised equipment.

IAAC (I am a chemist).

Re:Seriously? IDIOTS

[Anonymous Coward]

Who's using "limited ingredients and improvised equipment"? These are trained chemists with access to fully equipped labs, high-purity materials, and the original authors' descriptions of their procedure. If they can't replicate the findings of the original chemists by following the exact same procedure, something is badly wrong.

Re:Seriously? IDIOTS

[isomer1]

Eh. These aren't random idiots. They are graduate students. Typically they know more about the nuts and bolts than the PIs. They are trying to show the absurdity of the current system. I for one heartily applaud their efforts. The frustrating issue with science in academia, and I say this from 3 years of graduate work and 5 years of post-graduate work at major research universities, is that the process has become just another lame industry. The purpose of modern publicly funded laboratories is to churn out papers. Yes those papers are peer reviewed. But that doesn't make the science any more sound, it just insures the arguments in the paper are internally consistent. I've seen people repeat an experiment 20+ times, then publish the results of the SINGLE experiment that happened to give the numbers that matched their model. That doesn't mean the experiment worked that time, it just randomly gave a value that a reviewer would accept. Frankly, Chemistry is among the easiest of the physical sciences. I say this as the physicist who was tasked by the chemists to fix their gear when it broke down. Their papers are almost universally set forth as an over glorified recipe. If you look across the fields chem majors have a higher percentage of publishing as undergrads, because the papers are so damn easy. You mix your bits, filter and then quantify the results. Boom, paper. Yes I'm oversimplifying, but not by much. The joke goes that if you want to make a breakthrough in modern chemistry you'd better find a physicist. If you want a breakthrough in modern physics you'd better find a mathematician. The overarching problem is that the system pushes for papers period. Not worthwhile science, not correct science, simply papers. This has been passed down by the NIH to whom it was in turn handed to by congress and the public at large. People demand some mechanism to quantify how the tax dollars are spent, and papers became the most convenient metric. Thus the number of papers have ballooned astronomically, while the value has plummeted. People think they're so damn clever when the talk about how our pace of innovation accomplishes in months what used to take generations. They're just bullshitting themselves. Yes, there are *some* papers that are absolutely fantastic today, but the signal to noise ratio is far lower than it was even twenty years ago.

Re:Seriously? IDIOTS

[Anonymous Coward]

You might find chemistry hard, but the Universe does it all the time without complaining.

Re:Seriously? IDIOTS

[Zontar The Mindless]

I can tell you're the real article. It's well known that physicists have no concept of paragraph breaks. ;)

More seriously, thank you for the excellent post. Which, BTW, backs up everything that my dad (retired physics/math prof) has been complaining about in this area for the last 20 years or so.

Re:Seriously? IDIOTS

[paiute]

Replicating chemistry isn't hard with good instructions.

Organic chemists often lump reactions into two classes: peak reactions and plateau reactions. Peak reactions give a maximum yield of product only for a narrow range of conditions. Plateau reactions work over a broad range of conditions. If you go to run a plateau reaction, you don't need a very good SOP, but if you are running a peak reaction you need very specific conditions and detailed instructions for every step. And you generally don't know what type of reaction you are going to have on your hands just by looking at it on paper.

Re:Seriously? IDIOTS

[paiute]

Frankly, Chemistry is among the easiest of the physical sciences. I say this as the physicist who was tasked by the chemists to fix their gear when it broke down.

Organic chemistry is quite difficult. The purpose of synthesis is not as you suppose, just mix A and B, see what happens and publish. Most organic chemists are trying to make specific transformations on certain parts of molecules in high conversion and trying to control the variables of time, temperature, concentration, reagent reactivity with substrate functional groups, etc.

Physics is just a block on an inclined plane and variations.

Re:Seriously? IDIOTS

[Anonymous Coward]

And you generally don't know what type of reaction you are going to have on your hands just by looking at it on paper.

If this distinction is so important and difficult to figure out, why is it almost never mentioned which class of reaction is going on in the paper that describes the synthesis? Why don't reviewers, and other chemists, demand that such information be included?

Re:Seriously? IDIOTS

[BobGod8]

Not really germane to my point, but I know what you mean. I'm just frustrated that the article makes their work sound like exactly what you picture 5 year olds doing with test tubes. I'm sure grad students will fair better, but it will still be difficult--I've done exactly what's being described, and it's anything but easy. There's simply too many variables in real world conditions to even consider, let alone write about.

I once tried to replicate an experiment that it turned out the silicate structure of the glass acted as a catalyst. Change the glass type, change the reaction.

Re:Seriously? IDIOTS

[fearofcarpet]

Frankly, Chemistry is among the easiest of the physical sciences. I say this as the physicist who was tasked by the chemists to fix their gear when it broke down. Their papers are almost universally set forth as an over glorified recipe. If you look across the fields chem majors have a higher percentage of publishing as undergrads, because the papers are so damn easy. You mix your bits, filter and then quantify the results. Boom, paper. Yes I'm oversimplifying, but not by much. The joke goes that if you want to make a breakthrough in modern chemistry you'd better find a physicist. If you want a breakthrough in modern physics you'd better find a mathematician. The overarching problem is that the system pushes for papers period. Not worthwhile science, not correct science, simply papers.

By your logic Physics is even easier that Chemistry; just push "go" in LabView and publish the resulting graph. And Biology practically researches itself; just shake up a jar of bacteria and watch them grow.

The nuances of synthetic chemistry--a sub-discipline of Chemistry--are far more complex than mixing A and B, which is the core problem of reproducibility. If you can prove the structure, then you obviously made it, but conveying precisely what you did can be a challenge as there are too many factors to control for. Perhaps you ran the reaction in a flask that was once used for Pd chemistry. That little bit of residual metal may unknowingly catalyze the reaction, which then fails when someone tries to reproduce it with a different flask. Scale is also a huge issue; reactions do not always scale up or or down. Temperature gradients, rates of heating, rates of stirring, rates of addition, argon or nitrogen, oxygen concentration, methods for cleaning/drying glassware, etc; there are too many parameters to include for each reaction. That is why there used to be a rigorous standard across synthetic journals for how to report a procedure. Unfortunately those standards have been lost due to the one observation you made that isn't born from myopic ignorance; that science--all science--is driven by papers, not science.

Re:Seriously? IDIOTS

[mysidia]

I'm just saying these people sometimes get away with doing an extremely poor job at documentation.

They're not the only ones... it happens in IT a whole lot, as well.

Re:Seriously? IDIOTS

[Coisiche]

Chemistry is among the easiest of the physical sciences. I say this as the physicist

That just created the image in my mind from an episode of The Big Bang Theory of Dr. Sheldon Cooper at the inter-departmental paintball tournament shouting "Geology isn't a real science!"

Re:Seriously? IDIOTS

[paiute]

And you generally don't know what type of reaction you are going to have on your hands just by looking at it on paper.

If this distinction is so important and difficult to figure out, why is it almost never mentioned which class of reaction is going on in the paper that describes the synthesis? Why don't reviewers, and other chemists, demand that such information be included?

Sometimes the researchers will mention it when discussing their results. Plateau reactions are good, and if you find one you tout it, but often the original researcher has not run enough variations on the conditions to even be able to define it. It is often later when several papers are available that you realize that the new reagent, say, works well hot or cold, works well in protic or aprotic solvents, tolerates a wide range of functional groups, etc.

Papers are not little textbooks. We don't expect them to be complete and always accurate. I expect a paper to give me a hint about how to use a new tool to solve a problem I have been working on.

Re:Seriously? IDIOTS

[vigour]

Physics is just a block on an inclined plane and variations.

Stop talking shite. You obviously don't know what you're talking about, all disciplines of science are difficult and complex. If I so wished I could be just as dismissive and say chemistry is just dirty, poorly explained physics (as some physicists up their own arses think). The boundary between physics, material science, and chemistry is very blurred. We use many of the same techniques, just with different approaches, language, and mathematical models. What the chemist is interested in the physicist doesn't care about, and vice versa.

I have worked in physics, material science, and electrochemical labs over the last 10 years (hint, that's not as a student). It's typical of organic chemists to have a poor grasp of physics.

Good luck with that

[WrongMonkey]

It's not a secret that about half of published synthesis methods are garbage and yield values are wildly creative. Reviewers don't have the means to verify these, so anything that seems plausible gets published. Then researchers are left to sort out the best methods based on which ones get the most citations.

Re:Good luck with that

[TrekkieGod]

It's not a secret that about half of published synthesis methods are garbage and yield values are wildly creative. Reviewers don't have the means to verify these, so anything that seems plausible gets published. Then researchers are left to sort out the best methods based on which ones get the most citations.

It's not just synthesis methods. I remember taking a graduate control theory class in which the final project was for the class to replicate the results of a paper with a particular control algorithm. It just...wouldn't...work. Not a single person managed to replicate the results, which simply led to the inevitable conclusion they were fudged.

I'm not and would never defend anyone who publishes any data that has been tempered with, but I still find it annoying that we've set up a system where there's an incentive to do so. There's tremendous pressure to publish in academia, starting in grad school. Combine that with the difficulty in publishing papers that have negative results and a lack of interest in replicating any experiments that are not groundbreaking and you end up with the quality of papers we have here. I'd love for it to be standard for grad students' first papers, as they're learning to write them, to be just replicating results from other papers. And have journals actually recognize the importance of such work, and publish the results often. I think this would cut down the number of crappy papers, because first, you wouldn't want to publish something that's going to be shown to be bullshit in short order, and second, you'd be able to satisfy your publishing requirements by doing the important task of verifying other people's work.

Re:Good luck with that

[ygtai]

So true and well-said. Grad students' first papers should be reproducing results from other published papers and they should be accepted for publication if well-conducted and well-written. Wish I had mod points.

Re:Good luck with that

[Anonymous Coward]

Meh... then you would just get cases where Prof X publishes his result, which is immediately followed by his 4 grad students validating the result... or perhaps students who want to be his grad students...

Re:Good luck with that

[Anonymous Coward]

While what you posted is a result of perverse-incentives, I kinda agree with the GP; I can recall a time I got down-graded for _not_ being able to reproduce the results of an experiment and later thought to fudge numbers so they "fit," but refused. I learned more that way than just accepting... -T

live-blogging

[Anonymous Coward]

Seriously?
It's pathetic enough when you have all these people "live-blogging" Apple's latest release. (Like I can't wait until their press conference is done to read the results). Now they are love-blogging chemistry experiments? Better to do the experiment, take down the results, and then do real analysis and write a proper paper about your findings.

Is anyone actually watching these things? I can see it now:
1. I'm adding 250ml of the calcium carbonate
2. Ok now I am adding the 35cc of Iodine now
3. Now we are turning on the Bunsen burner
4. And three drops of Nitric acid ...
3 hours later the comments start
AC: "Then?! what happened?"
AC2: "I guess it blew up since the posts stopped."

Re:live-blogging

[Anonymous Coward]

Better to do the experiment, take down the results, and then do real analysis and write a proper paper about your findings.

Oh, yeah, that'll work real well when the point of the exercise is to show that the "proper paper" business is seriously fucked up right now.

Re:live-blogging

[vlm]

I've been watching this project for awhile, and it seems to function a lot more like a flash mob than a live-blog.

Its called...

[Hangtime]

calling out BS, exaggerated, wrongly calibrated, and/or embellished results. This makes perfect sense to me. If you publishing a paper on a subject then it should be a repeatable recipe.

Which scientific method are they testing?

[EmagGeek]

Are they testing the tried and true scientific method that *real* scientists used for centuries to arrive at the cumulative knowledge of mankind, or are they testing the modern scientific method that involves drawing a conclusion and then trying to find data that fits your model, discarding any data that doesn't?

Re:Which scientific method are they testing?

[Anonymous Coward]

But, what is in reality the "scientific method".

Each science question, field or problem has its one way to be approached, but there are people that claim that the "scientific method" is how to be approached but referencing only on concrete way and ignoring all the other possible variants of the "scientific method", more if its applied cross-field knowledge. Actually anyone enough smart and clever that had studied different science fields know that there are problems that are actually the same problem with the same solution but that they are presented with different names and actors.

Could this be why?

[EzInKy]

Could this be why the general public doesn't trust science and instead rely on ancient mystical texts to make sense of the world they live in? Maybe a push to show the "hoi poloi" are perfectly capable of replicating the results researchers have observed would advance the cause of science?

Re:Could this be why?

[BobGod8]

Not really. A lot of science is easily accessible, but trying to replicate cutting edge chemistry is not the place to start there. Extracting asprin from bark, 100 year old processes, these are the (still scientific) easy experiments. The public doesn't trust science because we don't even do those with kids--why I have no idea.

Of course the other reason the public distrusts science is because they've been told to by people whom the science inconveniences, but that's not really the scope of this article.

Re:Could this be why?

[RespekMyAthorati]

Maybe a push to show the "hoi poloi" are perfectly capable of replicating the results researchers have observed would advance the cause of science?

How many of the "hoi poloi" would have any idea what the paper is even trying to demonstrate? Or how to test it?

Re:Could this be why?

[jo_ham]

Could this be why the general public doesn't trust science and instead rely on ancient mystical texts to make sense of the world they live in? Maybe a push to show the "hoi poloi" are perfectly capable of replicating the results researchers have observed would advance the cause of science?

Not really. Also, it wouldn't matter - being a skilled synthetic chemist is a lot more than simply following the instructions, no matter how well they are written out.

Perhaps you could make it easy enough for an undergrad with some reasonable lab experience to do it, but the problem with public perception of science is not optimistic yield reports and poor experimental write up in research papers.

Chemistry

[93 Escort Wagon]

It's the major people that can't handle physics switch to.

(I kid, I kid)

Re:Chemistry

[Spiridios]

It's the major people that can't handle physics switch to.

Chemistry is just applied physics.

Re:Chemistry

[Anonymous Coward]

I always disliked xkcd # 435. They obviously missed logicians which would be far, far to the right of mathematicians. Some say that logic is a subset of math, but logic is the study of reason. Math is merely an application system. I suspect this was omitted on purpose because despite what they say, mathematicians believe deep down in their surly hearts that the most fundamental subject of study is not within the math department, but philosophy.

Re:Chemistry

[WrongMonkey]

Complexity is more interesting than purity. Physics is just the branch of science that is simple enough to be modeled accurately with math.

LSD

[Anonymous Coward]

And today, we synthesize LSD, again

Combine Forces

[englishknnigits]

http://openscienceframework.org/project/EZcUj/wiki/home

The Open Science Framework focuses on psychology instead of chemistry but I would imagine they could both utilize similar frameworks and methodologies. I'm not a chemist or a psychologist so there may be some major incompatibility I don't know about.

Re:Combine Forces

[englishknnigits]

I was actually describing the Reproducibility Project which uses the Open Science Framework. My bad.

A consequence of publish or perish

[g01d4]

I suppose reviewers are supposed to catch these things but they're probably too busy with their own 'discoveries' to give more than a cursory glance. Hopefully the bloggers will get enough recognition from their efforts to spur others; especially if the trend towards publishing direct online and away from peer reviewed journals continues.

It might be epic

[Okian Warrior]

The bloggers are not testing the scientific method, they are testing methods that are scientific. Those are two vastly different concepts. Their work is important, but not epic.

I'm not so sure about that.

We believe in a scientific method founded on observation and reproducible results, but for a great number of papers the results are not reproduced.

Taking soft sciences into consideration (psychology, social sciences, medical), most papers hinge on a 95% confidence level. This means that 1 out of every 20 results arise from chance, and no one bothers to check.

Recent reports tell us depression meds are no better than chance and scientists can only replicate 11% of cancer studies, so perhaps the ratio is higher than 1 in 20. And no one bothers to check.

I've read many follow-on studies in behavioral psychology where the researchers didn't bother to check the original results, and it all seems 'kinda fishy to me. Perhaps wide swaths of behavioral psychology have no foundation; or not, we can't really tell because the studies haven't been reproduced.

And finally, each of us has an "ontology" (ie - a representation of knowledge) which is used to convey information. If I tell you a recipe, I'm actually calling out bits of your ontology by name: add 3 cups of flour, mix, bake at 400 degrees, &c.

This assumes that your ontology is the same as mine, or similar enough that the differences are not relevant. If I say "mix", I assume that your mental image of "mix" is the same as mine. ...but people screw up recipes, don't understand assembly instructions, and are confused by small nuanced differences in documentation.

Does this happen in chemistry?

(Ignoring the view that reactions can depend on aspects that the researchers were unaware of, or didn't think were relevant. One researcher told me that one of her assistants could always make the reaction work but no one else could. Turns out that the assistant didn't rinse the glassware very well after washing, leaving behind a tiny bit of soap.)

It's good that people are reproducing studies. Undergrads and post-grads should reproduce results as part of their training, and successful attempts should be published - if only as a footnote to the original paper ("this result was reproduced by the following 5 teams..."). It's good practice for them, it will hold the original research to a higher standard, and eliminate the 1 out of 20 irreproducible results.

Also, reproducing the results might add insight into descriptive weaknesses, and might inform better descriptions. Perhaps results should be kept "Wikipedia" style, where people can annotate and comment on the descriptions for better clarity.

But then again, that's a lot of work. What was the goal, again?

Re:It might be epic

[Areyoukiddingme]

I have mod points, but stupid Slashdot isn't showing me the moderation option on posts. So I will snark instead.

But then again, that's a lot of work. What was the goal, again?

Uhm. Publish or perish, I think it was...... Full speed ahead and damn the torpedoes. Verification studies don't count on your CV.

Good idea to harness the slave labor though. That's what grad students are for.

And how many grad students will actually be willing to do this verification work? None, who can think politically enough to stay in academia. What are you asking them to do? Verify published papers. Who published those papers? The people who will be sitting on their doctoral thesis board. The people who will be peer reviewing their papers. The people who are on the tenure committee. The people who are on the funding committee. I could go on.

The scientific method is a marvelous thing, but the way the system is rigged, we're left to amateurs to verify anything. All hail the internet and bored people, I guess. I also guess this is a measure of our wealth as a society, that there are people with the time and money to donate to verifying synthetic chemistry results. Expect a lot of shouting and angst when they can't verify results though.

Re:It might be epic

[Skippy_kangaroo]

Many commenters seem to be mistaking some idealised thing called the Scientific Method with what is actually practiced in the real world when they claim that the scientific method is not being tested. Damn straight this is testing the scientific method - warts and all. If the scientific method as practiced in the real world does not deliver papers that provide sufficient detail to be reproduced then it is not working properly and is broken. In fact, I'm sure that most people who actually publish papers will acknowledge that the peer review process is broken and does not approach this idealised notion of what peer review is in many people's minds. If peer review is broken then a critical element of the scientific method is also broken.

Reproducing results is hard every time I have done it - and that includes times when I have had access to the exact data used by the authors. (And sometimes even the exact code used by the authors - because I was using a later version of the statistical package and results were not consistent between versions.)

So, if people want to claim that the Scientific Method is perfect and this is not a test of it - it would be interesting if they could point to anywhere this idealised Scientific Method is actually practiced (as opposed the the flawed implementation that seems to be practiced in pretty much any field I have ever become acquainted with).

Re:It might be epic

[jo_ham]

And finally, each of us has an "ontology" (ie - a representation of knowledge) which is used to convey information. If I tell you a recipe, I'm actually calling out bits of your ontology by name: add 3 cups of flour, mix, bake at 400 degrees, &c.

This assumes that your ontology is the same as mine, or similar enough that the differences are not relevant. If I say "mix", I assume that your mental image of "mix" is the same as mine. ...but people screw up recipes, don't understand assembly instructions, and are confused by small nuanced differences in documentation.

Does this happen in chemistry?

It absolutely happens - chemistry is like cooking in that respect. Especially some of the trickier synthetic chemistry where your technique really matters. Chemists of different skill levels are going to get vastly different results from following the same prep.

What these guys are doing is no different to what grad students do all the time - papers are a great source of preps for compounds that you want to use as building blocks for other things you want to do if your starting materials are either not readily available from a supplier or they're absurdly expensive. The only difference is that they're walking through this process and publicly giving their impressions rather than simply noting down the modifications you needed to make to get it to work in your lab book for the next time you need it.

Re:It might be epic

[MozeeToby]

It's good that people are reproducing studies. Undergrads and post-grads should reproduce results as part of their training, and successful attempts should be published

Dang, you were doing so well too. How about we publish both the successes and the failures, rather than just the successes. Just publishing the successes is how we get stuff like the 11% reproducibility rates you talked about earlier. So a footnote in the original paper: "this procedure was replicated by 23 teams, those results supported the results of this paper in only 3 cases."

Re:It might be epic

[Anonymous Coward]

Nothing in the real world works like some idealistic fantasy. I don't know anyone who claims the scientific method is perfect. I know many people who claim it's useful and probably the best way to discover new information about reality. Good luck defeating your straw man, it should be an easy fight.

Re:It might be epic

[ababydingo]

Taking soft sciences into consideration (psychology, social sciences, medical), most papers hinge on a 95% confidence level. This means that 1 out of every 20 results arise from chance, and no one bothers to check.

Recent reports tell us depression meds are no better than chance and scientists can only replicate 11% of cancer studies, so perhaps the ratio is higher than 1 in 20.

1 in 20 would be expected if errors are random and if no biases are in place like publication bias. We know that's not typically the case, so we can indeed expect it's worse than 1 in 20. The more I read about anti-depressant trials, the more I think we will come to view them as particularly egregious examples of deliberately badly designed trials. I've just read Bad Pharma by Ben Goldacre and it's mind-boggling what happens in real world trials. One little example: to take advantage of the 1 in 20 chances, simple hold 20 little trials and publish the one that gets good results! Or ignore your stated trial goals and measure and just cherrypick any good result afterwards. If you have many variables you're bound to get some correlations post hoc, just by chance alone.

Re:So, this theory?

[Anonymous Coward]

Fuck off spammer.

P.S. Your site sucks ass.

good, but...

[stenvar]

Replicating published experiments is a worthwhile effort, and there should be more of it. However, it's already pretty well known that scientific papers have a high rate of irreproducible results, due to both fraud and error. If they fail to replicate particular experiments, it's not an indictment of the scientific method, but of the particular scientists.

(Also, some experiments in the natural sciences are tricky to do and require experience to work, but they can still be replicated.)

Nobel prizes come from "Funny, that's odd"

[Anonymous Coward]

And just those things you didn't realize happen is how new science is discovered.

Incorrect headline

[Anonymous Coward]

Good for them, great cause, and can only help their understanding.

How did a factually inaccurate headline mike it to the front page? They are not testing the "scientific method". Nature.com gets it right "Bloggers put chemical reactions through the replication mill"

Yeah, because thats exactly who I trust.

[Anonymous Coward]

A bunch of bloggers? Yah, like I actually believe any person who has a blog on the internet about anything /rollseyes

Its bad enough science is a problem because most of can be made up. A lot of things anyone can say anything they like and skew whatever they please. A lot of science is pure hard fact data, but even that is laughable at times. Like how in the 70s scientists allover the world proved and knew nuclear energy was causing a ice age but of course we all know how good their factual evidence turned out to be.

So why would exactly would I care or even believe some guys with blogs doing experiments? Because they say they are smart? Because they say that are qualified? Oh and lets not forget, they have blogs....how amazing! We all know only the smartest, most intelligent, honest and qualified people on the planet are allowed to have a blog. Bloggers are part of a super secret and super strict club called "everyone".

Besides really all they did was say "Hey, we proved whats already been proven. We did what others have already done! Arent we so super awesome?!"...no, no you are not.

Hell I could replicated someone elses theory on quantum mechanics but that doesnt really mean Im smart or know what I am doing. Or hell, I could do a shot by shot remake of raiders of the lost ark but it doesnt mean Im a good film maker.

Re:Yeah, because thats exactly who I trust.

[docmordin]

Like how in the 70s scientists all over the world proved and knew nuclear energy was causing an ice age but of course we all know how good their factual evidence turned out to be.

Or like how you're wrong about that, since there was no scientific consensus in the 1970s that Earth was headed into an ice age:

T. C. Peterson, W. M. Connolley, and J. Feck, "The myth of the 1970s global cooling scientific consensus", Bulletin of the American Meteorological Society, 89: 1325-1337, 2008.

Re:Yeah, because thats exactly who I trust.

[Anonymous Coward]

So why would exactly would I care or even believe some guys with blogs doing experiments? Because they say they are smart? Because they say that are qualified? Oh and lets not forget, they have blogs....

Have blog-envy because you don't have your own blog? Or just bitter because some blogger killed your dog? Why are you emphasizing the fact they have a blog when that isn't relevant? Maybe they should be trusted because they have experience working in a research lab, and they would have to justify their results just like anyone else. The fact they blog about it is irrelevant beyond it is just a communication medium.

Like how in the 70s scientists allover the world proved and knew nuclear energy was causing a ice age but of course we all know how good their factual evidence turned out to be.

Maybe before questioning your trust of bloggers, you should question your trust of your own memory...

Re:Yeah, because thats exactly who I trust.

[Anonymous Coward]

Like how in the 70s scientists allover the world proved and knew nuclear energy was causing a ice age but of course we all know how good their factual evidence turned out to be.

Nuclear energy causing an ice age? The only connection having made between nuclear technology and climate I'm aware of is the so-called nuclear winter after a global thermonuclear war. Fortunately this one has never been tested, and I hope it won't ever get tested.

How about unpublished protocols ?

[Taco Cowboy]

There are protocols which are published and then there are protocols that remains unpublished

How about those protocols which have remained unpublished?

Any way to test those?

Re:How about unpublished protocols ?

[alvinrod]

Sure, there is. All you have to do is send me $500 and I will test them according to some protocols which I have developed. I'd like to share them with you, but unfortunately due to proprietary business secrets which must remain undisclosed, I will be unable to publish them.

Re:How about unpublished protocols ?

[__aaltlg1547]

There are protocols which are published and then there are protocols that remains unpublished

How about those protocols which have remained unpublished?

Any way to test those?

That's not science. That's grandma's secret recipe.

Re:How about unpublished protocols ?

[c0lo]

How about those protocols which have remained unpublished?

Ok... for you, let me be as explicit as possible: put you faith in God and pray!

Is it clearer now?
If not, do consider the context (or else).

Still confused? I hope this will clear the matter: whoosh!

Re:How about unpublished protocols ?

[dkf]

How about those protocols which have remained unpublished?

You need to test "Just make it up so that results give the answer we want in order to get more funding"? Really?

Not that good scientists ever use that method. Alas, not all scientists are good, and that's why it is important to check published results. (It's a good task for grad students...)

Re:How about unpublished protocols ?

[rickb928]

That which cannot be patented or copyrighted remains secret by necessity. Lest someone else profit from your work, which cannot be permitted.

It's all about money. All of it. All the time. For everyone. Accept this, and you will do more than survive, perhaps. Ignore this, and you will suffer. Reject this, and you will fail, either in your philosophy or in your efforts. You don't have to like it nor do you have to perpetuate it, but it is what it is.

Re:How about unpublished protocols ?

[mapsjanhere]

The journal for those are the Tetrahedron Letters, and chemists have in vain tried to replicate them for a century.

Re:How about unpublished protocols ?

[Anonymous Coward]

Go read some psychology, and you'll find that your idea is wrong. Money is a means to an end. It often is motivating - but it is far from everything that motivates people. If it was all about money, nobody would anonymously give to charity, and there would be no prostitution, as there would be no buyers.

Re:How about unpublished protocols ?

[cellocgw]

Well, there are the unpublished protocols, and the published unpublished protocols, and finally the unpublished unpublished protocols.

(Just ask Rummy)

The work of Harold Garfinkel

[aussersterne]

is instructive (though still not very readable) here.

The founding ethnomethodologist, Garfinkel argued that much of science depends on practical assumptions and habits of which researchers are only vaguely aware, leading to the "loss" of the phenomenon.

This is both good and bad. On the one hand, it means that a phenomenon is real, with real implications (useless theory and tautology are marked by the difficulty of losing the phenomenon), but on the other hand it means that what is said about the phenomenon is often missing the most critical bits of information, unbeknownst to the PIs themselves because they are unaware of practical (embedded in the practice of) assumptions and habits, something that makes it seem likely that many scientific truths are either solidified far later than they might otherwise have been or incorrectly lost to falsification rather than pursued.

Things I Won't Work With

[Required Snark]

Another Fun Chemistry Blog: Things I Won't Work With http://pipeline.corante.com/archives/things_i_wont_work_with/

It's by Derek Lowe, a pharmaceutical chemist who blogs about chemical compounds that are way too dangerous. His position is that the closest you want to get to any of these things is reading about them. The closest I want to get is reading what he has to say about them.

Take FOOF, aka F2O2, aka Dioxygen Difloride. Lowe calls it "Satan's kimchi".

The latest addition to the long list of chemicals that I never hope to encounter takes us back to the wonderful world of fluorine chemistry. I'm always struck by how much work has taken place in that field, how long ago some of it was first done, and how many violently hideous compounds have been carefully studied. Here's how the experimental prep of today's fragrant breath of spring starts:

"The heater was warmed to approximately 700C. The heater block glowed a dull red color, observable with room lights turned off. The ballast tank was filled to 300 torr with oxygen, and fluorine was added until the total pressure was 901 torr. . ."

And yes, what happens next is just what you think happens: you run a mixture of oxygen and fluorine through a 700-degree-heating block. "Oh, no you don't," is the common reaction of most chemists to that proposal, ". . .not unless I'm at least a mile away, two miles if I'm downwind." This, folks, is the bracingly direct route to preparing dioxygen difluoride, often referred to in the literature by its evocative formula of FOOF.

His latest posting is about the compound C2N14, which is two carbon atoms with 14 nitrogen atoms.

The compound exploded in solution, it exploded on any attempts to touch or move the solid, and (most interestingly) it exploded when they were trying to get an infrared spectrum of it. The papers mention several detonations inside the Raman spectrometer as soon as the laser source was turned on, which must have helped the time pass more quickly.

He doesn't just blog about things that go bang, he also covers things that smell really really bad. It's good he doesn't get stuck in a rut.

He has a fine turn with a descriptive phrase and a dry sense of humor. Check out his blog.

Re:Things I Won't Work With

[jfengel]

Thanks for that. That's hilarious.

A very old elephant in the room

[decora]

These many many decades ago I went to a "top 5 STEM school" and enrolled in Chemistry. While all of the students had Very Expensive Machines to play with, they also had No Time To Do Careful Work.

"Flubbing" of results was considered kind of, you know, ordinary, and if you didn't get what was expected, well, you were expected to just kind of ignore it.

People like to harsh on the 'soft' subjects like history ... but I can assure you that many a history teacher would chew you up one side and down the other for saying things that are patently false and blatantly contradict the evidence - while many a Chemistry teacher would simply tell you "well, it should have worked. and you understand the basic ideas. so lets move on."

I'm not sure what the issue is - if it's just too expensive to do experiments, or maybe the point is not to learn 'experimenting' but rather to learn 'theories... with some hands on experimenting to give you a flavor of it"????

Re:A very old elephant in the room

[mysidia]

"Flubbing" of results was considered kind of, you know, ordinary, and if you didn't get what was expected, well, you were expected to just kind of ignore it.

See... research teams should have separation of duties, to avoid the temptation to cheat. The scientist does the experiment, someone else takes the measurements, someone else records the results, and enters into digital format; which quickly become immutable, digitally signed and timestamped; with noone taking or recording measurements allowed to be the people that know exactly what that measurement "is supposed to be".. finally, an independent auditor monitors, and signs off on the results.

The peer review journals then require the scratch sheets for each trial of the experiment, as signed off by the auditors and measurement takers :)

Re:A very old elephant in the room

[TheRedSeven]

That reminds me of (aeons ago) my 8th grade science project. While most of the other kids were testing "Which battery lasts the longest?" I decided to test the effect of humidity on the speed of sound. Seemed relevant to my 13 year old mind, I couldn't find a lot of information on it, and I had 3 possible outcomes: H0 was that higher relative humidity has no effect; H1 was that higher relative humidity made the speed of sound faster; H2 was that higher relative humidity made the speed of sound slower.

The experiment involved a trip to a nearby college's physics lab, a big old aquarium, some PVC pipe, a humidifier, a microphone attached to some old Apple computer of some sort with audio software, and a wimshurst generator (the only thing that could produce a brief enough noise so the echo could be differentiated from the continuing reverberations). The result was that H0 was disproved and the evidence pointed toward HIGHER speed of sound in higher relative humidity.

I loved the whole thing. Physics! And testing! And math!

And then the judging came. Most of them loved my experiment and gave the whole thing high marks. But one happened to be a college physics professor who walked up, took a look at my results, and said, "You did a lot of fine work, but your results are wrong." And despite my protests of "But those are the results I got," proceeded to give me essentially a 0, making my experiment one of the few that failed.

I still hold a grudge against that physics prof. Not for crushing all the fun out of experiments, but for trusting the 'right' answer over the experimental one. If that's the kind of scientists we're pushing out these days, we've got some serious issues to deal with.

Re:A very old elephant in the room

[Anonymous Coward]

"Humidity has a small but measurable effect on sound speed (causing it to increase by about 0.1%-0.6%), because oxygen and nitrogen molecules of the air are replaced by lighter molecules of water. This is a simple mixing effect."

http://en.wikipedia.org/wiki/Speed_of_sound

Re:A very old elephant in the room

[fearofcarpet]

These many many decades ago I went to a "top 5 STEM school" and enrolled in Chemistry. While all of the students had Very Expensive Machines to play with, they also had No Time To Do Careful Work.

"Flubbing" of results was considered kind of, you know, ordinary, and if you didn't get what was expected, well, you were expected to just kind of ignore it.

People like to harsh on the 'soft' subjects like history ... but I can assure you that many a history teacher would chew you up one side and down the other for saying things that are patently false and blatantly contradict the evidence - while many a Chemistry teacher would simply tell you "well, it should have worked. and you understand the basic ideas. so lets move on."

I'm not sure what the issue is - if it's just too expensive to do experiments, or maybe the point is not to learn 'experimenting' but rather to learn 'theories... with some hands on experimenting to give you a flavor of it"????

Don't conflate education and research. If an experiment goes wrong in a classroom setting, then there simply isn't time/resources/patience to repeat it. For instance, when a student uses the wrong solvent and starts a fire, you grab the extinguisher, deal with the fire, and move on. Sometimes it is the students fault, sometimes the professor, other times the TA. Some people are just not meant to do chemistry--they are afraid of chemicals and/or they just don't "get it." Other people are just unlucky. It simply doesn't add value to have such people repeat a reaction/experiment until it actually works. Besides, if you can draw it on paper fine, but can't make it in the lab, then maybe the chemistry gods are telling you to go into theory.

Now, in a research setting, a fire is no excuse to give up. I have, in the course of research, had numerous fires, co-workers going to the emergency room, explosions, fainting and--my favorite--detonating a flask full of a reagent that ethylates just about anything it comes into contact to (e.g., your DNA) and having to put on a space suit to decontaminate the lab. (And once the same situation when a mercury manometer exploded in a teaching lab.) In every case, we made notes on how to avoid such accidents in the future, and repeated the experiments until we had the necessary data to write a paper.

Re:A very old elephant in the room

[vlm]

Did he bust you for error bars / sig figs? How many sigma was your "signal" above the noise? "pointed toward" doesn't sound like a firm result, so I can see the old prof getting kinda wound up if you wrote a relatively strong conclusion in English language prose, but provided math showing a very weak conclusion. That's "TV commercial logic" not scientific logic. Still the guy was a bastard about it, because he didn't explain his (likely) reasoning to you, or at least he didn't do it very well.

Narrowing the error bars is a long tradition in physics... you don't have to announce the precise discovery of "whatever" to make a good experiment, so your hypothesis could have been fine tuned... "Can you reduce your std deviation to less than 0.1 m/s with that relatively crude gear, and if so, what is the result?" would have been a decent "real world" experimental hypothesis. I know that "science fairs" demand certain firm outcomes which make them "not science" so you gotta play along to win, but...

Re:A very old elephant in the room

[TheRedSeven]

8th grade. I honestly don't remember if we had the math to figure out things like confidence or error intervals at that point in our education. Probably should have done, but I have no idea. But the prof wasn't even in the range of "That's wrong, but within the error range given $Factor." It was simply, "I don't care what results you got, you are wrong."

And looking back, the methodology wasn't great -- tube physics do funky things to sound, so measuring through the PVC wasn't a good thing; the 1 meter length of the aquarium made the interval between echos really uncertain; the microphone was only sampling at 0.01s intervals, so the echo wasn't particularly distinct all the time; etc. But to simply be written off because I was "book wrong" was pretty disheartening at the time. And in retrospect, horrific of a college-level prof to be expecting that kind of result from an 8th grade kid.

Human Error

[Anonymous Coward]

When a new student tried to pick up where I left off on my PhD, she wasn't able to reproduce any of my results. Doubt was cast and, under the suspicioun of fabrication, I was forced to come back to the lab to duplicate my results. I did. And so did my PhD committee when I showed them how to do it - exactly as I'd explained in my methodology portion of my thesis.

The problem wasn't with the process, but with the scientist - and it's hard to control for ineptitude. I mention this because the inability for a blogger (even one with a scientific background) to reproduce peer-reviewed results is not necessarily an indictment of the results. It could just as well be a problem with the scientist.

If Michael Jordan published a paper about how to slam dunk a basketball, it wouldn't disprove his paper just because I can't jump high enough to touch the rim.

Not just chemistry

[crepe-boy]

Drug companies often find that biological science from academia cannot be reproduced (or is much less robust than indicated. Amgen and Bayer have both published on this topic, and when they called the researchers to ask why, they were told that the bettter results had been picked for the publication. Reuters article

computer science too

[Anonymous Coward]

The sad truth is that this happens everywhere, also in computer science. I once implemented a software pipelining algorithm based on a scientific paper, but when implemented, the algorithm appeared to be broken and had to be fixed first. In this case, it probably wasn't malice either. But the requirements to get something published are different from getting it to actually run. Brevity and readability may get in the way of correctness, which may easily go unnoted.

Re:computer science too

[mupuf]

So true! When will CS conferences will require an open source demonstrator?

Re:computer science too

[hawkfish]

So true! When will CS conferences will require an open source demonstrator?

Some of the database conferences (SIGMOD, VLDB) have a "reproducibility" program. You can submit your code and results to a volunteer review team and get a "reproducibility stamp" with your paper if they can duplicate your results.

Isn't there an easier fix?

[Anonymous Coward]

If there was a significant reward (monetary or otherwise) for proving that results of a paper have been fudged or there otherwise hasn't been given adequate information for replicating them... and there was penalty (monetary or otherwise) for having someone prove that your submitted paper was piece of crap... We should see the number of such papers go down very fast, without having to depend on any kind of cultural changes in the journals or communities where we already know that the culture can't be trusted (or we wouldn't be in this situation in the first place).

It seems like a very soviet approach to say "Well, we need to satisfy the publishing requirements but many papers are crap, so maybe we could demand some people to fill their publishing requirements by just trying to test which papers are crap and hit two birds with one stone" instead of saying "Let's discourage submitting crappy papers and if that causes the total number of papers to go down, the publishing requirements will be adjusted in the process".

Re:Its called... business as usual

[Anonymous Coward]

Having done some lab time, I can tell you there's a lot more to synthesis than what gets written down. Even the most precise method involves many invisible personalised adjustments that the chemist does not consider noteworthy. The easy stuff goes through to completion more or less whatever you do, while the hard stuff frustrates and denies. It can take a postgrad over a year of attempts before they hone their method to get the very best outcome.

I ran a cephalosporin conversion (a good training exercise), following tried-and-true methods to the letter, and I got 0.5% yield instead of the 60+% that was expected. Was it age and condition of reagents? Was my glassware too clean or too dirty? Did my refluxing gear not achieve sufficient temperature? Ambient humidity, pressure and temperature? Was there a hidden ingredient that I missed? A little water contamination can have surprising results, both positive and negative. There are hundreds of variables that might be inconsequential but it only takes one to completely mess up your outcome. In this case, most others in the same lab succeeded with much more reasonable results, but nobody achieved the literature value without cheating. That's what happens when someone practices something for a year. They get better at it.

I approve of the effort of the bloggers, because there is too much fudging in chemistry, but to assume you can recreate great results in only one or two attempts is foolhardy. I think getting even half the yield on first try should be considered a great success!

In fact, while I'm at it, the word "recrystallise" is doom to your literature yields. This step alone can be responsible for massive swings in yield. All it takes is for the crystals to grow too fast and you've got powder you can scarcely handle, never mind weigh. I recall losing half my product to a breeze once, because for the first time in ages I'd grown really nice fluffy crystals. The scope for error is huge.

misleading title

[Anonymous Coward]

They don't put the scientific method to the test -- they use the scientific method to test claims made by others. It's a completely different thing.

Not reproducible

[jotajota1968]

I am a physicist. I do believe that a large percentage (less say 50%) of scientific publications do not meet basic quality standards, let alone the scientific method. It depends also on the level of the journal. But even in the best journals you can find articles and results that can not be reproduced. The pressure to publish is too strong. Anyway, 70% or more of the articles are only academic exercises, or do not have robust statistics or do not receive more than one or two citations (including a couple of my own). Only the best of the articles (should) prevail with time (Darwinism).

Unreplicable Experiments No Surprise

[SoothingMist]

As a scientist, I have many people who deliberately do not put sufficient detail into their papers to enable replication of their experiments and results. Given the way R&D is funded these days and the constant quibbling over patents, this should be no surprise. The ancient publish or perish paradigm is still in vogue so people still publish but they also have to protect their intellectual property.

Re:Its called... business as usual

[Electricity Likes Me]

And yet, I would argue it's an ongoing disaster that a lot of the time we're not really trying to quantify these types of parameters. Nothing is gained if we require everyone to go through the exact same learning exercise, and we never actually dissect the common problems and figure out what causes them.

One of the main reasons I left synthetic chemistry

[Anonymous Coward]

Was the total unpredictability of reactions working or not. Published reactions often fail, even textbook ones. The other reasons were colleague competitiveness and, would you believe it, boredom during purification by column chromatography. Probably it was my ineptitude, but it was frustrating as hell. Glad to be working as a scientific programmer now, never looked back!

Me! Me! Me!...

[Genda]

Oooh, oooh, oooh I get to synthesize the MDMA!!! Rave anyone?

ACademic fraud is rife

[Anonymous Coward]

Academic fraud is rife in the brain sciences I can testify. Where I went to school at a MAJOR HARD TO GET INTO university system in California, our professors were HEAVILY engaged in fabricating both test results and grades for their undergraduates. The University system as we know it has 5-15 more years to live,. What does it do? It doesn't teach. Its business model is effectively one thing- hording knowledge. It doesn't evaluate. It doesn't police itself. It doesn't change or improve or lower it cost structure. It's populated by sociopathic careerists and grandiose administrators overseeing their feifdoms and bloatocracies. Very smart people are electing not to go into 6 digit unbankruptable debt - thew debt that keeps this bubble economy dreamworld alive. The sooner US university system comes crashing to the ground - and it won't be long now- the better. Sorry for the 1 in 3 hard working honest researchers who are going down with it.