Slashdot Mirror


Journal Published Flawed Stem Cell Papers Despite Serious Misgivings About Work

sciencehabit writes: As two discredited, and now retracted, stem cell papers have produced an almost unimaginable fallout — a national hero accused of scientific fraud, the revamping of one of Japan's major research institutes, and the suicide of a respected cell biologist — researchers have privately and publicly asked how Nature could have published work that, in retrospect, seems so obviously flawed. Another piece of the puzzle has now come to light. The Science news team received a copy of email correspondence between a Nature editor and Haruko Obokata, the lead author of the papers, which indicates the work initially received as rocky a reception there as at two other journals, Cell and Science, that had rejected the work previously. The email, dated 4 April 2013, includes detailed separate criticisms of the two papers and suggestions for new data to support the authors' claims of a simple and novel way to make stem cells that could form the myriad cell types within a body. The Nature editor rejected the papers, but left open a window, writing, "Should further experimental data allow you to address these criticisms, we would be happy to look at a revised manuscript." The two papers were published 10 months later.

35 comments

  1. Not that unusual by CajunArson · · Score: 5, Insightful

    It's very common for a paper to get rejected on the initial go-around but for the journal in question to provide hints about how the problems with the paper could be addressed to make it publishable.

    The bigger issue here appears to be that the followup process didn't happen in a thorough and rigorous manner or that all the extra data the journal requested ended up being manipulated/faked.

    --
    AntiFA: An abbreviation for Anti First Amendment.
    1. Re:Not that unusual by Anonymous Coward · · Score: 0

      Yeah, its called a revise and resubmit, and that is usually a good sign. Reject, Revise and resubmit, Accept. Few just get straight accepted, most submissions are rejected or get a revise and resubmit.

  2. This is how science works by geekoid · · Score: 2

    Notice: it was science that led to finding out they were wrong and the retraction.
    People make mistakes, that why the normal scientific process is to check it.

    Publishing is the first step of the peer review process.

    The suicide is sad.

    --
    The Kruger Dunning explains most post on /. http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
    1. Re:This is how science works by Firethorn · · Score: 1

      Publishing is the first step of the peer review process.

      Actually, I'd say writing the paper is step 1,
      Step 2 is the paper being accepted by the journal, where it is then sent out to peers for review
      Step 3 is the peer review, if it passes -
      Step 4 is final editing and publication

      --
      I don't read AC A human right
    2. Re:This is how science works by AthanasiusKircher · · Score: 2, Insightful

      Notice: it was science that led to finding out they were wrong and the retraction.

      Yes, but it was also common sense. The journal published something that was demonstrably false. A newspaper that was given bad information from a source doesn't need "science" or some sort of formal scientific review to publish a retraction saying the source was wrong.

      People make mistakes, that why the normal scientific process is to check it.

      Actually, not really. Sure, in an ideal world this is true, but not necessarily in the "normal scientific process" as practiced.

      Research grants are awarded, publications are selected, tenure is granted, etc. mostly on the basis of NEW research, not on checking up on other people's results (which is generally not considered notable unless previous results were wrong, hard to get grant money for, and hard to publish).

      We have a systemic bias against replicating research in the way a lot of modern science operates. It would be better if what you said was true more often. But paper retractions are exceedingly rare, even if much (if not most) published research has serious flaws.

      There are some relatively recent serious efforts to fix this, though. But it's a misrepresentation to say that checking research is part of the "normal scientific process" -- it only tends to happen in high-profile articles or ones that make extraordinary claims. For most run-of-the-mill research, it's definitely not as common as it probably should be.

  3. peer review is a low bar by phantomfive · · Score: 5, Interesting

    Peer review filters out the stuff that is obvious crap, stuff that doesn't even fit the form of a proper scientific article. The purpose is not to say that articles are true, but rather to get rid of articles that are obviously wrong.

    If the scientists are lying about their data, it's hard for peer review to catch that. That's why reproducibility is important. If it's a result you care about, you can reproduce it.

    --
    "First they came for the slanderers and i said nothing."
    1. Re:peer review is a low bar by LateArthurDent · · Score: 3, Interesting

      Peer review filters out the stuff that is obvious crap, stuff that doesn't even fit the form of a proper scientific article. The purpose is not to say that articles are true, but rather to get rid of articles that are obviously wrong.

        If the scientists are lying about their data, it's hard for peer review to catch that. That's why reproducibility is important. If it's a result you care about, you can reproduce it.

      Well, reproducibility is part of peer review. If anyone is making decisions based on the results of one paper, they're idiots. Even if the research methodology was flawless, and the researchers are brilliant and honest with all their data, certain results can still come about as a result of chance. Obligatory xkcd

      I wish we'd put more emphasis on reproducing published results, though. I've mentioned this before, but I feel like this would be the ideal work for grad students during their first few years, before they're deep in their own research. They need to get papers published, there should be journals devoted to publishing data from reproducing results. Students get experience writing papers and conducting research and everyone gets stronger peer review in their fields.

    2. Re:peer review is a low bar by jae471 · · Score: 1

      there should be journals devoted to publishing data from reproducing results. Students get experience writing papers and conducting research and everyone gets stronger peer review in their fields.

      Which is all fine and good until the results aren't reproducible, then the 2nd year grad student is left challenging The Distinguished Expert Doctor Professor Smith's results, which tends not to go over so well. (Good luck getting funding/grants in the future, when Dr. Smith just happens to sit on a few NSF committees. And don't expect the advisor to risk their funding/grants.)

    3. Re:peer review is a low bar by Anonymous Coward · · Score: 0

      I wish we'd put more emphasis on reproducing published results, though. I've mentioned this before, but I feel like this would be the ideal work for grad students during their first few years, before they're deep in their own research. They need to get papers published, there should be journals devoted to publishing data from reproducing results. Students get experience writing papers and conducting research and everyone gets stronger peer review in their fields.

      Yeah, I hear ya. Preach it, brother! I think it would also make for a better educational experience for the students. Unfortunately, this will take a very large cultural shift in academia where first to publish is considered the gold standard. "I confirmed someone else's research" just doesn't make for flashy headlines. It also has some very real professional hazards for the student: what if they find a major flaw in the research of their advisors? I just don't see this happening anytime soon.

    4. Re:peer review is a low bar by madboson · · Score: 1

      Peer review filters out the stuff that is obvious crap, stuff that doesn't even fit the form of a proper scientific article. The purpose is not to say that articles are true, but rather to get rid of articles that are obviously wrong. If the scientists are lying about their data, it's hard for peer review to catch that. That's why reproducibility is important. If it's a result you care about, you can reproduce it.

      However in this case, the reviewers at science did indeed complain about aspects of the paper that ended up being part of the faked results http://www.sciencemag.org/cont...:

      For the Cell submission, there were concerns about methodology and the lack of supporting evidence for the extraordinary claims, says [stem cell scientist Hans] Schöler, who reviewed the paper and, as is standard practice at Cell, saw the comments of other reviewers for the journal. At Science, according to the 8 May RIKEN investigative committee’s report, one reviewer spotted the problem with lanes being improperly spliced into gel images. “This figure has been reconstructed,” the RIKEN report quotes from the feedback provided by a Science reviewer. The committee writes that the “lane 3” mentioned by the Science reviewer is probably the lane 3 shown in Figure 1i in the Nature article. The investigative committee report says [co-author Haruko] Obokata told the committee that she did not carefully consider the comments of the Science reviewer.

      and even the nature reviewers complained http://news.sciencemag.org/sit...

      All three Nature reviewers concluded that the data presented in the submitted manuscripts were not enough to support such radical claims. “I would recommend the authors to be extremely cautious in their claims . The authors should look into the actual effect that the treatment elicits in the genome and they should assess genomic instability,” one writes. “There are several issues that I consider should be clarified beyond doubt because of the potential revolutionary nature of the observations,” writes another.

      So in the end the editors seemed to just want the sensational paper published and let the community sort it out later. Retraction watch has a nice compilation about it all http://retractionwatch.com/cat...

      --
      Mo00o
    5. Re:peer review is a low bar by nmr_andrew · · Score: 1

      OK, 100% in agreement that there should be more emphasis on reproducing results, but someone needs to step up and pay for it - the current funding situation is far from pretty. Also, in the current "publish or perish" climate, nobody is going to spend time working on reproducing somebody else's results unless it directly impacts their own work. So we'd need to change that mentality as well.

      However, if you truly believe that reproducibility is part of peer review, I've got a bridge to sell you. With most (all?) journals, reviewers aren't paid. The time they spend on peer review is typically spent AFTER they put a full day into their own research; their home institution doesn't expect any less output. They get minimal if any recognition for it, in an academic position it checks a little box for "good scientific citizen" and maybe puts a bullet point on a resume/CV. Sure, they'll ding the paper if the claims seem outlandish, or something just doesn't seem to add up, or grandiose claims are made with a sample size of n=1, but they generally expect that the submitting group has done the controls, the replicates, etc. The purpose of peer review is for the paper to pass the sniff test, not to guarantee there's no fraud.

    6. Re:peer review is a low bar by Anonymous Coward · · Score: 0

      You don't know what you're talking about. Scientists make decisions based on papers all the time. If a single paper is pertinent to your current experiment, then you need to use that as a guide. Is that smart? Maybe not, but other scientists will ask where you chose your current values from, or how you started out with some concentration of drug, etc. The purpose of publishing is to be used as a building block for future scientists.

      You literally have no idea what you're talking about.

    7. Re:peer review is a low bar by phantomfive · · Score: 1
      Well that is something different, then.

      So in the end the editors seemed to just want the sensational paper published and let the community sort it out later.

      I subscribed to Science for a while, and that more-or-less matches the quality of article I found there.

      --
      "First they came for the slanderers and i said nothing."
    8. Re:peer review is a low bar by phantomfive · · Score: 1

      I feel like this would be the ideal work for grad students during their first few years, before they're deep in their own research

      That is a good idea.

      --
      "First they came for the slanderers and i said nothing."
    9. Re:peer review is a low bar by phantomfive · · Score: 1

      If anyone is making decisions based on the results of one paper, they're idiots.

      btw, a lot of the problems mentioned in the second half of this article still exist today

      --
      "First they came for the slanderers and i said nothing."
  4. Not so schocking by Anonymous Coward · · Score: 1

    Human beings are flawed, even scientists. People pursue agendas and that which will make them both money and garner attention. The problem is that group think impacts scientists every bit as much as any other profession. The question is, how can we verify science when most of the population don't even understand it? Much like politicians and covert policy, the public has to trust experts in their fields. With so little oversight what can we do?

    1. Re:Not so schocking by ColdWetDog · · Score: 3, Interesting

      Human beings are flawed, even scientists. People pursue agendas and that which will make them both money and garner attention. The problem is that group think impacts scientists every bit as much as any other profession. The question is, how can we verify science when most of the population don't even understand it? Much like politicians and covert policy, the public has to trust experts in their fields. With so little oversight what can we do?

      Science is, eventually, self correcting. It may take months (in this case), decades (cf, Plate Tectonics) or hundreds of years (the nonsense spouted by Pliny and Aristotle). There are probably large swaths of what we take for scientific understanding that are still wrong (or not even wrong). But there is a self correcting mechanism. In this, Science is rather unique among human endeavors.

      --
      Faster! Faster! Faster would be better!
    2. Re:Not so schocking by Anonymous Coward · · Score: 1

      With respect to science being unique in being self-correcting.... bullshit. Almost all areas of "human endeavor" are self correcting, even religion, Virginia. Babies would never learn how to walk or talk, otherwise. You might tell I'm a little tired of hearing science being treated like it is a special little snowflake of truth or fact. (And the myth that Pliny and Aristotle get lumped into being "scientists.") And I'm actually a supporter of science, BTW.

    3. Re:Not so schocking by Anonymous Coward · · Score: 0

      Notably, you have absolutely no evidence that science is actually self-correcting. You espouse that premise purely on faith (not the typical religious usage of "faith" as confidence based on partial information, but "faith" in the Dawkinsian equivocation style--you have -absolutely- no evidence of your stance, by definition).

      We have seen manifold instances of science being wrong in the past, and the model has been changed to something you presume is now indefinitely "finally correct". You have no reason to conclude this is any given case, other than you like the idea that you can make that determination, which you can't. You have no instances to draw from which meet the criteria of "self correcting", that is, implicitly known to be ultimately correct, or to be corrected in a finally definitive manner.

      So: The data points actually in your favor--none of them.

      Your statement as to which data points are in your favor--all of them, even the ones provably not.

      I suggest you re-examine your epistemology and/or claims to psychic powers as to the indefinite future a bit more closely.

    4. Re:Not so schocking by AthanasiusKircher · · Score: 1

      Science is, eventually, self correcting. It may take [snip] ... hundreds of years (the nonsense spouted by Pliny and Aristotle).

      While it's somewhat refreshing to see someone here acknowledge that ancient writers did do something akin to modern science, it's a little strange to try to lump Pliny, Aristotle, and modern science into one continuous method.

      Pliny the Elder was trying to write an encyclopedia, essentially a collection of anything anyone had ever reported or discussed about the natural world. While Pliny did some notable investigations of his own, his work in the Natural History is more of a collection of claims rather than anything like modern science. (For some entertainment, read the parts of the Wikipedia article about his methodology, or better yet, read the description in the original and his nephew Pliny the Younger's discussion of the method -- basically, the guy just read books day and night, made extracts, and shoved them all in a giant treatise. Hardly "science.")

      As for Aristotle? Well, he definitely did experiments, and contrary to the way we often talk about him in the history of science, his explanations were often pretty good and logical. He was respected as authority not just because he was an authority, but because his models of the world seemed to make sense according to observations of the time (such as they understood them).

      But a lot of the modern scientific method was about overthrowing various aspects of Aristotelianism in the 17th century. And I'm not talking about specific facts or theories -- I mean the basic method.

      Anyhow, my points are that (1) these writers didn't deliberately write any "nonsense" and they were widely respected for THOUSANDS of years because a lot of their writings agreed with things as observed and known at the time, and (2) despite this, what ultimately corrected their misunderstandings was a reconfiguration of the way we do research and model the external world... so the methods of thousands of years ago vs. today are not really comparable, nor is the method of correction.

    5. Re:Not so schocking by Anonymous Coward · · Score: 0

      Not op, but you've made an assumption in your response that isn't necessarily correct.

      We have seen manifold instances of science being wrong in the past, and the model has been changed to something you presume is now indefinitely "finally correct". You have no reason to conclude this is any given case, other than you like the idea that you can make that determination, which you can't. You have no instances to draw from which meet the criteria of "self correcting", that is, implicitly known to be ultimately correct, or to be corrected in a finally definitive manner.

      The incorrect assumption is that 'self correcting' means 'has become correct'. Something that's self-correcting moves closer to being correct, but that does not automatically mean that it ever becomes correct.

      There are a great number of theories held today that are demonstrably 'better', or 'more correct' that they were in the past. A simple example would be the concept of elements - moving from the Greek basics of Water, Fire, Earth, and Wind, to atomic elements such as hydrogen and aluminium, to sub-atomics and modern quantum theory. Each additional step along the way gives us greater precision, enabling us to better predict the results of certain actions and, once applied, opening up new material sciences.

      The proof is in the application - Simple building construction and metallurgy to petrochemical engines & synthetic materials (eg, nylon) to atomic power & smoke alarms to flash memory.

    6. Re:Not so schocking by Anonymous Coward · · Score: 0

      The proof is in the application - Simple building construction and metallurgy to petrochemical engines & synthetic materials (eg, nylon) to atomic power & smoke alarms to flash memory.

      What this seems to "prove" is we have a model that works in a given context. The notion that the model can be functional while not correct in its underlying premises, and/or localized in its applicability, tends to get lost in discussion. The Luminiferous Ether model worked great for most all purposes, until it was refuted. "Correct" should ultimately be binary--it either is, or it is not. To really get a clear idea of what "more correct" actually means I think would take quite a bet of careful parsing for your and other examples. It seems we tend to use this term colloquially to mean "more comprehensive" or "more functional" or "more reliable".

      I suspect that like it is often an arbitrary choice whether to express geometrical relationships using either Euclidian or Riemannian geometry, or the various ways we can conceptualize "gravity" in terms of what it "really is", or, say, the fact we design circuits "as if" electricity flows from positive to ground, when it actually does the reverse, there are multiple constructs that could be fundamentally contradictory to each other and yet "work" on a practical level (physics according to the various Interpretations of QM, for example). I am not sure that "correctness" specifically is the best attribute to infer from the fact a model works in this sense.

      In any case, interesting questions. My previous post (and from your response I sense you agree) was intended to demonstrate much more nuanced statement is required, to be accurate, than essentially "no worries, science always corrects itself".

  5. So, you're saying that... by Jawnn · · Score: 2

    ...the peer review process for publication actually works? You just have to use it? Who knew?

  6. Publish or perish by Anonymous Coward · · Score: 0

    That Japanese dude to this to heart.

    1. Re:publish or perish by Anonymous Coward · · Score: 0

      gives these guys something to do until they ultimately discover that God is real.

      Sooo, until never, then?

    2. Re:Publish or perish by halivar · · Score: 2

      I think you accidentally a verb there.

  7. publish or perish by Anonymous Coward · · Score: 0

    Publishing a paper and then retracting it gets you twice as much publicity. Stephen Hawkings knows this. NOT publishing a paper gets you fired. It is getting a little crazy nowadays, with scientists publishing papers contradicting basic stuff like speed of light. Still, it's a good read and gives these guys something to do until they ultimately discover that God is real.

  8. not a fair criticism by Goldsmith · · Score: 1

    I think the majority of the scientific publishing culture and industry is bad for science. That said, this is not a fair criticism. It's entirely reasonable to tell someone you expect to see more data in order to publish and to start a conversation among the editor, reviewers and PI as to what is necessary to prove a point. Research is not a perfect process and does not progress in an orderly, predictable manner. There are going to be typos and blind spots in any paper.

    In this case, obviously Nature should not have published in the end. We can't know how that decision was reached unless we see all the correspondence between the editor, reviewers and PI. It would be much more useful to the scientific community to see how the PI managed to convince the reviewers to allow publication, rather than to debate what is really a standard rejection response.

  9. The papers have Harvard names on them.. by Anonymous Coward · · Score: 0

    This is why. Nobody dares to criticize the author from Harvard. Instead, the Japanese researchers become victims.

  10. free the (also) innocent stem cells by Anonymous Coward · · Score: 0

    little miss dna cannot be wrong https://www.youtube.com/results?search_query=stem+cell+therapy all the moms everywhere are crying for us all the time now?

  11. Nature is a FOR PROFIT journal by Anonymous Coward · · Score: 0

    so they have an interest in publicity (an PR is good PR)
    google "benveniste telephone nature"

  12. This changes anything? by oheso · · Score: 1

    As others have pointed out, such criticism is typical early in the process. And it does nothing to alleviate Nature's failure to appropriately review the final paper before publishing.

  13. Re:As the Bush Crime Family... by Anonymous Coward · · Score: 0

    greenwow, you fail at trolling. Quit trying you moron.

  14. Half the papers in Nature are wrong by Anonymous Coward · · Score: 0

    The other half are retractions.

    Speaking as someone, back in the day, who had a paper rejected from Nature,
    and later quite happy about that fact, I can say that Nature is not really a
    science journal. It is a science magazine, pretending to be a journal.

    At nearly the same time that I had my paper rejected from Nature (and later
    published in a respectable journal), I saw them publish the absolute
    worst paper I've ever seen in my field before or since. In a two page letter,
    there were at least 3 independently fatal errors and about half a dozen
    addtional debatable points.

    Nature is the worst publication venue I could ever imagine.