Slashdot Mirror
Scientists Themselves Play Large Role In Bad Reporting
Hugh Pickens writes "A lot of science reporting is sensationalized nonsense, but are journalists, as a whole, really that bad at their jobs? Christie Wilcox reports that a team of French scientists have examined the language used in press releases for medical studies and found it was the scientists and their press offices that were largely to blame. As expected, they found that the media's portrayal of results was often sensationalistic. More than half of the news items they examined contained spin. But, while the researchers found a lot of over-reporting, they concluded that most of it was 'probably related to the presence of ''spin'' in conclusions of the scientific article's abstract.' It turns out that 47% of the press releases contained spin. Even more importantly, of the studies they examined, 40% of the study abstracts or conclusions did, too. When the study itself didn't contain spin to begin with, only 17% of the news items were sensationalistic, and of those, 3/4 got their hype from the press release. 'In the journal articles themselves, they found that authors spun their own results a variety of ways,' writes Wilcox. 'Most didn't acknowledge that their results were not significant or chose to focus on smaller, significant findings instead of overall non-significant ones in their abstracts and conclusions, though some contained outright inappropriate interpretations of their data.'"
The article was on quantum mechanics fer chrissakes!
.
Prisencolinensinainciusol. Ol Rait!
Just make it standard for science reporters or editors for the science section to ignore the abstract entirely.
I've decided to stop wasting my time responding to AC trolls/sockpuppets... so if you want a response from me... login.
Maybe Christie Wilcox is also doing too much spinning on the subject ...
Who knows
Whereas the mundane gets nothing. For every person murdered, or in a car accident, there are thousands in the area who had a humdrum day. For every house that burns down, thousands don't.
People who hear about these bad things and think the world is going to heck, are forgetting that nobody cares to hear about nothing happening.
Fund science like you fund business, and it becomes an exercise in marketing and hot topic buzzwords.
OK, it might take more energy to make a solar panel than we'll ever get back from it, but look at the economies of scale that we're leveraging!
If you were blocking sigs, you wouldn't have to read this.
Who was it that reported this result? The reporters!
So basically, most reporters just regurgitate press releases rather than doing any of that actual journalism stuff. That's not unique to science/medical reporting. It happens in political reporting, business reporting, hell even sports reporting. The bad science reporting is just more obvious because it's easier to debunk.
Support Right To Repair Legislation.
Let's face it, to be successful in one's lifetime in any field requires some sort of self-promotion. I'm sure having a well known name makes it a LOT easier to get funding, tenure, book deals, etc ...
I remember writing a post about this phenomena about a year ago. The short version of the story is that over the last 30-40 years, universities and research institutes have increasingly recruited "scientist" with strong tendencies towards showmanship, fraud, lying and bullshitting. This change is largely due to changing nature of incentives as well as methods of evaluation and promotion in these institutions. Peer reviewed research and grants are probably the biggest culprit. Here is the link: http://dissention.wordpress.com/2011/02/06/why-all-publicised-breakthroughs-are-lies/
but are are journalists, as a whole, really that bad at their jobs?
If I didn't see egregious errors like this while reading online journal articles I'd say no, but as it stands...
This is nothing more than the so called journalist trying to pass the blame. The fact is, stories are sensationalized all the time and it does not matter what science it is, or what the subject is. Science, Education, Politics, Economic, Environment, etc. etc. So stop blaming the scientist.
If even 1/10 of the hype about "breakthroughs" in solar cell efficiency were actually to be combined and made real in the marketplace, we'd all be charging the utility companies now instead of the other way around.
1. Science is quite boring. By nature it's supposed to be, objective, logical, and devoid of feelings. But Scientists themselves are not typically boring people, they're humans, and humans are emotional beings.
2. Scientists aren't communications experts and suck at making dry discipline accessible to the public. Never was this more obvious than when I was in college. How many brilliant researchers really sucked at teaching? Pretty much most of them.
3. Scientists want to think their work matters, and therefore are inclined to extrapolate applications of their science to the public. When those applications get reported as a sure thing, then an exaggeration is bound to happen.
4. And of course, Science that can be show to be of great public benefit gets funding. Cha-ching!
http://www.beanleafpress.com
To be fair, university press releases are not written by the scientists who did the research, and in my experience the scientist often doesn't even get the chance to proof and correct them. I myself had my 15 minutes of international fame several years ago (the phone literally didn't stop ringing, interview requests from around the world, etc), all on account of a shockingly inaccurate press release from the university about some interesting but not earth-shattering research that I did.
I wonder if the abstracts contain spin 0, 1, 1/2, 3/2 etc. ? If the don't contain spin, is it a new type of physics?
Silence is a state of mime.
Scientists say we can't trust them!
Abstract: USA TODAY LOOK HERE! Extensive research conducted in my garage on 3 pet gerbils has yielded statistically questionable results indicating jelly beans may prevent cancer. One gerbil that was lethargic and probably had cancer was fed jelly beans over a period of 2 days. The second gerbil, which probably also had cancer, wasn't fed anything. The third gerbil was control. At the end of the 2 days, the lethargic gerbil was happily running in his wheel, completely cured, while the other 2 gerbils died of malnutrition.
Bad reporting such as taking a subset (medical science), claiming in your abstract that it applies to a much larger case (all science), and the saying that things come from a press office but blaming the scientists alone in your headline?
Yeah, I can see how scientists are to blame for that...
The question is not "are are journals, as a whole, really that bad" ... the question is...
IS OUR CHILDREN LEARNING YET?!!
Support my political activism on Patreon.
They define spin as: "“spin” (specific reporting strategies, intentional or unintentional, emphasizing the beneficial effect of the experimental treatment)" They also mention: "We considered “spin” as being a focus on statistically significant results ... an interpretation of statistically nonsignificant results for the primary outcomes as showing treatment equivalence or comparable effectiveness; or any inadequate claim of safety or emphasis of the beneficial effect of the treatment." (emphasis added)
I understand the last two, but the first point doesn't make any sense at all. You can't really make conclusions (you can, but scientists will not believe it) about statistically insignificant results. "Spin" can be good in some cases (maybe not at all in clinical research): a research group that studies DNA repair might state, "Our findings on the function of the yeast homolog of SLHDT in dsDNA break recognition may represent a novel target for cancer therapeutics." In this case, the research group doesn't study cancer at all and have no business at all (from their results) mentioning it, but this might convince a cancer researcher to consider reading the paper and possibly looking into doing a quick/cheap experiment targeting SLHDT and testing this claim.
Global Climate Change
Science news cycle
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
Let me think about it for a moment. ...
YES!
Of course, if you redefine their job as what journalists do today (i.e. repeat "two sides" of everything uncritically), then no. However, since science doesn't give a shit about "balance" between views unless both are equally appliccable and useful, journalism is never going to manage to report on science. Which would make science journalism pointless.
I guess scientists are supposed to do all of the work now, are they? They have to do the science, write the papers, market it for their funders, write the articles for the news corporations, AND be the fall-guys when something isn't 100% accurate?
Whew. I'm glad I didn't remain a researcher.
A congressman was touring his district when he came upon a bunch of people in a big field with bows and arrows. They were all firing arrows in all different directions.
"What are you doing?" asked the Congressman.
"We are shooting arrows," said the archers.
"But there is nothing to shoot at," said the Congressman. "Those arrows are provided at taxpayer expense! How dare you waste them in this way?"
"Well," said the archers, "as you can see, we are very skilled archers. We can shoot arrows so far that they go over the horizon and we can't see them any more. We think there are targets out there over the horizon that we can hit, even though we can't see them yet."
The Congressman said, "Very well. But how do you know where the targets are?"
One archer said, "We just have to fire in random directions, because we don't know where the targets are."
Other archers agreed with the first one.
But then one of the archers said, "I have a different strategy. I am pretty sure that there is a target roughly in this direction, so I am shooting towards it. In fact, I think I may have already gotten a bullseye or two."
"You don't know that," said the others. "You've never been over the horizon to see whether there is a target or not. You have no more idea than the rest of us"
"Stop arguing," said the Congressman. "All of you lot, give all your arrows to this gentleman here. He is clearly the only one who has a concrete plan for hitting a target. I can't have you wasting any more taxpayer money shooting arrows at nothing."
"Wait!" said another archer. "For all I know, I might have gotten a bullseye also! I don't know where the target is, but it is possible, you have to admit!"
"Hmm," said the Congressman. "Give this lady some of the arrows too."
"Ah!" said another archer. "You know, the same thing is true of me!"
"Yes," said another. "And me!"
Pretty soon all of the archers had explained to the Congressman that they, too, could possibly have hit a bullseye, and had all been allocated arrows.
"There," said the Congressman at last. "Now the public can have confidence that their money is allocated to worthwhile projects. Keep up the good work, but don't let me catch you wasting taxpayer money like you were before." And he walked off, while the archers resumed firing arrows in the same directions as before.
Surprising xkcd link
Is that "surprising" in the sense of "not an"?
systemd is Roko's Basilisk.
What a horrible, defeatist attitude. I can't stand bugs in software (the vast majority exist because low standards are cheap). So I should chose a different profession?
I've known since 1978 that "bugs were part of the job" and yet I persist.
Yours is an interesting perspective. The optimal solution to the marriage problem of jobs to talent is the assignment of least remorse: scientists who research on animals should have no feeling for animals, computer programmers should feel no embarrassment over bugs, politicians should enjoy lying, racers in the Tour should be human pincushions, etc.
To some extent, the world does work this way, but it's a strangely sociopathic step to actively endorse this.
It is worse than this, most of them actually have no idea what they are doing. 80% of what is published is actually inconclusive due to crappy study design.
I thought the peer review process and the integrity of scientists, their ideals and dedication to the truth made science unimpeachable. You mean scientists can be BIASED? Noooooo! They're objective! Say it aint so, Joe!
The abstract for this study itself was sensationalized.
Hardly "Scientists themselves", is it?
Guess I'll go back to buying organic food again now.
There is of course some spin in scientific reporting. This is to get published. Journals will only publish article there they find the most interesting. Scientists are required to put the most optimistic language, the most exciting looking data, so that scientists use their work to build off and cite the publications.
This is a basic problem in the way science currently operates. The funding for each scientist is usually based on how many articles they publish, compared to the competitors. So we have to spin every result we can get our hands on to publish and get cited in other publications.
The competitive nature of funding in science has forced us resort to spin rather than accurate language and completely ignore negative results.
One of the worst "bad abstract tricks" is putting your findings as Odds Ratios. What's an Odds Ratio? You probably know that the "probability" of an event is "Event over Total". The probability of rolling a 6 on a standard die is 1/6. The "odds" of an event is "Event to Not Event". The odds of rolling a 6 are not 1:6, they are 1:5 for (or more often said, 5:1 against). So then the odds ratio (OR) of two groups is the ratio of ratios, or the ratio of the odds for one event compared to the odds of another. So a big source of confusion is thinking the odds and probability are the same thing. Clearly they aren't. And clearly the closer they get to even odds, the bigger the difference. The odds of tossing a coin and getting heads are 1:1, but that's a 1:2 probability.
An example of the odds ratio in action: You ask 1000 men if they smoke, and you get 300 who say "yes" (made up statistics). That's odds of 300 to 700, or 3:7. You ask 1000 women if they smoke, and 250 say "yes". That's odds of 250:750, or 1:3. The odds ratio is then (3:7) : (1:3) or 9:7, or 1.2857...:1 So in the abstract you will see that this study has found that males have an OR of 1.29 when compared with women. And they'll just sit back and let the journalists call that "almost 30% more likely!" When it's not. That's how much higher the odds are, and odds are not probability! And of course you can't forget about confidence intervals. It's actually even worse than that. An increasing number of medical papers will take the OR of 20:1 and go straight to "20 fold more likely to blank!" when the probability ratio is 3.5:1 not 20:1.
Part of the problem is not enough statistics courses for scientists. I had to take 2 as part of my degree, and they never covered odds ratios, or odds at all actually. Only probabilities, which are more useful to reason about usually. This is further compounded by people using odds and probability interchangeably. I see on things like scratch and wins and store give aways "Odds of winning 1 in 3", which is a probability.
ASCII stupid question, get a stupid ANSI
This article is very well worth reading:
http://www.theatlantic.com/magazine/print/2010/11/lies-damned-lies-and-medical-science/308269/#
I am sure that there are many other solipsists out there.
xkcd has so gained so much control over Slashdot users that they assume every web comic must be xkcd.
If all you look at is the abstract and "conclusions", of course you're going to get an unbalanced view of what the study said. Think of all the other information that is contained in the body of the paper. There's a discussion of the methodological limitations of the study, there's a discussion of all the outcome measures which DIDN'T reach statistical significance, there's a discussion of adverse events, and there's usually also a discussion of where this study fits into our knowledge of the topic as a whole (e.g., "We found fish oil supplements to be effective for arthritis but 5 of 7 previous studies have shown the opposite results"). All of this is crucial information, but you couldn't stuff it all into the abstract even if you tried.
So yeah, I'm going to lay 90% of the blame for this on the journalists. My impression is that most of them don't read the body of the article. In some cases, I would bet that they don't even possess copies of the article (since articles, unlike abstracts, are usually kept behind a very expensive paywall). That would have been an interesting statistic for the researchers to look at.
This seems like a weak attempt to shift the blame for bad reporters. Their job is to get at the facts and report what is really true. That's what reporters do - at least if they're any good. So scientific press releases contain spin? Shocking! Just like press releases in absolutely every other field. Any reporter who just parrots a press release without understanding it and getting at the truth is a bad reporter.
Yes, science is complicated. Yes, it takes specialized knowledge to understand. Just like every other field. That's why there are science reporters who supposedly have that specialized knowledge.
"I'm too busy to research this and form an educated opinion, but I do have time to tell everyone my uninformed opinion."
Check the first sentence assholes.
http://www.accountkiller.com/removal-requested
'Half the money I spend on [advertising/science/military/etc.] is wasted; the trouble is I don't know which half.'
Comment removed based on user account deletion
This just in: Victims play a significant role in robberies, handing over thousands of dollars of possessions each year and incurring many times that in medical costs.
When reached for comment on their continued collaboration with their robbers, victims remained unhelpful and unwilling to comment.
You did it just there, for example: "researchers don't need to exaggerate? Then why do they... frequently?".
Where?
How frequently does frequently have to be? The same "frequently" that a uranium atom is frequently decaying in a ton of radioactive U235?