Slashdot Mirror
Registered Clinical Trials Make Positive Findings Vanish
schwit1 writes: The requirement that medical researchers register in detail the methods they intend to use in their clinical trials, both to record their data as well as document their outcomes, caused a significant drop in trials producing positive results. From Nature: "The study found that in a sample of 55 large trials testing heart-disease treatments, 57% of those published before 2000 reported positive effects from the treatments. But that figure plunged to just 8% in studies that were conducted after 2000. Study author Veronica Irvin, a health scientist at Oregon State University in Corvallis, says this suggests that registering clinical studies is leading to more rigorous research. Writing on his NeuroLogica Blog, neurologist Steven Novella of Yale University in New Haven, Connecticut, called the study "encouraging" but also "a bit frightening" because it casts doubt on previous positive results."
In other words, before they were required to document their methods, research into new drugs or treatments would prove the success of those drugs or treatment more than half the time. Once they had to document their research methods, however, the drugs or treatments being tested almost never worked. The article also reveals a failure of the medical research community to confirm their earlier positive results. It appears the medical research field has forgotten this basic tenet of science: A result has to be proven by a second independent study before you can take it seriously. Instead, they would do one study, get the results they wanted, and then declare success.
In other words, before they were required to document their methods, research into new drugs or treatments would prove the success of those drugs or treatment more than half the time. Once they had to document their research methods, however, the drugs or treatments being tested almost never worked. The article also reveals a failure of the medical research community to confirm their earlier positive results. It appears the medical research field has forgotten this basic tenet of science: A result has to be proven by a second independent study before you can take it seriously. Instead, they would do one study, get the results they wanted, and then declare success.
It appears the medical research field has forgotten this basic tenet of science: A result has to be proven by a second independent study before you can take it seriously. Instead, they would do one study, get the results they wanted, and then declare success.
Uhh... that's not even wrong. It's either something that started out OK and has been edited down into nonsense, or it shows such a fundamental misunderstanding of how controlled experiments work that, well, it's not even wrong.
Similar issues have shown up in other fields. Psychology has had serious faillures to replicate many major studies http://www.slate.com/articles/health_and_science/science/2014/07/replication_controversy_in_psychology_bullying_file_drawer_effect_blog_posts.html and when there have been attempts to replicate them they have often not gotten the same results. And there are very similar problems in education https://www.insidehighered.com/news/2014/08/14/almost-no-education-research-replicated-new-article-shows. Pre-registration of experiments is important, but it would also help a lot if there were journals dedicated to replication and also if academia took replication more seriously: I know people who are tenure track who haven't tried to replicate some studies because it doesn't look as good for tenure promotion to just replicate something rather than do something new. There are serious cultural issues that need to change.
The basic flaw is worse. They didn't just run one test, find the results they wanted and go with it. They ran a test with only an idea of what they wanted, then took all the results they got and picked out ones that were positive for conditions or treatments they could go with. It's like going into a test for a drug to treat heart attacks, finding that it doesn't do anything for heart attacks but does seem to lower cholesterol levels, and announcing that the trials of your new cholesterol medication were positive.
Having to declare up front what their goals are destroys the ability to cherry-pick like this. What we're seeing with the drop in positive results isn't so much the difference in clinical effectiveness of the drugs but the dragging into the spotlight of the pharma companies' ability to predict what their drugs will do and how well they'll do them. There's a very interesting blog here that covers a lot of this, and one conclusion that keeps coming up again and again is that medical biochemists and researchers don't really have a good way of predicting from lab results what a compound will do in a live human. It also highlights fairly often how the drug companies will keep pushing a drug through trials even though the results aren't encouraging. It's a common attitude in business and finance, that now that you've invested this much money in something you have to get some return out of it to justify the cost. It's also a common failing in gambling, the belief that now that you're in the hole you have to dig yourself out somehow. But in gambling, if you're holding a bad hand your best bet is to fold. Don't worry about how much you've already got in the pot, it's already lost. Fold and cut your losses before you throw any more money away. Drug companies are notoriously bad at making that decision to walk away. They're also notoriously bad at dealing with a field where there aren't many good rules you can follow to get results. MBAs like process and procedure and predictable results, and right now biochemical research is in a situation where the new stuff is all likely out in areas where there isn't a lot of research, there isn't a good map of the territory and you're going to be doing a lot of "poke it with a pointy stick and let's see what it does" work.
Nobody "forgot" that. It's just that clinical trials often take years and involve thousands of people in the later stages, making them very expensive. Funding to replicate results is always lacking, particularly if it might disconfirm a positive finding - you're not going to convince a drug company to pay to have the effectiveness of their drug thrown into dispute.
It's the same in computer science. Many measurement methodologies are plain wrong or misleading. And in many cases, the source code of what people did is not available, so independent evaluation is not possible (someone also published a paper about that, where the author couldn't even get the code in many cases after explicitly asking for it, but I forgot the title). It's not just a problem of alpha sciences.
Donate free food here
...such rigor were required of Climate Science.
And now the onus is on your to back up what you say and prove that such rigor isn't required of climate science, or that the results are being significantly skewed because of their funding source. Just because what the science says does not match your gut instinct is not proof. Just because it could happen is not proof; we need evidence of inconvenient funding sources being omitted from research, or a meta study showing differing results depending on how the methodology is described.
In fact, I would go so far as to say that most climate scientists have done a good job of keeping corporate interests away from their research. If a scientist wanted to ensure future funding and be free of political interference, then they would take the easy path of downplaying global warming. No scientific research organisation has been defunded or disbanded because they gave evidence that contrary to man's impact on the environment.
...profit!
You'd find out we're in even deeper shit than even the alarmists realize.
no, no, no...
it's not about the 'basic tenet of science',
it's about the 'basic tenet of profit'.
It appears the medical research field has forgotten this basic tenet of science:
It's almost as if it isn't science at all, but rather advertising, where the target audience is a government agency that gives the company permission to transfer the product to their other advertising division who then advertise it to doctors and the public. What percentage of these clinical trials are trials of something not destined to produce wealth for an organization if the results of the trial are positive? When a wealth generating organization approves expenditure of the large amount of money to do a clinical trial, I'm sure it also adds extra management, or transfers management of the research so as to help the results of the trial be more profitable than just doing science ever could be. I'm thankful that this requirement to register these trials exists.
I want to show that all cars are red, so I go out and start counting cars. It turns out that not all cars are red, but by choosing every seventeenth car, except if the previous one had a number-plate ending with an odd number OR a number x such that the x-th fibonachi number has a 4 as the second digit, THEN all those cars are red! Dang, that's actually what I meant, forget that thing about all cars red. Publish!
Another serious issue is that it is hard publishing negative results. So even though every researcher is playing by all the rules, they might not know that 500 others have done the same trial before and failed, and this is just the 1 in 500 which accidentally gives a positive result.
This basic idea should be applied much more broadly, and given that most publishing is electronic now, all experiments should be published and evaluated on the validity of their methodology and how interesting or insightful the hypothesis being tested is, not their result.
How much money and time is wasted trying the same negative result experiments at different labs because they don't know that someone else has tried it and gotten a negative because it was not published?
@Gadget_Guy ...prove that such rigor isn't required of climate science...
>
One way to evaluate scientific hypotheses is to look at what the events they predict, then observe nature to see if the predicted events correspond to reality. In that sense, modern climate science is an epic fail because the "global warming" predicted by their models failed to happen. (Prompting the climate-alarmist "true believers" to switch to 'climate change' (so, up or down, can't lose))
http://www.americanthinker.com...
> ...most climate scientists have done a good job of keeping corporate interests away from their research...
BS. 'Big Oil' is a red-herring to divert attention away from 'Big Government', whose grants and funding tend to force researchers to become, in effect, lobbyists for political activism in order to 'pay the rent'.
when did academia get taken over by idiots? now that the gatekeepers are dumb we're fucked and all the smart people just go make money.
Well, actually, academia wasn't taken over by idiots. It was taken over - infiltrated - by smart people who were much more interested in money, prestige and power than in scientific truth.
That's the unfortunate fact about American culture. The USA was founded on the belief that all people (well, all white males of a certain age with property, but that's a small detail) should be treated alike. No titles of royalty, nobility or gentry. No class system. No special distinctions or honours.
The result, which became obvious very early on, was a society in which the only value was money. And money, it turns out, corrodes everything that is honest, decent and worthwhile. Now that culture is flooding the rest of the world - although some nations have done their valiant best to build dykes to keep it out.
"As the sociologist Georg Simmel wrote over a century ago, if you make money the center of your value system, then finally you have no value system, because money is not a value".
– Morris Berman, “The Moral Order”, Counterpunch 8-10 February 2013. http://www.counterpunch.org/20...
I am sure that there are many other solipsists out there.
This is what happens when they intermingle...
“He’s not deformed, he’s just drunk!”
In psychology, individuals are ... individual. If you can replicate the results exactly, then you must be doing something wrong.
This is confused. People are individuals but studies look at general trends. And those should remain the same. Humans being individuals doesn't make them immune to statistical analysis.
Similar issues have shown up in other fields.
Indeed. The biggest issue is statistical ignorance, but even people with a decent amount of training in stats can be fooled if they want to find a particular result. Anyhow, whenever things like this come out, everyone always thinks it's about scientists who manipulate data deliberately. While that happens, it's more often just researchers who "try things out" after collecting data and notice a pattern (unintentionally skewing things). If they have to declare methods and statistical tests beforehand, it's harder to make these errors.
A few months back, I happened upon a very useful guide to the problems in modern scientific publication, which can be found partly online here. I ended up buying the print edition, and the sheer number of examples of completely bogus research ending up being accepted in various scientific fields due to erroneous stats and various biases that creep into the publication process... well, it's just shocking. Seriously.
As the book notes, the other problem is that even finding these errors is incredibly time-consuming and labor-intensive. I specifically remember one case where a new oncology test was proposed by Duke researchers and seemed to have great results. This case eventually became so infamous that it was reported on in the popular media.
Anyhow, basically they had a couple independent statisticians analyze the work (where they found HUGE numbers of problems in mislabeled data, mistakes in analysis and basic computation, etc., which appear par for the course in many labs, if you believe the studies on this stuff in the book). Ultimately, estimates are that it took TWO THOUSANDS HOURS of work for these independent statisticians to complete their analysis and render a verdict.
And once they did this, the statisticians tried to publish it -- but major journals didn't want it. Groundbreaking results are much more interesting that tedious statistical analysis. The National Cancer Institute caught wind of the problems and initiated an independent review, which found no errors (probably because the review was done by cancer experts, not stats experts, and they hadn't been giving the stats analysis done by the other researchers).
The only reason any of this ever really got much attention is because one of the lead researchers was accused of falsifying some aspects of his resume, which led to people actually going back and questioning his papers.
The book is full of stories like this, though, as well as citations of analyses of how many journal articles in various fields suffer from serious statistical problems.
It's all really scary when you start realizing how much bogus research is out there... most of it completely unintentional, and most of it passing peer review because it follows the field's "standard methodologies."
You hit the nail on the head. By forcing researchers to declare, a priori, what they were looking for and how they were going to measure the effect they were looking for, the registry prevented cherry-picking. A researcher could not change his or her mind after the results came in, and choose another effect that had shown up in the data. Cherry picking is like staring at random noise and accidentally seeing patterns that aren't there.
Where are the experimental controls? This brazenly silly meta study sports all the symptoms of a classic research pathology: confirmation bias.
It depends on what you're looking at.
Is this entirely independent research by respected labs? Or is this research in one particular area, by a specialised lab, with a particular sponsor? The latter is, we all know but can't prove easily, biased.
Is this a paper designed to do nothing more than back up a sponsor's advertising claims? Or is it something groundbreaking from a lab with few political ties, nothing to prove, and some serious science behind it.
It all changes the perspective of what a "scientist" is (i.e. someone who work not-for-profit to forward the cause of humanity, versus some guy in a lab coat with a PhD who's sold out?). As difficult as it is for us to distinguish, imagine what that means to the general public. Those people who invent terms for things for shampoo commercials that have fuck all to do with making your hair shiny, even if they sound like that, are held in the same regard as that guy doing genetic research to hunt down some elusive connection is an ultra-rare but devastating condition.
The problem is that there is nothing to distinguish the two, and both are technically "science", and thus can generate papers and be done by postdocs.
Sorry, but this is why I believe that referees on paper should be chosen specifically, why all connections should be documented IN THE PAPER (didn't declare who sponsors your lab? Bam, you're in the shit pile and your paper is forever disregarded), and why anything published should not be accepted until it's been confirmed - ideally by a bitter rival.
Too much shit is published nowadays as "science", commercial crap, ridiculous notions, unreviewed papers, basically anything from arxiv.org, etc. And the requirements aren't strict enough.
My girlfriend's a PhD in a medical field. Her contributions to a book were copied basically verbatim by someone else and published as another chapter in the very same book. It discredits her work, and that of the plagiarist, not to mention the primary author/editor. Things like asking people to reproduce their work under independent scrutiny are the only way to verify what's true, who did it, and how it was done so it can be repeated by future generations. We've lost all that in the last 20 or so years, even in things like Maths, etc. by having people referee their own friend's papers and other crap like that.
Exactly: If you have a bunch of random data, and want p that specific effect, in order to ensure that you aren't reacting to random noise.
Enjoy life! This is not a dress rehearsal.
I also had this problem. And when I asked an PhD student about a thing that seems suspicious in his papers experiment setup he admitted that it's actually not as great as described.
Another big problem is research in development processes like Scrum, XP, Waterfall etc.. I argued quite a bit with my professor as all the experiments cited where like "take a bunch of students and let them work on a toy project, which never has changing requirements and is tiny in code size." Well this of course can't be used to make any conclusions of real software projects with real programmers, yet the results where generalized. This is just bad science. And we should call people out for it and definitely not publish this stuff. A better but still wrong approach was getting real programmers and let them work for _one_ day. This is outright ridiculous. What would be the result of this? The problems in software projects arise after _months_ not hours. Also people need time to get used to a style of working, which will at least take a couple of days.
Even worse the professor was really in awe with this experiment, as somebody actually used real developers. When I talked with him about this flaw he just waived his hands telling me, then no real experiment could be conducted. This is actually right, but what he meant was actually: this is better than nothing. I strongly disagree.
Greed is not the only problem. Yes it is a big factor, but I also saw researchers doing things they should have gotten be laughed about, not published, which happened just because of lack knowledge or rigor.
That paper did get published in a smaller journal, or something, right?
So far as I'm concerned they hadn't been testing new drugs thoroughly enough, and people were getting hurt or dying because of it.
Are YOU using the TOOL, or is the TOOL using YOU? Think about it!
That's not sufficient. In many experiments, it is easy for multiple independent researchers to share incorrect assumptions or to make the same mistake.
Trusting scientific results is something that should take decades, many repeated experiments, and an understanding of how those results fit in with the rest of science.
Americans are actually considerably less materialistic than other nations: http://www.ipsos-na.com/news-p...
Money to most Americans is a means to an end, a way to pursue the values and ideals they actually care about.
Well, Georg Simmel certainly got his wish, because about a decade after his death, his country made community, fairness, and equality the center of its value system, under the NSDAP party program. And it did it again two decades later in the GDR. And it was a resounding success: people indeed stopped caring about money, because they ended up caring mostly about not starving, not getting shot, and not getting gassed. Germany's tradition of academic philosophy that Simmal was part of bears a great deal of the blame.
And, of course, Simmel himself came from a well-off family and never had to worry about a lack of money limiting what he could do. His teachings are the philosophical equivalent of "let them eat cake".
"Americans are actually considerably less materialistic than other nations: http://www.ipsos-na.com/news-p... [ipsos-na.com]"
Americans SAY they are considerably less materialistic than other nations. FTFY.
Have you noticed that people from the wealthier nations tend to say they are less materialistic? Funny that. Moreover, I didn't say that Americans were unduly concerned about material possessions. I said that the American culture is obsessed my money - quite a different statement. In fact, money is often prized as a measure of status rather than for what it can buy. The poorer the nation, the more value is attached to possessions - again, hardly surprising.
I am sure that there are many other solipsists out there.
... if they work really hard at curbing egos and huge financial incentives. As it is, quite a bit of medical "research" is utterly pathetic and counter-productive. Unfortunately, more and more of that is happening in other fields too. For example in CS, not a lot of real scientific progress has been made in the last 30 years, due to the same stupidity, arrogance and perverted incentives.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
"The problems in software projects arise after _months_ not hours."
I'm doing some game coding right now. I get problems cropping up every COMPILE. What the hell do you mean by MONTHS?
Still waiting on Serviscope_minor to wake up to fucking reality and realize that Jessica Price isn't going to fuck him.
Materialism is a personal preference or choice, and asking is a reasonable way to find out what that is. But if you think they are lying, there is tons more data supporting the notion that Americans are generally less materialistic than other nations.
Nothing "funny" about it. You need to reach a certain level of material wealth to be less materialistic.
Americans are "obsessed by money" only to the degree that we can make decisions about our money that Europeans are denied. A European accusing an American of being "obsessed by money" is like an impotent man accusing a healthy man of being "obsessed by sex" simply for having working equipment and using it.
Beyond rational investment decisions that any mature adult should be allowed to do, Americans aren't very obsessed about money; most of us just accumulate it gradually. In fact, as research by Thomas Stanley and others has shown, wealthy Americans generally dislike displaying their wealth and live quite frugally.
In my personal experience, conspicuous displays of wealth and status consciousness are much more common in Europe. In fact, the pettiness with which people squabble over status, proper forms of address, proper attire, etc. in Europe is really off-putting. Of course, Europe does allow people to gain status through a few other ways, like sucking up to the political or academic power hierarchies; that isn't a good thing either.
Uh, "not sure if serious"!?
Forget compile time bugs or errors in algorithm, think more towards project management, development processes, maintainability, impact of requirement changes or new feature requests to a large project after it's basically done etc.
That is why I believe funding of trials should not be done by pharmaceutical companies. It is a conflict of interest from beginning. Somethings by their very nature should be funded by taxes, because it is too important to simply let the organization with the biggest wallet win.
Exactly so. Money and its pursuit cannot replace the more human values such as honesty, courage, kindness, courtesy, imagination, charity, etc. There is something uniquely corrupting about money, in that when you start to think about it a lot you eventually become almost incapable of thinking about anything else. That's when you start sacrificing the good things of life for more money - and more, and more. If you are Larry Ellison, you want to be richer than Bill Gates. And so on.
I am sure that there are many other solipsists out there.
Any problem the compiler can find is easy.
John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
Problem is, I doubt the actual researchers could do anything about it. If I, as a researcher, decided tomorrow morning that I'd attempt to reproduce results from a bunch of other papers, I'd most likely lose all of my funding and would have no support from my university or research institution. You'd need a full top-down rework of science for something like that to pan out.
It's ironic too because we tend to waste a lot of time trying things that don't work, concluding that they don't work, and moving on to the next thing. If we'd published the negative finding, we'd ultimately save everyone else's time.
Money grubbers are nice people next to power grubbers.
John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
Yes because climate scientists are all trying to get rich by pandering to the Government? THis is the most ridiculous argument against climate change. The only "scientists" with a demonstrable financial interest are the corporate shills denying the evidence. Take a look at the history of the campaign to end leaded additives in gasoline. The kind of corporate-funded "research" trying to discredit the voices sounding the alarm against lead in fuels sound eerily similar to what is going on in the climate change debate today.
You're kidding, right? How often do we have to chant the mantra that climate is not the same as weather. The climate models may suggest that we will see in increase in precipitation, but that doesn't mean that in one specific location that there will be more rain. Also, there are other localised factors with droughts.
None of what was in that article is enough to make the claims that all the models have failed. I also don't understand the point of the article. If 97% of the scientists agree that man has a hand in global warming, it doesn't mean that they agree on all the details. Nor does it mean that any disagreement within the community is proof that the whole thing is a big fat lie.
BS. 'Big Oil' is a red-herring to divert attention away from 'Big Government', whose grants and funding tend to force researchers to become, in effect, lobbyists for political activism in order to 'pay the rent'.
And that is even more BS. There is no proof that there is any "Big Government" that is attempting to control the scientific community, especially when 50% of those people in power are actively against the idea of climate change. Whenever you hear of political interference with the scientific process, how often is it some left-wing conspiracy to force the hand of scientists compared with conservatives attempting to shut down institutions that do research into climate change? Where is the evidence of this giant conspiracy, other than far-right pundits speculating as if it was fact?
Unfortunately, not just social science.
Sent from my ASR33 using ASCII
Scientists have a job because the vast majority of funding for climate science comes from government or NGO organizations that only fund research that is looking to confirm human caused global warming er global climate change.
Which is why most of the criticisms come from older scientists with tenure and no longer trying to maintain a research lab so don't need funding or are simply retired.
because the "global warming" predicted by their models failed to happen
Just for kicks, look at this set of organizations that disagree with you.
For what you're saying to be true, not only every one of those organizations would have to be concealing data or faking models, every single organization on the planet would have to be in on it - - because science is science.
So the claim is that the entire global scientific community is suddenly full of shit, but only on this one subject (although of course the implication is that nothing affiliated with any university or mainstream research institute can be trusted, because scientist == liar in that world).
As always, all IMO. Insert "I think" everywhere grammatically possible.
This was explained in the journal xkcd.
https://xkcd.com/882/
Ok, it's not peer-reviewed. But it has a very high impact factor.
Say you want to find out if certain candies act as a libido increaser.
You a do a study on 20 different candies. One of them seems to act like a weak form of viagra, at a 95% certainty that it isn't random chance. Except 95% is also known as one in 20 and you tested 20 candies...
If you didn't start out saying you were testing if M&M's were the viagra, you can claim that you have a positive effect. But if you were forced to specify the candy you were testing, you get negative results - and a hint to try another study.
In science this kind of thing happens often - only instead of testing 20 candies, they test one item against 20 different 'cures' (decrease weight, increases sensitivity to insulin, sleep aid, congestion aid, etc. etc.),
excitingthingstodo.blogspot.com
focused on human randomized controlled trials that were funded by the US National Heart, Lung, and Blood Institute (NHLBI). The authors conclude that registration of trials seemed to be the dominant driver of the drastic change in study results. They found no evidence that the trend could be explained by shifting levels of industry sponsorship or by changes in trial methodologies.
Are you kidding? If a scientist could actually make a good case that the current model is wrong his career would be made and he would rapidly become so rich he wouldn't need government funding to continue his work. Nothing succeeds in science like upsetting the old paradigm. Ask Marshall and Warren (Nobel prize in Medicine).
This is great news, it shows that money paying for results spoils the process. When it's opened up for peer review, one leg of any review being the ability to repeat the experiment using the exact same conditions and expecting the exact same results, corporate bias in reporting results shows up like Fat Waldo in a herd of adele penguins.
The bias being brought about by the sponsored research simply dismissing negative results as experimental error rather than seeing it as legitimate reading as far as variance in their method allows. For instance, the efficacy of vaccines has long since been biased by virtue of the historical infection rate being ignored (which for most diseases, including polio, indicates the infection rate actually reducing - naturally - to almost nil *before* the vaccine hits the market. The given example also has a side effect brought about by the use of live but attenuated cultures that has been given a name intended to hide its origin: ALS. Together these show that polio vaccine is not only ineffective, it is also unnecessary and in fact demonstrably harmful).
Political debates have me rolling my eyes so much I think I got optical whiplash. I should sue. - Foamy The Squirrel
when it was discovered that one could make a fuckload of money from presenting pseudoscience and revisionism as Truth and indoctrinating the next generation of fresh young minds to the dogma of corporate rote and the prospect of production line drudge their entire working lives.
"Would you like fries with that?"
- The single most repeated line of a university graduate's professional career.
Political debates have me rolling my eyes so much I think I got optical whiplash. I should sue. - Foamy The Squirrel
There's more to it still: you can actually exploit the incertitude on the measurements you're using to categorize your subjects into subgroups, in a way to ensure your drug WILL report positive effects even if it has zero real effect. It's very well explained in this short article by Tom Naughton, complete with a numerical demonstration.
To put it shortly: you can design the subgroups' criterion in a way that overrepresents false positives and underrepresents the false-negatives that would otherwise counterbalance them.
Maybe we deserve this world ?
admitting the obvious.
"Science" is not the high ideal people here tend to believe it is, the bulk of published stuff is garbage, scientists lie all the time, and most so-called "research" is never verified much less challenged.
Stop worshiping and start using your ability to think critically for a change.
Only every experiment that gets readings from instruments with known biases? What the fuck do you think scientific instruments are, oracles? You think they're made of unobtainium and starshine, wishes and wands? Lab instruments can be expensive and delicate and uniform.
Weather stations can't. Lab instruments can be operated in temperature- and humidity- and everyotherfuckingthingelse-controlled environments. Weather stations can't. They degrade, making the instruments behave differently. Conditions change, making the instruments behave differently.
But then, people do tend to judge others' characters by their own.
As always, all IMO. Insert "I think" everywhere grammatically possible.