Slashdot Mirror
Crowdsourcing and Scientific Truth
ygslash writes "In an
opinion piece in the New York Times Sunday Review,
Jack Hitt states that comments posted to on-line articles, and elsewhere on line,
have de facto become an important factor
in what is accepted as scientific truth. From the article: 'Any article, journalistic or scientific, that sparks a debate typically winds up looking more like a good manuscript 700 years ago than a magazine piece only 10 years ago. The truth is that every decent article now aspires to become the wiki of its own headline.'"
It's long been true that a top reason to go to academic conferences isn't only for the paper presentations, but rather for the hallway/dinner/bar conversations about those papers. More formally, many scientific journals will publish short letters or commentaries about papers they've previously published, and that practice used to be even more widespread (at some journals a "letter" has morphed into a mini-paper, but they used to really be letters to the editor).
The same is now true online with something like Terence Tao's blog: it's interesting as much for what other mathematicians post in reply, as for what Tao himself posts (though his posts are quite interesting). The main difference as I see it is that the number of people participating is much greater (which has good and bad parts), and, in comparison to hallway conversations, the conversations persist and get referenced back to more, since they're in a semi-durable written medium (that's the "wiki-like" aspect the article discusses).
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
Since you're our resident first-poster for this article you're forgiven for not having RTFAed. :) The article actually sings the praises of comments sections in their ability to dissect evidence more efficiently than one or a handful of time-constrained professionals, points out how similar annotations in old books gave rise to the first dictionaries, argues that we need to treat 'comments' sections more respectfully as a result (and call them perhaps 'glosses', borrowing the term used for said mediaeval–Renaissance marginalia), and then insinuates that the original article was undoubtedly in error, particularly since the bird has been "sighted" very several very convenient times in the past when conservationists' efforts were frustrated.
Bio questions? Ask me to start a Q&A journal. Computer analogies available for most topics!
Is the article author aware of how pervasive astroturfing is in the comments sections? Perhaps if the article is about a subject that no one has a financial or political interest in, comments sections could serve this way. But as soon as someone's got an interest to protect, you can't trust the comments to be anything other than posts made by paid people creating fake personas to do so. Slashdot has had articles about this type of astroturfing before.
Its interesting to observe how much commentary on articles is devoted to shouting down the opposition rather than making observations and/or corrections of the original content. Given the accelerated pace of such discussions on-line, the utility of spurious research in support of questionable legislation has been reduced significantly. In other words, if you spot an ivory-billed woodpecker today, your claim might not survive long enough to secure funding or implement conservation measures. That is; without your supporters declaring that the time for further research and comments is over and now its time to act.
Today's crowdsourcing serves to reduce the half-life of bad science.
Have gnu, will travel.
There is a huge difference between scientific truth and, alternately, popular perception. I don't even want to try to explain that, it's so obvious - and there may some be more pertinent matters to address, in this.
I think we can accept that comments sections do not make much of a forum for development of scientific anything. Comments are comments. Comments are not journal articles. Comments can be said to be peer reviewed, to some extent, but then again, comments are not journal articles, comments need not follow any specific format for reporting of questions and results, comments are just comments.
I'm afraid that that all may be beside any points raised in the linked article, however. What the article looks like to me, in all my sense of bias: It looks like a way of trying to excuse a lack of significant content in articles, in lieu of some kind of perceptual bias about comments. It think it's just as well for the birds, though I know it's been said, "It's the thought that counts."
Definitely not. In that sense, it's very predictably naive. I mean, you can trust me on it or RTFA, your choice. :) If only NYTimes had a non-paywalled, publicly-accessible comments section so we could tell them...
Bio questions? Ask me to start a Q&A journal. Computer analogies available for most topics!
Comments are comments. Comments are not journal articles. Comments can be said to be peer reviewed, to some extent, but then again, comments are not journal articles, comments need not follow any specific format for reporting of questions and results, comments are just comments.
I did not RTFA. I second your point. But even if we were to take a more generous view of commenting sections, the problem of noise filtering remains. Comment sections are a perfect example of what Asimov said best :
“Anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that 'my ignorance is just as good as your knowledge.'”
The amount of effort required to parse through comments to find gems of significant value is enormous. I know that this is that age of the crowd and so on, but there are certain issues on which the opinion of the crowd has on average very little value because of the complexity of the topic and the years of experience required to make informed conclusions. The trade-off between expert opinion and open crowdsourcing varies widely depending on what is the topic under discussion, and the userbase of the particular site. Vaccines and autism on a Californian site, for example.
Entia non sunt multiplicanda praeter necessitatem.
As OP (here on /., not the author of the article), I'd like to re-raise here in the comments a point in my original post
that got edited out:
There are many who disagree with the thesis of TFA. It is interesting to note that they are trying to make their point - where else? - in the comments on the article, in comments here on ./,
and elsewhere in the blogosphere.
In thinking about how I look at articles and commentary I realized I factor in comments almost as much as the article itself, particularly any inherently subjective article, for example one that discusses the social or economic impact of a scientific discovery.
The article itself is likely to have a good signal to noise but suffers from bias, the comments typically have very poor signal to noise but can often correct or at least expose the original biasing. Taken together I at least feel like I have a better sense of 'truth', particularly if the subject is likely to expose my own bias.
In other words, yeah the article makes sense initially but I'll reserve judgment till more people have posted about it on slashdot.
Although the author set out to analyze the role comments play, I found his objective disposition of woodpecker sightings' impact on environmental fundraising. Almost bordering on cynical, the author does point out the suspicious nature of theses sightings in a way that I'm sure will ruffle the feathers of many set-in-stone environmental saviors.
https://www.accountkiller.com/removal-requested
Since "science" cannot prove historical events, the only thing left is opinion. By definition, if something is repeatable or testable, it cannot be "proven" by scientific methods. All you are left with is belief.
It'd be difficult to find a more misleading characterization of science.
First off, as various historians and theoreticians of science have observed, scientific methods rarely (if ever) "prove" anything. Scientific methods are all based on 1) proposing explanations for observations, and the 2) attempting to disprove those explanations. After sufficiently many such attempts at disproof have failed, an explanation gets promoted to "hypothesis", and then to "theory". But these are only tentative, with further attempts at disproval continuing whenever anyone can come up with a new test that hasn't been tried.
As part of this, an explanation that is untestable isn't considered scientific at all. It's neither true nor false by scientific standards, until someone comes up with tests that could possibly disprove it. Some explanations (e.g., "God did it") have remained in this state for centuries.
Actually, there is one situation where there is a sort of scientific "proof". This is dealing with negative claims of the form "There are no X".A canonical example is the old "There are no Black Swans". This was disproved by the discovery of a species of swan that is (mostly) black. It lives in Australia, so at one time it was Unknown to Science. You can rephrase this in the positive form, "There are Black Swans", and such existence statements can be "proved" by simply presenting examples. But this is generally classified as data collection, which is understood to always be incomplete. And such negative claims are generally not taken seriously by scientists unless you can give good reasons why X can't exist, based on previously accepted theories. Even then, a single (non-fraudulent) example can suffice to shoot down your reasoned argument against X existing.
In any case, "proof" is something done by mathematicians, not scientists. If you reject science that doesn't present proof, you reject all science, since proof isn't what science does.
If all you have left is belief, then you are susceptible to being defrauded by anyone who comes along with a new belief. But history shows that science's testing process has been pretty good at disproving most beliefs. In the process, the leftover ("not disproved") beliefs that fell out of the process have led to all the technical advances of the previous several centuries, something that the earlier purveyors of belief systems ("religions") have failed to do for as long as we have recorded history.
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
What's common is the accusations of astroturfing by fans of competing products, theories, or systems. It's become the "in thing" to claim that any dissenting opinion is astroturfing.
Unfortunately for those who like to use terms without understanding their meaning, it's only "astroturfing" if you're compensated for broadcasting a statement. People are entitled to share their opinions as they see fit provided they're not paid to take a side.
Even if they disagree with you.
I do not fail; I succeed at finding out what does not work.
In truth, marginalia practicaly never make it into proper glosses. Glosses are usually assembled from authoritative texts that discuss the passage in question. And very few texts get the Gloss treatment: in the Latin world, it's the Bible, Corpus Iuris Civilis and Decretals above all. Some other texts might get glosses, but they rarely get a glossa ordinaria-class treatment.
..., for these reasons. I disagree, rather saying this, for these reasons. To his reasons, I reply..."
And to the midrashim comment in TFA, I'd point out that Rashi did a bang-up job himself in Hebrew.
For the scholastic Middle Ages, criticism usually took the form of "one doctor says this
Same as it ever was.
The Web over Internet is a true cognition multiplier.
Very true, the only problem is that the multiplicative factor is highly variable and in many cases can be much less than 1.0.
Somehow I don't think we're exhaustively on the same page here. That's just a summary of the article—in practice the web is as full of bugs and dirt as anything else. Wikipedia notes that Dr. Barbour's theory of timeless physics is considered controversial, and the fact that he stopped publishing academically in the late sixties suggests that his work may not have been put through rigorous critique. If anything what you're saying highlights the greatest danger of the web: it becomes trivial to avoid critical analysis of what you're looking at if it makes you uncomfortable, and it's hard to be certain of the authoritativeness of a given statement.
Bio questions? Ask me to start a Q&A journal. Computer analogies available for most topics!
Astroturfing is creating a grassroots appearance when no such grassroots movement existed previously. Money doesn't have to change hands as directly as you hint at, and certainly not 'if you're compensated for broadcasting a statement'.
Here's the difference: Astroturfing is also the term for when a well-funded interested party creates and funds a 'think tank' or 'activist group' that has the unstated mission of forwarding their goal. The astroturfing group can recruit people that share an interest, track membership, fund gatherings, push information to assist their membership, and promote spokespeople without giving any of these people money. Every person that appears on behalf of that position has their activist efforts easier due to all these little helpful moments, but can be completely unaware that they're 'funded'.
A mob that splits the work and cost among many members to lighten the load, and lacks deep pockets behind the scenes is a 'grassroots' organization.
An org with deep-pockets supporters lightening the load is astroturfing.
Having (insert celebrity) call attention to your case isn't astroturfing.
Having (insert celebrity or wealthy benefactor) provide substantial direct funding is astroturfing.
There are many shades of grey between these two, but I'd venture that a key detail is whether wealthy benefactor(s) want to use grassroots' perceived lack of economic motive to hide an existing economic motive, and/or to hide their financial backing.
Sorry this went long; am interested in the topic-- I'm willing to concede this was written ad hoc, so other views / opinions are welcome.
I just occasionally see 2nd-order astroturfing firms (for lack of a better term) out there that do as I've described: surround themselves with unpaid teammates and pretend like they're so good at all the hassles of a campaign because of supporters or great karma, rather than the cash that funds all the background efforts that otherwise would take considerably more volunteer supporters.
Heh. We all can have fun mocking the "social science" sort.
But there's the perspective of the old wise crack, attributed to various singers (Louis Armstrong, Bill Broonzy, et al.) when asked if what he was singing was folk music: "I ain't never heard a horse sing a song.".
Similarly, all known science is done by gangs of humans, so it is inherently a social construct. Various historians of science have elaborated on this, by explaining that the scientific method has been rediscovered many times in many societies, but only one of them has developed a successful "science". The reason is that, in all the others, scientific methods have been developed by very small groups of people, typically in a "guild" or sorts, who hold the information very close and don't share it or their methods with outsiders. Eventually a small group dies out, and their knowledge dies with them.
The founders of modern, Western "science" developed not because they also discovered scientific methodology. Rather, their success was from their process of open publication. This enabled the "standing on the shoulder of giants" phenomenon, as Isaac Newton put it. But even that quip wasn't original with him; he just found a more elegant and memorable way of expressing it than his predecessors. It was published, so we remember it. With open publication of methods and results, Western science became a social construct that slowly spread to a large population. With that population, plus all the published material from previous generations, it all snowballed into the world-changing system that we see today.
No single human could have ever done this. It required a social system, with massive sharing of information. Calling it a "social construct" is merely an elegant way of saying all this. It's why modern science has been so more successful than previous local, personal development of knowledge. It's also much of what gave a small, local population on the western fringe of a continent so much control over the rest of the world.
As the biologists have been telling us, the dominant species tend to be the social ones. And it's their social behavior that makes them win over their less-social relatives. We see this within our own species with the sub-population that developed the social construct that we call "science".
Of course, I wouldn't deny the fun we had when Alan Sokal managed his publication feat. That was hilarious. Nobody says that social things are always correct. The history of science is full of mistakes and dead ends. Open publication means that society can learn from them, and not repeat them.
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
Is it just me, or is it quite ironic that at the time of writing this, this will be only the 47th comment on a subject active for 12 hours on slashdot. Apparently metacrowdsourcing (crowdscourcing about crowdsourcing) isn't all that popular.
For reference sake, there have been 7 more articles up (at least by my filters) which have already garnered the following amounts of comments. Notice that the less scientific seems to be where more, er, uh "crowdsourcing" happens. :)
Is Google the New Microsoft? --> 366
Study Aims To Read Dogs' Thoughts --> 113
Apple Security Blunder Exposes Lion Login Passwords In Clear Text -- >144
Biochemist Creates CO2-Eating Light That Runs On Algae --> 76
Some USAF Pilots Refuse To Fly F-22 Raptor --> 191
Ask Slashdot: What Language Should a Former Coder Dig Into? --> 229
Unblocking The Pirate Bay the Hard Way Is Fun --> 51
One man's pink plane is another man's blue plane.
I would say "no". To my mind there has to be malice, i.e. a hidden agenda for an expression to count as astro turfing. I.e. it's a fake, grass roots movement. An organisation/special interest group that is open with the fact that that's what they are, cannot thus be astro turfing.
Stefan Axelsson