User-Generated Content Vs. Experts

Jay points out a Newsweek piece which suggests that the era of user-generated content is going to change in favor of fact-checking and more rigorous standards. The author points to Google's Knol and the "people-powered" search engine Mahalo as examples of the demand for more accurate information sharing. Quoting: "User-generated sites like Wikipedia, for all the stuff they get right, still find themselves in frequent dust-ups over inaccuracies, while community-posting boards like Craigslist have never been able to keep out scammers and frauds. Beyond performance, a series of miniscandals has called the whole "bring your own content" ethic into question. Last summer researchers in Palo Alto, Calif., uncovered secret elitism at Wikipedia when they found that 1 percent of the reference site's users make more than 50 percent of its edits. Perhaps more notoriously, four years ago a computer glitch revealed that Amazon.com's customer-written book reviews are often written by the book's author or a shill for the publisher. 'The wisdom of the crowds has peaked,' says Calacanis. 'Web 3.0 is taking what we've built in Web 2.0--the wisdom of the crowds--and putting an editorial layer on it of truly talented, compensated people to make the product more trusted and refined.'"

Comments

"1% doing 50% of edits" is not bad

[1 a bee]

The fact that 1% of users do 50% of the edits at wikipedia should not surprise anyone. There are 2 things that a "user" can do at a wiki: read or write. Reading is much easier and faster than writing (duh). So you'll expect a lot more reading to go on, than writing. The "surprise", apparently, is that this writing is not distributed evenly among users who both read and write. In fact, this one data point suggests a power law may be at work here, e.g. 1% of users do 50% of edits, 2% of users do 75% of edits, 4% of users do 87.5% of edits, ... Now what would be so surprising about finding a power law in an organic, social phenomenon like a wiki?

Actually, I find this 1/50 statistic for wikipedia quite impressive. I would have thought--mod me down, I don't care--that there would be even fewer industrious wiki-heads doing even more of the editing. (And hey, don't forget, a lot of this editing *is* simply tedious work that most of us cannot bother with.)

--
Statistics? Sure, just tell me what you want me to prove..

Re:Knol story

[siddesu]

You've them K-things mixed up a bit. Knol is some sort of a project of Google, and is probably unrelated to the cute polar bear Knut. Whoever modded you informative is almost certainly a noted Wikipedian though.

Re:Wtf

[Gromgull]

Actually - the 1% are the users who hang around and correct grammar and punctation mistakes, they clearly make many edits because each edit is only a few chars. The majority of NEW content on the other hand is added by users who may make few other edits. Aaron writes more on this: http://www.aaronsw.com/weblog/whowriteswikipedia

Pareto Principle

[yuna49]

the Pareto Principle (aka, the 80/20 rule)

Actually this is not the Pareto Principle. See the Wiki article on Pareto efficiency for details. Pareto-optimality, as it's referred to in social choice and economic theorizing, concerns making comparisons between two "states of the world." If State A improves the lot of one person and leaves everyone else's situation unchanged, the the "strong" Pareto principle says that State A ought to be preferred by "society." (A weaker form requires only that state B not be chosen.) Another word for the Pareto principle is "unanimity," since Pareto improvements (I'm better off, no one else is worse off) should be acceptable to everyone in a society.

In an abstract free market, transactions among perfectly informed buyers and sellers should reach a Pareto-optimal distribution of prices and quantities. Nevertheless Pareto tells us nothing about distributional issues. As the famous economist Amartya Sen once wrote, "the world can be Pareto optimal and still be perfectly disgusting." One of the most profound findings of social welfare theory is that it's possible to select any Pareto-optimal distribution of prices and quantities, then choose a distribution of incomes that achieves the desired result.

Review of science

[Dire Bonobo]

Even scientific articles are "fact-checked" this way: throw it out there. Typically the reviewers are peers, and quite knowledgeable. This works better than with trade publishers because the reviewers have specific knowledge about that particular field. But does the publisher fact-check themselves? No! I should add that the pay scale for reviewers goes up depending on the relative reliability of the reviewers. Reviewers for scientific reviewers are often paid in the several hundreds range.

You clearly have no idea how reviews of actual science are done.

I have, at this moment, four papers to review for the top conference in my field. I will get paid nothing, and neither will the reviewers of my paper. No reputable conference or journal that I'm familiar with pays reviewers; it's an expected part of being a scientist.

The articles are also by no means "thrown out there"; they're given to a primary reviewer who's a recognized expert in that area, and he or she selects several additional reviewers who he knows to be sufficiently knowledgeable in that area that they'll be able to understand and effectively evaluate the work. This is, effectively, selection of experts by experts, and it's utterly crucial to the peer review process. Simply "throwing it out there" would be a mess.

Unless by "scientific articles" and "review by peers" you're talking about pop-sci magazine articles or something. Calling a piece in Wired a "scientific article" is an enormous stretch, and an actual scientific article goes through a very, very different review process than the one you suggest. One which - not coincidentally - relies heavily on authenticated experts.

Imagining that democracy can replace expertise fits the currently-trendy memes very well, but it's a fantasy. A million monkeys on a million typewriters might crank out Hamlet, but no number of monkeys is going to recognize and select Hamlet, or any other worthwhile piece of writing. Leveraging the work of non-expert crowds is very powerful, but it's not a magic bullet that can solve everything, and it's sheer populist fantasy to imagine that it is.