User-Generated Content Vs. Experts

Jay points out a Newsweek piece which suggests that the era of user-generated content is going to change in favor of fact-checking and more rigorous standards. The author points to Google's Knol and the "people-powered" search engine Mahalo as examples of the demand for more accurate information sharing. Quoting: "User-generated sites like Wikipedia, for all the stuff they get right, still find themselves in frequent dust-ups over inaccuracies, while community-posting boards like Craigslist have never been able to keep out scammers and frauds. Beyond performance, a series of miniscandals has called the whole "bring your own content" ethic into question. Last summer researchers in Palo Alto, Calif., uncovered secret elitism at Wikipedia when they found that 1 percent of the reference site's users make more than 50 percent of its edits. Perhaps more notoriously, four years ago a computer glitch revealed that Amazon.com's customer-written book reviews are often written by the book's author or a shill for the publisher. 'The wisdom of the crowds has peaked,' says Calacanis. 'Web 3.0 is taking what we've built in Web 2.0--the wisdom of the crowds--and putting an editorial layer on it of truly talented, compensated people to make the product more trusted and refined.'"

Comments

Re:web 3.0?

[DKlineburg]

Note:
Web 2.0 = http://www.webstandards.org/action/acid2/ "Acid 2 test"
Browsers have gotten to Acid 2 compliant or closer. I would call this the current Web 2.0.

Web 3.0 = http://www.webstandards.org/action/acid3/
The Web Standards Project, seeing that the web is on the Web 2.0 level, is pushing the envelope again for Acid 3, or what I would call Web 3.0. It is interesting to note, that it took 6 months after Acid 2 came out for Safari to get it's browser compliant. ( http://en.wikipedia.org/wiki/Acid2 ) This is applying to the display of content, but I feel this also collates to phases of the web. And yes, I cited Wikipedia for my source, I hope that 1% edited that entry to make sure it is completly correct, although I do know from news articles I can't find that the safari part is.

"1% doing 50% of edits" is not bad

[1 a bee]

The fact that 1% of users do 50% of the edits at wikipedia should not surprise anyone. There are 2 things that a "user" can do at a wiki: read or write. Reading is much easier and faster than writing (duh). So you'll expect a lot more reading to go on, than writing. The "surprise", apparently, is that this writing is not distributed evenly among users who both read and write. In fact, this one data point suggests a power law may be at work here, e.g. 1% of users do 50% of edits, 2% of users do 75% of edits, 4% of users do 87.5% of edits, ... Now what would be so surprising about finding a power law in an organic, social phenomenon like a wiki?

Actually, I find this 1/50 statistic for wikipedia quite impressive. I would have thought--mod me down, I don't care--that there would be even fewer industrious wiki-heads doing even more of the editing. (And hey, don't forget, a lot of this editing *is* simply tedious work that most of us cannot bother with.)

--
Statistics? Sure, just tell me what you want me to prove..

He's right but completely wrong!

[DigitalisAkujin]

He's implying that because of the failure of some web communities in keeping out "drama" that the web will revert to a centralized editorial authority.
The problem is..... it's gonna happen the opposite way.

Wikipedia's next step is to make a functioning and bug free GUI for it's editor to increase the amount of people that edit.

This would increase the size of the governing pool of moderators. You could then do votes on the edits themselves and automate the process. Moderators would get slightly more weight then normal users cause there's a lot less of them. Then get the organizations of the various topics to become moderators on that topic or category. I'm sure if they actually reached out to scientific communities to help with cataloging things it would help on the site with fact checking. And to avoid floods from normal users (the 'wikiality' scenario made prudently clear by Colbert) you could simply look at ratio's of edits. If the ratio is way above other articles you know the normal users are flooding. Moderators aren't likely to flood something with 100,000 votes....

So basically the Internet is gonna see an evolution by more processing and calculating web applications that arise from Solid State Hard-drives, ridiculous amounts of memory, and multi core CPUs. Web applications that work on media are especially thriving from the hard drive market right now. Once SSD comes on the scene with cheaper drives and they replace normal disk hard drives the database intensive area of the code will suddenly have a lot more write capacity as read gets memcached and write has more time but is able to achieve an even faster write. Essentially this would allow anywhere from 5 to 10 times more writes at capacity then compared to today's "reasonable" setup for a 3k server. This would make it cheap for startups with ideas to actually experiment and create competition.

Back to the article though... "Revenge of the Experts". I guess being able to skip over a topic in a general format while sounding like you're explaining the most difficult thing in the world falls into the category of being an "expert". How can you expect people to explain the finer details of what and why in between commercials.

Re:Knol story

[siddesu]

You've them K-things mixed up a bit. Knol is some sort of a project of Google, and is probably unrelated to the cute polar bear Knut. Whoever modded you informative is almost certainly a noted Wikipedian though.

Re:Ya

[Yoozer]

Mod parent up, this is so correct it hurts.

Re:Wtf

[Gromgull]

Actually - the 1% are the users who hang around and correct grammar and punctation mistakes, they clearly make many edits because each edit is only a few chars. The majority of NEW content on the other hand is added by users who may make few other edits. Aaron writes more on this: http://www.aaronsw.com/weblog/whowriteswikipedia

Pareto Principle

[yuna49]

the Pareto Principle (aka, the 80/20 rule)

Actually this is not the Pareto Principle. See the Wiki article on Pareto efficiency for details. Pareto-optimality, as it's referred to in social choice and economic theorizing, concerns making comparisons between two "states of the world." If State A improves the lot of one person and leaves everyone else's situation unchanged, the the "strong" Pareto principle says that State A ought to be preferred by "society." (A weaker form requires only that state B not be chosen.) Another word for the Pareto principle is "unanimity," since Pareto improvements (I'm better off, no one else is worse off) should be acceptable to everyone in a society.

In an abstract free market, transactions among perfectly informed buyers and sellers should reach a Pareto-optimal distribution of prices and quantities. Nevertheless Pareto tells us nothing about distributional issues. As the famous economist Amartya Sen once wrote, "the world can be Pareto optimal and still be perfectly disgusting." One of the most profound findings of social welfare theory is that it's possible to select any Pareto-optimal distribution of prices and quantities, then choose a distribution of incomes that achieves the desired result.

Review of science

[Dire Bonobo]

Even scientific articles are "fact-checked" this way: throw it out there. Typically the reviewers are peers, and quite knowledgeable. This works better than with trade publishers because the reviewers have specific knowledge about that particular field. But does the publisher fact-check themselves? No! I should add that the pay scale for reviewers goes up depending on the relative reliability of the reviewers. Reviewers for scientific reviewers are often paid in the several hundreds range.

You clearly have no idea how reviews of actual science are done.

I have, at this moment, four papers to review for the top conference in my field. I will get paid nothing, and neither will the reviewers of my paper. No reputable conference or journal that I'm familiar with pays reviewers; it's an expected part of being a scientist.

The articles are also by no means "thrown out there"; they're given to a primary reviewer who's a recognized expert in that area, and he or she selects several additional reviewers who he knows to be sufficiently knowledgeable in that area that they'll be able to understand and effectively evaluate the work. This is, effectively, selection of experts by experts, and it's utterly crucial to the peer review process. Simply "throwing it out there" would be a mess.

Unless by "scientific articles" and "review by peers" you're talking about pop-sci magazine articles or something. Calling a piece in Wired a "scientific article" is an enormous stretch, and an actual scientific article goes through a very, very different review process than the one you suggest. One which - not coincidentally - relies heavily on authenticated experts.

Imagining that democracy can replace expertise fits the currently-trendy memes very well, but it's a fantasy. A million monkeys on a million typewriters might crank out Hamlet, but no number of monkeys is going to recognize and select Hamlet, or any other worthwhile piece of writing. Leveraging the work of non-expert crowds is very powerful, but it's not a magic bullet that can solve everything, and it's sheer populist fantasy to imagine that it is.

Re:Ya

[TapeCutter]

"Just because someone has a degree does not elevate him above the rest of the people around him. It tells me just that he learned something. Whether that something is 'right' is up for debate."

Agreed!

"I agree that in a perfect world with perfect peers, a peer review would be the perfect way to discriminate between good and bad research."

It's a strawman to talk of perfection, since science specifically does not recognise that state of affairs (that's the 'up for debate' part). I don't think you are trying to mislead but it does appear that you have made this argument because of a lack of understanding of what science is and how it progresses. OTOH: If you have something better than peer-review, I'm all ears.

What do scientific credentials tell?

Would you prefer a mechanic or a lawyer to fix the cracked head on your car? If someone has a degree they are more likely to be up to date with what is considered 'right' and therefore more usefull to disprove. They could also be expected to understand why peer-review is important to the scientific method. (This of course only applies to science degrees)

"Most of the time, though, it just means that he agrees with the results"

You claim to have spent time at uni (implying scientific credentials) so I shouldn't need to tell you this: the whole point of the review is to (dis)agree with the results by examining the methodology.