Could a Reputation System Improve Wikipedia?

Acidus writes, "There is an excellent article in this month's First Monday about using reputation systems to limit the effects of vandalism on public wikis like Wikipedia. It discusses the benefits and weaknesses of various algorithms to judge how 'reliable' a given piece of text or an edit is. From the article: 'I propose that it would be better to provide Wikipedia users with a visual cue that enables them to see what assertions in an article have, in fact, survived the scrutiny of a large number of people, and what assertions are relatively fresh, and may not be as reliable. This would enable Wikipedia users to take more advantage of the power of the collaborative editing process taking place without forcing that process to change.'"

Comments

Easier said than done?

[gasmonso]

I agree that they need to do something, but that is a fantastic challenge. Look at your major encyclopedias, they have a team of several thousand to do fact checking on a paid basis. I'm not saying people wouldn't fact check, but its a great challenge. How would you know that people aren't just saying its legit or not just for fun?

http://religiousfreaks.com/

Re:Easier said than done?

[truthsearch]

With the method recommended in the article the system automatically assumes that a section of text which has not been modified over a series of edits to be more likely to be accurate. It's not that someone denotes a section as fact checked. But if a page has been edited many times, yet one section of it has not been modified, it assumes that unmodified section is more likely correct and colors it appropriately.

Re:Easier said than done?

[beheaderaswp]

Well the biggest problem they are dealing with is standards. In academia there's a criteria for what constitutes a "fact" and a formal process for fact checking. On the "internets", well anyone can submit edits, and claim a factual basis for it.

So a reputation system is pretty useless because special interest groups can mobilize to skew reputation.

Want to have Intelligent Design show more favorably? Ok, get a bunch of like minded people to raise your reputation.

Heck, we even see it on Slashdot when a conservative or liberal viewpoint gets buried in moderation because people of a certain political belief gang up on the opposition.

A lot of what Wikipedia is dealing with is a direct result of the deep divisions on our society. And the fact that unlike World Book Encyclopedia, apparently *everyone* is allowed into the research offices. In a virtual sense of course...

Re:Easier said than done?

[miyako]

There are two big problems I see with this:
The first is that there are a lot of articles and sections of articles in wikipedia that are heavily edited without the facts changing much. This is mostly a good thing, cleaning up grammar, etc. But the if that is used as a basis for how reliable the information is, it could be misleading because the software won't know if the facts have changed, or just their wording
The other problem that I see with this is that it makes it easy for people who "disagree" with facts to make edits to the sections to reduce their rating without just deleting them. It just makes me thing of those people who say "yeah, but evolution is only a theory" to undermine it, I can see them making minor changes to wordings of things to make the facts seem less debatable.
Of course, if someone was doing that, it would be impossible to say if they were doing it because they wanted to supress facts by making them look less reliable, or if there were simply trying to contribute to the quality of an article.

Re:Easier said than done?

[Anonymous Coward]

For instance, something has been wrong with an article from the very begnining http://en.wikipedia.org/w/index.php?title=Gato_cla ss_submarine&diff=2048162&oldid=2048151 Like the Horsepower on the submarine being four times its actual size. Then it gets repeated on every single entry for the class http://en.wikipedia.org/wiki/USS_Tinosa_(SS-283) and then gets repeated as fact on thousands of wiki-replicas http://www.google.com/search?q=four+6500-hp+Diesel +engines to the point where you can only find the truth on a few sites obscured sites if you search for the HP of a Gato class submarine. http://www.bergall.org/mechinfo.html It doenst matter how long it has stood, and especialy not that other sites share the same misinformation, all that matters is if someone knows what they are talking about, or if someone's "source" got their information from wikipedia to begin with.

I can give you the answer without even RTFA

[RLiegh]

That answer is "no". We've seen numerous ratings and karma systems set up on a variety of boards and time and time again they've been defeated by people willing to take the time to game them for whatever reason.

It's typical nerd hubris to believe that you can solve social problems through technological means.
It's been proven time and time again that you can't.

Re:I can give you the answer without even RTFA

[Anonymous Coward]

A rating system is a social solution to a social problem.

Re:I can give you the answer without even RTFA

[NewWorldDan]

Firstly, the word was "improve", not "solve". I think Wikipedia would improve substantially if it added an editorial supervision system. For example, changes were not posted until approved by a randomly assigned editor. The random part is important. Sure, it's still possible to trash the system, but that takes a lot more effort. And then you need a rating system for editors, and so on and so forth. But the Wikipedia is run on a volunteer basis. There are limits to what it can accomplish without resorting to professional oversight, which would change the very nature of the beast. Ultimately, I think we just have to accept that it is what it is.

Re:I can give you the answer without even RTFA

[catbutt]

Good the Wright brothers didn't say that because lots of attempts were made at flying that failed.

Slashdot's karma system is far from perfect, but at the end of the day it works. Can you game it? I don't really think so, at least not without a LOT of effort, which generally means contributing a lot of good content/ratings so that you can sneak in a very small amount of biased content or ratings.

Whether "ungameable" is possible or not I don't know, but I am quite sure that wikipedia's system could be improved upon massively.

Re:I can give you the answer without even RTFA

[AndyG314]

I think that almost any reputation system woulth threaten the impartial nature of wikipedia.

Re:I can give you the answer without even RTFA

[Yokaze]

> It's typical nerd hubris to believe that you can solve social problems through technological means.

That is not what nerds are trying, it is what society is doing: Trying to solve social problems through software for wetware (laws).

In the case of computer based communities, that laws are codified in programming languages, whereas in RL it is codified in legalese.

> It's been proven time and time again that you can't.

Yes... like flying.

I admit, my first statement seem to be more an argument against the possability of creating a good working online community, as we cannot really claim to have solved our social problems in real life.

However, one has to remember, that an on-line community only has to solve a small subset of the problems, which one has to solve in real life.

IRC, some time ago (one or two years ago) there was a post on Slashdot about a sociology study on the matter on reward and penalty system in communities, which claimed to have isolated some simple rules for a thriving one. I cannot remember having seen it implemented in software.

Re:I can give you the answer without even RTFA

[Anonymous Coward]

Slashdot's karma system works fine for Slashdot, but wouldn't work at all for Wikipedia. Slashdot, despite how much you may like it, is pretty far from neutral. It does the same thing this reputation thing would, represent the opinions of those most active in using it. Wikipedia would be ruined.

Re:I can give you the answer without even RTFA

[catbutt]

How does wikipedia, in its current form, *not* represent the opinions of those who use it (including the opinion that it shoudl be as neutral as possible)?

If people using it today value neutrality, how is that going to change when a karma system is put in place?

Re:I can give you the answer without even RTFA

[Jerf]

But he's not talking about assigning reputation to people, he's talking about assigning reputation to content.

There is a world of difference.

Re:I can give you the answer without even RTFA

[mdwh2]

Slashdot's karma system is far from perfect, but at the end of the day it works.

I disagree. I set my preferences so anons don't have a -1, and people with karma bonuses aren't shown with +1.

If you have something good to say, it should stand on its own merit - and that applies to Wikipedia. We should accept information there because it is verifiable, and not because we trust the random person who wrote it.

Re:I can give you the answer without even RTFA

[mdwh2]

How does karma stop spam? Not moderation (which Wikipedia already has on a far more powerful level than Slashdot, in that edits can be reverted rather than simply moderated down), but the karma bonus system? I see no way to distinguish the trolls from the majority of Slashdotters who are not trolls, but simply do not have a karma bonus.

Re:I can give you the answer without even RTFA

[jZnat]

Then it isn't a wiki anymore. I'm sure a lot of edits these days are clarifications, typo fixes, and other minor modifications. Having to approve them all would be tedious and would grind Wikipedia to a halt.

The Slashdot moderation system proves....

[no_nicks_available]

this wouldn't work.

Re:The Slashdot moderation system proves....

[joe 155]

I must disagree, the /. system is actually working pretty well. If you say something which is needlessly offensive you will be modded flaimbait, the same would go if you're trying to start a flame war with comments like "GNOME smells of cheese and suX!!11!". If you make some "GNAA!!!!!" type posts, that'll be a troll. If you say something which is completely off topic, it gets modded as such. Both of these things mean that modding becomes pretty much a true/false kind of thing, which meta modding can comfirm. It also stops being seeing it as a default and makes the best shine out...

Possitive modding is a little more shakey with "informative/interesting/insightful" all meaning pretty much the same thing in most people's mind, but that's not too much of a problem.

Group think can cause issues, but in reality there is such a wide range of modders it is often avoided (you can see some pro-MS or anti-Apple comments come through)... although the system isn't perfect I guess group think at least only makes content that most would want to see if they come here.

It is also interesting to note that most people do care about karma and do like to get modded +5, maybe the wiki system would work in a similar way - where people will care.

Re:The Slashdot moderation system proves....

[AlphaWolf_HK]

If you say something which is needlessly offensive you will be modded flaimbait, the same would go if you're trying to start a flame war with comments like "GNOME smells of cheese and suX!!11!". If you make some "GNAA!!!!!" type posts, that'll be a troll. If you say something which is completely off topic, it gets modded as such. Both of these things mean that modding becomes pretty much a true/false kind of thing, which meta modding can comfirm. It also stops being seeing it as a default and makes the best shine out...

This is not true. I have seen posts where somebody points out a mere fact about microsoft that gives them a positive light get moderated as a troll just because somebody doesn't like microsoft. It also occurs often that somebody will post comments that have mere facts which support a particular conservative viewpoint, yet just because somebody who may be a liberal doesn't like it they downmoderate it as overrated or as flamebait.

Then the karma system basically says "don't make these kinds of comments again, or else all future comments you make will be ignored." People realize this, thus they are reluctant to speak their mind when they already know that the group (due to its demographic) is going to disagree with them. By having a karma system, you essentially introduce the "fear" element common among e.g. dictatorships. That is, fear of speaking your mind lest you offend somebody who reduces your group reputation. Another fitting way of describing group think would be "group censorship."

The meta moderation system attempts to solve this, but it falls far short for various reasons. First of all the meta moderator could agree with the moderator, even though both of them are biased. Second of all, not everybody wants to spend enough time reading each subject and then each post in order to understand the context fully, but they still want the moderator points anyways so they may just pick answers at random.

This would be terrible for an encyclopedia, since if somebody doesn't like a fact (even though it may be true) they can go along with the groupthink and censor it anyways. An encyclopedia is not, nor should it ever be, a democracy. An encyclopedia should follow the facts as they are, not the facts how people want them to be.

If you still feel that this isn't the case, then explain why it often occurs that people feel the need to post as anonymous coward when posting a view that they already know the group won't like?

Re:The Slashdot moderation system proves....

[mdwh2]

And what is wikipedia today if not "groupthink"? As much as they like to say their is "NPOV" and that this is purely objective, I call BS.

But if what you say is true, then a karma system would only make this worse. Who would get the most karma? Those people who are spending all their time editing on Wikipedia, of course.

rep farming

[Sebastopol]

of course it won't help. people will just grind for rep and then vandalize.

what we need are national ids and biometric logins.

i kid... i kid...

solution in search of a problem

[capoccia]

this is a solution in search of a problem. wikipedia does not have a problem with ordinary vandalism that could result in a reasonable measure of a user's reliability. wikipedia's biggest problem is with unfounded but believable information. in this case, the measure of reliability of a user would be nearly useless because the reliability of their edits is unknown.

Not a complete solution

[br00tus]

On Wikipedia, pages relating to quantum mechanics are sometimes vandalized, but 99% of people are on the same side in terms of keeping it accurate. So there are various ways this can be improved.

On the other hand, the pages regarding the fight between Hamas and IDF are as much a battleground as is the area around the Israeli/Lebanese border. I have been involved in Wikipedia for years and have just seen things deteriorate around these types of flame-wars. Wikipedia's leadership is not dealing with it well. Imagine Slashdot setting up a wiki where we had to determine which was better - Debian or Gentoo (or Ubuntu etc.), BSD or Linux, vi or emacs etc.

We are technical people, and there's the old thing about when you have a hammer everything looks like a nail. But I don't think a technical solution will help much in regards to this. I'm not even sure you really can have a neutral view about wars in the Middle East. And even if you could, Wikipedia's "cabal" is nowhere near able to deal with it, and I doubt they ever will be. Personally, I think most of the people in high positions at Wikipedia are jerks, all the flamewars and such seem to have driven most of the nice people off.

Things like Wikipediareview.com convince me that what will ultimately happen is alternatives to Wikipedia will pop up. Wikipedia is a new phenomenom, and it makes sense everyone edits on the same wiki, but why should that be? Why should pro-Hamas and pro-Israel people edit and battle on the same wiki? It makes little sense, and I'm sure in time, just as IRC went from one network to EFnet and Anet, and then split even more, I'm sure we'll see splits with Wikipedia. In the old days, the Encyclopaedia Britannica had one view of history and the Great Soviet Encyclopedia had another, why should the future be any different?

Vanadlism is not Wikipedia's main problem

[gnetwerker]

In my opinion, vandalism is not the primary Wikipedia problem. Yes, it is embarrassing, but ultimately only a secondary symptom of the central problem: when you have an "encyclopedia that anyone can edit", anyone does edit it. The clear observtion (can't remember who said this first) is that twenty teenage idiots do not collaboratively make an expert. The perhaps more important corollary is that twenty teenage idiots plus one expert are indistinguishable from twenty-one idiots.

Larry Sanger has acutely commented on Wikipedia's anti-elitism and the way they have run experts off the system. Experts don't have the time or energy to debate fundamental points of well-understood scholarships with game-playing trolls. Further, even when they aren't teenagers, Wikipedia has become the home of everyone who wants history and scholarship to read the way they like it rather than representing some academic consensus. As a result we have politicians trying to rewrite their personal biographies (or those of their opponents), partisans on each side of the world's conflicts burnishing their allies and undermining their opponents (Israel/Palestine, Turkey/Armenia, US/everyone else), and devotees of everything from Microsoft Vista to Nintendo to PETA skillfully expunging objective truth from their deifications of the chosen object of worship.

So doling out karma to 100,000 teenage idiots is not going to solve Wikipedia's problem. In order to save Wikipedia, we need to destroy it -- it needs to be edited by more experts and fewer "normal people".

Re:Vanadlism is not Wikipedia's main problem

[Anonymous Coward]

Guess what? Anyone who thinks that Wikipedia some sort of expert-system and that it is being dragged down with anti-elitism has no idea at all what WP has become.

Right now, Wikipedia strives to attribute all statements in its articles to a reliable source. This is something an idiot can do, once they are shown how.

Assuming that debates would somehow be fewer or settled faster and with universal agreement if there were more 'experts' shows no understand of human nature what-so-ever. The best WP can hope for is that everyone agrees that, yes, a prominent NYC newspaper says {Hezbollah statement 1} and a prominent Delhi newspaper says {Hezbollah statement 2}. Note, this results in information that is usable.

If an expert-based WP would work, there would be one. At best, it would grow slowly and many topics would be half-complete. Think of the times when you've looked for an obscure topic on the web, only to find that the best source of information is someone's personal web project that was last updated in July 2003 with "Sorry about that long absence, I'll update more from now on." If you insist on having an experts-only volunteer system, that is what you should expect.

If you're happy with the goal of having all important (or sometimes just -all-) facts related to a topic located on one mostly up-to-date page arranged in a somewhat-organized manner, then join me as a cautious WP fan and contributor.

If you insist on anti-anti-elitism, then I suggest you get all of your information exclusively from the most expensive peer-reviewed journals possible, but make sure that you personally investigate the backgrounds of every author and duplicate experiments when possible, because nobody in the world but you can do it right...right?

If you're not happy with this false dichotomy, then note that I only cared enough about what you have to say to write this post, and not a damn cent more.

Wikipedia worked fine!

[Anonymous Coward]

All these proposals to enhance (read limit) Wikipedia always misses the point. There is a huge amount of evidence, TWO MILLION PLUS ARTICLES, proving that the basic wiki model really works! All changes that try to further limit the openness of Wikipedia need to take that into account.

The reason that proposals that limit Wikipedia seem so attractive is because only the negative sides is making the headlines. It is similar to how most people believe the crime rate is going up, while the statistics show that it is decreasing in most places. The media and its sensationalism is to blame. Instead of carefully measuring vandalism rates and the average time it takes for vandalism to be reverted, we have guys like John Siegenthaler publishing an editorial in Washington Post whining about how Wikipedia contained libel about him for many months. Ofcourse that is very bad for him, but decisions on how Wikipedia should work shouldn't be made solely due to so exceptional screwups. In general, Wikipedia articles are factual and do not contain libel.

It is very unfortunate that Jimmy Wales (founder of Wikipedia) have bought the journalists sensational thinking and are now in the process of implementing more and more protective measures which will make the Wikipedia process more like a normal boring editorial system. Nupedia's fiasko seems to have been forgotten...

And for evidence of how worthless reputation systems are, and how much they raise the barrier to entry, check the modding score of this fine comment.

Re:Yes

[drinkypoo]

While the slashdot moderation system is complete crap, can you imagine what this place would be like without anything at all? Me neither.

So now we rate people too... Bad move.

[kinglink]

I'm not going to get in the politicing and all. The simple fact is the only response back you'll get from this is how many reverts have been done when you post and those arn't always your fault.

The best parts of Wikipedia is a fast and easy way to edit information, no hassles, no extra effort required. You get out what you put in and that's it. You want to put in the work to be a vandal you're a vandal, but in the end you already know what you're doing. Type in a good sentance but someone replaces it with a better paragraph that's fine.

But instead of working on the core of the experience now we are going to spend time rating each others' facts, rating each other. Basically just killing time. The simple fact is we don't need it, this system is in place in a lot of other places and in effect it basically weeds out the bad apples at the inconvience of all the good users. "You'll have to do 5 discussion posts before you can edit an article" "you have to edit three more articles before you can add an article". This stuff doesn't help or appeal to anyone but "karma whore" types.

If I write a well written page about the new player on the Red Soxes, I should be able to go in to a page that links to it create that page, set up my links and go. I should be able to do this on the first day as well as the fifth year with the same ease. Adding in safe blocks and guards will only hurt wikipedia's overall goals, not help the ideas it promotes. The best thing to do is start handing out serious penalties for vandalism or obvious weasel words.

This doesn't even get into the idea of being able to do fast edits with out logging in, something that's helpful at times.

technology is the answer

[klenwell]

One point with Wikipedia that seems to get overlooked -- or at least taken for granted -- is the power and ingenuity of the code that runs it. Technology is part of the solution here. If nothing else, Wikimedia deserves credit for putting together a state-of-the-art wiki machine -- an open source state-of-the-art wiki machine. Some of its features are dauntingly obscure and complex but it falls back quite gracefully to allow even the newest user to function with it effectively. I'd argue Wikipedia has succeeded in large part due to the technology.

That said, there seems to be two alternate proposals here in the summary. (1) A karma system and (2) a new color-coded visual feature. I agree that #1 would be vulnerable to all sorts of gaming schemes -- which isn't to say it wouldn't help, but it'd have to be smart. #2 sounds like it would be a more unequivocal benefit.

Both would be interesting innovations and consistent with the progressive user-friendly code behind Wikimedia.

No, it isn't.

[Anonymous Coward]

The current way Wikipedia does these things-- the use of talk pages, strict documentation of edit and contribution histories, and allowing some people to have more clout based on past contributions-- is a social solution to a social problem.

A reputation system is not a social solution. It is a number. It is a technical solution.

The crucial difference here is that social systems have the ability to be smart. They can understand things like context, or changes in circumstances. If for example someone makes 400 high-quality additions to sports articles on wikipedia, then abruptly shifts gears and starts randomly seeding pages about World War 2 with outright falsehoods, a social system is potentially smart enough to realize something changed there and reject the World War 2 changes quickly based on their content, despite his edit history. But a rep system, a dumb technical solution, would be obligated to help this person along in his changes to World War 2 articles based solely on the numbers gained from his sports edits.

Yes, but worth the effort

[RareButSeriousSideEf]

Pardon my nerdocratic hubris here, but IMO Wikipedia would be a fantastic petri dish for evolving a robust reputation system, and the result could be useful in a wide variety of applications that facilitate pseudonymous communication & transactions.

In the beginning, I'm sure this would just gather data & have little to no impact on the content. But over time, it could well become increasingly effective at improving content quality as its designers started to identify patterns & meaningful correlations in the collected data.

This isn't so different from SPAM filters that need constant training, or PageRank, or eBay feedback scores, or AVN forum posting rules, etc. One needn't restrict the reputation data to any one data species; you could use a composite of community feedback + usage statistics + genetic algorithms etc., and over time tweak the weight any category of data is given to account for its sample size, its expected margin of error, and its track record in terms of predictive power.

Sure, it's a time consuming undertaking & it'll take patience before we see results, but I don't see the real difficulty being in rigging up the system; I think the real difficulty will be in defining exactly what constitutes a quality article.

Now, take a minute to share a utopian dream with me: Imagine the day when registered Wikipedia users with good reputations will be able to make edits from a Tor connection. :-)