Could a Reputation System Improve Wikipedia?

Acidus writes, "There is an excellent article in this month's First Monday about using reputation systems to limit the effects of vandalism on public wikis like Wikipedia. It discusses the benefits and weaknesses of various algorithms to judge how 'reliable' a given piece of text or an edit is. From the article: 'I propose that it would be better to provide Wikipedia users with a visual cue that enables them to see what assertions in an article have, in fact, survived the scrutiny of a large number of people, and what assertions are relatively fresh, and may not be as reliable. This would enable Wikipedia users to take more advantage of the power of the collaborative editing process taking place without forcing that process to change.'"

Comments

Easier said than done?

[gasmonso]

I agree that they need to do something, but that is a fantastic challenge. Look at your major encyclopedias, they have a team of several thousand to do fact checking on a paid basis. I'm not saying people wouldn't fact check, but its a great challenge. How would you know that people aren't just saying its legit or not just for fun?

http://religiousfreaks.com/

Re:Easier said than done?

[truthsearch]

With the method recommended in the article the system automatically assumes that a section of text which has not been modified over a series of edits to be more likely to be accurate. It's not that someone denotes a section as fact checked. But if a page has been edited many times, yet one section of it has not been modified, it assumes that unmodified section is more likely correct and colors it appropriately.

Re:Easier said than done?

[beheaderaswp]

Well the biggest problem they are dealing with is standards. In academia there's a criteria for what constitutes a "fact" and a formal process for fact checking. On the "internets", well anyone can submit edits, and claim a factual basis for it.

So a reputation system is pretty useless because special interest groups can mobilize to skew reputation.

Want to have Intelligent Design show more favorably? Ok, get a bunch of like minded people to raise your reputation.

Heck, we even see it on Slashdot when a conservative or liberal viewpoint gets buried in moderation because people of a certain political belief gang up on the opposition.

A lot of what Wikipedia is dealing with is a direct result of the deep divisions on our society. And the fact that unlike World Book Encyclopedia, apparently *everyone* is allowed into the research offices. In a virtual sense of course...

Re:Easier said than done?

[miyako]

There are two big problems I see with this:
The first is that there are a lot of articles and sections of articles in wikipedia that are heavily edited without the facts changing much. This is mostly a good thing, cleaning up grammar, etc. But the if that is used as a basis for how reliable the information is, it could be misleading because the software won't know if the facts have changed, or just their wording
The other problem that I see with this is that it makes it easy for people who "disagree" with facts to make edits to the sections to reduce their rating without just deleting them. It just makes me thing of those people who say "yeah, but evolution is only a theory" to undermine it, I can see them making minor changes to wordings of things to make the facts seem less debatable.
Of course, if someone was doing that, it would be impossible to say if they were doing it because they wanted to supress facts by making them look less reliable, or if there were simply trying to contribute to the quality of an article.

Re:Easier said than done?

[Anonymous Coward]

For instance, something has been wrong with an article from the very begnining http://en.wikipedia.org/w/index.php?title=Gato_cla ss_submarine&diff=2048162&oldid=2048151 Like the Horsepower on the submarine being four times its actual size. Then it gets repeated on every single entry for the class http://en.wikipedia.org/wiki/USS_Tinosa_(SS-283) and then gets repeated as fact on thousands of wiki-replicas http://www.google.com/search?q=four+6500-hp+Diesel +engines to the point where you can only find the truth on a few sites obscured sites if you search for the HP of a Gato class submarine. http://www.bergall.org/mechinfo.html It doenst matter how long it has stood, and especialy not that other sites share the same misinformation, all that matters is if someone knows what they are talking about, or if someone's "source" got their information from wikipedia to begin with.

I can give you the answer without even RTFA

[RLiegh]

That answer is "no". We've seen numerous ratings and karma systems set up on a variety of boards and time and time again they've been defeated by people willing to take the time to game them for whatever reason.

It's typical nerd hubris to believe that you can solve social problems through technological means.
It's been proven time and time again that you can't.

Re:I can give you the answer without even RTFA

[NewWorldDan]

Firstly, the word was "improve", not "solve". I think Wikipedia would improve substantially if it added an editorial supervision system. For example, changes were not posted until approved by a randomly assigned editor. The random part is important. Sure, it's still possible to trash the system, but that takes a lot more effort. And then you need a rating system for editors, and so on and so forth. But the Wikipedia is run on a volunteer basis. There are limits to what it can accomplish without resorting to professional oversight, which would change the very nature of the beast. Ultimately, I think we just have to accept that it is what it is.

Re:I can give you the answer without even RTFA

[Somatic]

I was tempted to mod that down cause it would have been funny. But no one but me would have got it.

Re:I can give you the answer without even RTFA

[catbutt]

Good the Wright brothers didn't say that because lots of attempts were made at flying that failed.

Slashdot's karma system is far from perfect, but at the end of the day it works. Can you game it? I don't really think so, at least not without a LOT of effort, which generally means contributing a lot of good content/ratings so that you can sneak in a very small amount of biased content or ratings.

Whether "ungameable" is possible or not I don't know, but I am quite sure that wikipedia's system could be improved upon massively.

Re:I can give you the answer without even RTFA

[AndyG314]

I think that almost any reputation system woulth threaten the impartial nature of wikipedia.

Re:I can give you the answer without even RTFA

[Anonymous Coward]

Slashdot's karma system works fine for Slashdot, but wouldn't work at all for Wikipedia. Slashdot, despite how much you may like it, is pretty far from neutral. It does the same thing this reputation thing would, represent the opinions of those most active in using it. Wikipedia would be ruined.

Re:I can give you the answer without even RTFA

[geekoid]

Wow, worst analogy on slashdot ever.

The wright brothers weren't trying to overcome social issues, they were overcoming a technical issue.

You can not solve social issues with technology. Social issue aren't always rational.

I have gamed slashdot karma many times.
In days of yor, I could intentional get a 50 karma in a week, and a -49 karma the next.
I can do it today.

Don't believe me? mod me up. ;)

solution in search of a problem

[capoccia]

this is a solution in search of a problem. wikipedia does not have a problem with ordinary vandalism that could result in a reasonable measure of a user's reliability. wikipedia's biggest problem is with unfounded but believable information. in this case, the measure of reliability of a user would be nearly useless because the reliability of their edits is unknown.

How about one for /.?

[ackthpt]

I mean, we could all moderate/evaluate the slashdot editors on their choice of stories and keep stats, like onna baseball card.

CmdrTaco
Dupes: 23
Veiled ads as news: 18
Old news: 17
Allowed Bad Grammar: 2,980
Allowed Bad Spelling: 9,874,376

Yes

[neonprimetime]

Could a Reputation System Improve Wikipedia?

YES - It works on /.

Re:Yes

[drinkypoo]

While the slashdot moderation system is complete crap, can you imagine what this place would be like without anything at all? Me neither.

What about...

[MarcoAtWork]

The site is /.-ed, but this got me thinking: what about having an additional page view that uses color to highlight text age? Oldest text would be black, newest would be something else (red? blue?), intermediate 'ages' in intermediate shades. This would make it quite obvious which parts of the article haven't been modified in a long time.

Re:What about...

[tonyr1988]

Better yet, we could use MySpace color schemes.

• Oldest: black with jet black background
• Little newer: Yellow on orange
• Tad bit newer: Light blue with seventeen exclamation points
• Couple days old: ALL CAPS BABY
• Posted today: Embedded into a crappy Flash movie from www.pimpandrockoutyourwikipediavids.com

Then it will be truly distinguishable.

Not a complete solution

[br00tus]

On Wikipedia, pages relating to quantum mechanics are sometimes vandalized, but 99% of people are on the same side in terms of keeping it accurate. So there are various ways this can be improved.

On the other hand, the pages regarding the fight between Hamas and IDF are as much a battleground as is the area around the Israeli/Lebanese border. I have been involved in Wikipedia for years and have just seen things deteriorate around these types of flame-wars. Wikipedia's leadership is not dealing with it well. Imagine Slashdot setting up a wiki where we had to determine which was better - Debian or Gentoo (or Ubuntu etc.), BSD or Linux, vi or emacs etc.

We are technical people, and there's the old thing about when you have a hammer everything looks like a nail. But I don't think a technical solution will help much in regards to this. I'm not even sure you really can have a neutral view about wars in the Middle East. And even if you could, Wikipedia's "cabal" is nowhere near able to deal with it, and I doubt they ever will be. Personally, I think most of the people in high positions at Wikipedia are jerks, all the flamewars and such seem to have driven most of the nice people off.

Things like Wikipediareview.com convince me that what will ultimately happen is alternatives to Wikipedia will pop up. Wikipedia is a new phenomenom, and it makes sense everyone edits on the same wiki, but why should that be? Why should pro-Hamas and pro-Israel people edit and battle on the same wiki? It makes little sense, and I'm sure in time, just as IRC went from one network to EFnet and Anet, and then split even more, I'm sure we'll see splits with Wikipedia. In the old days, the Encyclopaedia Britannica had one view of history and the Great Soviet Encyclopedia had another, why should the future be any different?

Re:Not a complete solution

[Kesch]

On Wikipedia, pages relating to quantum mechanics are sometimes vandalized.

Actually, every article on quantum mechanics exists in a state between vandalized and not vandalized. By viewing it you colapse the waveform and change it's value. Now, there is a good probability that it will turn out unvandalized, but as you have stated it occasionaly collapses into a vandalized article. After you leave the page, Wikipedia runs complex calculations in it's improbability engine and sends the article back into a quantum state.

P.S. This is presented as per my understanding of quantum mechanics which I learned entirely from Wikipedia. It may be wrong however as my viewing might have caused it to appear in a vandalized state.

P.P.S

Imagine Slashdot setting up a wiki where we had to determine which was better - Debian or Gentoo (or Ubuntu etc.), BSD or Linux, vi or emacs

Debian, Linux, emacs. That wasn't so hard. (Anyone who disagrees is a terrorist.)

Re:The Slashdot moderation system proves....

[joe 155]

I must disagree, the /. system is actually working pretty well. If you say something which is needlessly offensive you will be modded flaimbait, the same would go if you're trying to start a flame war with comments like "GNOME smells of cheese and suX!!11!". If you make some "GNAA!!!!!" type posts, that'll be a troll. If you say something which is completely off topic, it gets modded as such. Both of these things mean that modding becomes pretty much a true/false kind of thing, which meta modding can comfirm. It also stops being seeing it as a default and makes the best shine out...

Possitive modding is a little more shakey with "informative/interesting/insightful" all meaning pretty much the same thing in most people's mind, but that's not too much of a problem.

Group think can cause issues, but in reality there is such a wide range of modders it is often avoided (you can see some pro-MS or anti-Apple comments come through)... although the system isn't perfect I guess group think at least only makes content that most would want to see if they come here.

It is also interesting to note that most people do care about karma and do like to get modded +5, maybe the wiki system would work in a similar way - where people will care.

Vanadlism is not Wikipedia's main problem

[gnetwerker]

In my opinion, vandalism is not the primary Wikipedia problem. Yes, it is embarrassing, but ultimately only a secondary symptom of the central problem: when you have an "encyclopedia that anyone can edit", anyone does edit it. The clear observtion (can't remember who said this first) is that twenty teenage idiots do not collaboratively make an expert. The perhaps more important corollary is that twenty teenage idiots plus one expert are indistinguishable from twenty-one idiots.

Larry Sanger has acutely commented on Wikipedia's anti-elitism and the way they have run experts off the system. Experts don't have the time or energy to debate fundamental points of well-understood scholarships with game-playing trolls. Further, even when they aren't teenagers, Wikipedia has become the home of everyone who wants history and scholarship to read the way they like it rather than representing some academic consensus. As a result we have politicians trying to rewrite their personal biographies (or those of their opponents), partisans on each side of the world's conflicts burnishing their allies and undermining their opponents (Israel/Palestine, Turkey/Armenia, US/everyone else), and devotees of everything from Microsoft Vista to Nintendo to PETA skillfully expunging objective truth from their deifications of the chosen object of worship.

So doling out karma to 100,000 teenage idiots is not going to solve Wikipedia's problem. In order to save Wikipedia, we need to destroy it -- it needs to be edited by more experts and fewer "normal people".

I dunno...

[earthbound kid]

According to some preliminary research by Aaron Swartz about who write Wikipedia, while it's true that most of the editing is done by regulars of the sort who would have karma, most of the original content is added by people with few other contributions to Wikipedia. The regulars just go back and put everything into Wiki format, add tags, make things follow style guides, etc. Since the real work is done by anonymous people who may never come back to the site, it's important to keep the process as open as possible for people who are still new to Wikipedia.

So now we rate people too... Bad move.

[kinglink]

I'm not going to get in the politicing and all. The simple fact is the only response back you'll get from this is how many reverts have been done when you post and those arn't always your fault.

The best parts of Wikipedia is a fast and easy way to edit information, no hassles, no extra effort required. You get out what you put in and that's it. You want to put in the work to be a vandal you're a vandal, but in the end you already know what you're doing. Type in a good sentance but someone replaces it with a better paragraph that's fine.

But instead of working on the core of the experience now we are going to spend time rating each others' facts, rating each other. Basically just killing time. The simple fact is we don't need it, this system is in place in a lot of other places and in effect it basically weeds out the bad apples at the inconvience of all the good users. "You'll have to do 5 discussion posts before you can edit an article" "you have to edit three more articles before you can add an article". This stuff doesn't help or appeal to anyone but "karma whore" types.

If I write a well written page about the new player on the Red Soxes, I should be able to go in to a page that links to it create that page, set up my links and go. I should be able to do this on the first day as well as the fifth year with the same ease. Adding in safe blocks and guards will only hurt wikipedia's overall goals, not help the ideas it promotes. The best thing to do is start handing out serious penalties for vandalism or obvious weasel words.

This doesn't even get into the idea of being able to do fast edits with out logging in, something that's helpful at times.

Yes, but worth the effort

[RareButSeriousSideEf]

Pardon my nerdocratic hubris here, but IMO Wikipedia would be a fantastic petri dish for evolving a robust reputation system, and the result could be useful in a wide variety of applications that facilitate pseudonymous communication & transactions.

In the beginning, I'm sure this would just gather data & have little to no impact on the content. But over time, it could well become increasingly effective at improving content quality as its designers started to identify patterns & meaningful correlations in the collected data.

This isn't so different from SPAM filters that need constant training, or PageRank, or eBay feedback scores, or AVN forum posting rules, etc. One needn't restrict the reputation data to any one data species; you could use a composite of community feedback + usage statistics + genetic algorithms etc., and over time tweak the weight any category of data is given to account for its sample size, its expected margin of error, and its track record in terms of predictive power.

Sure, it's a time consuming undertaking & it'll take patience before we see results, but I don't see the real difficulty being in rigging up the system; I think the real difficulty will be in defining exactly what constitutes a quality article.

Now, take a minute to share a utopian dream with me: Imagine the day when registered Wikipedia users with good reputations will be able to make edits from a Tor connection. :-)