Slashdot Mirror
Algorithm Rates Trustworthiness of Wikipedia Pages
paleshadows writes "Researchers at UCSC developed a tool that measures the trustworthiness of each Wikipedia page. Roughly speaking, the algorithm analyzes the entire 7-year user-editing-history and utilizes the longevity of the content to learn which contributors are the most reliable: If your contribution lasts, you gain 'reputation,' whereas if it's edited out, your reputation falls. The trustworthiness of a newly inserted text is a function of the reputation of all its authors, a heuristic that turned out to be successful in identifying poor content. The interested reader can take a look at this demonstration (random page with white/orange background marking trusted/untrusted text, respectively; note "random page" link at the left for more demo pages), this
presentation (pdf), and this paper (pdf)."
Someone should make a wikipedia entry for this algorithm to see how trustworthy it is.
More
>If your contribution lasts, you gain 'reputation,' whereas if it's edited out, your reputation fails
...
And the editor wars start
Washington bullets will simply be known as the "Bulle
It'd be nice if it could be generalised to other sites...
Deleted
Every paper touting automatic adjustments for gaming the system becomes obsolete the moment it is published.
(Godwin didn't publish this, but I might get around to editing his Wikipedia entry to say that he did).
I've been noticing some of the edit histories for articles that are 5 years old on Wikipedia stop well before 5 years ago. Were some of the edit histories been lost or deliberately truncated?
you do something wrong, it will mod you "-1, Troll"
So, if there is a myth that a lot of people believe is true, then it will stay up there as it is not challenged. So, it still gets reputation, and therefore more credibility, making it more likely that the myth will be perpetrated.
Also, if someone hasn't noticed something that is wrong on an esoteric entry, it will also be given credibility, and once again be more likely to be considered to be fact.
While you could add voting to the algorithm to have people vote on whether it is true, that still gets destroyed by someone who just votes because they think it's true, not because they have verified it.
Either way, it potentially gives additional credibility to something that may be very wrong.
Seems to work, the entire page turned orange.
+0 Meh
binutils-2.18.tar.{gz|bz2} is ya GPLv3!
They should just call it wiki-karma.
It appears they include #REDIRECT pages; the very first page the random link took me to was Cheliceriformes, with the #REDIRECT line in orange. Seems an easy way to gain trust, once a redirect is created it is hardly ever changed.
Does it take into account magnitude of error corrections? If major portions of someone's articles are being rewritten, that's a good reason to de-rep them. If someone makes a bunch of minor spelling or trivial errors, then that's not necessarily a reason to do so.
And, of course, there is the potential for abuse. If the software could intelligently track reversions and somehow ascribe to those events a neutral sort of rep, that would probably help the system out.
As it stands, they're essentially trying to objectively judge "correctness" of facts without knowing the actual facts to check. That's somewhat like polling a college class for answers and assigning grades based on how many other people DON'T say that they disagree with a certain person in any way.
the relative controversy of the item being edited.
If I edit a history page of a small rural village near where I live, I can guarantee that it will remain unaltered. None of the five people who have any knowledge or interest in this subject have a computer.
If I edit an item on Microsoft attitude to standards, or the US occupation of Iraq, I'm going to be flamed the minute the page is saved, unless I say something so banal that noone can find anything interesting in it.
But my Microsoft page might be accurate, and my village history a tissue of lies....
Sounds like a worthy start to the process of introducing more trustworthyness into Wikipedia entries, but this maybe needs tuning for content type too.
Afterall just because someone is a reliable expert at editing the wikipedia entries on Professional Wrestling or Superheroes doesn't necessarily mean we should trust their edits on, for instance, the sensitive issues of Tibetan sovereignty.
erroneous: look me up in a dictionary
It's practically an automatic with people so codifying it for machine should be no surprise.
http://en.wikipedia.org/wiki/Appeal_to_authority
I realize that an encyclopedia by definition will always emphasize the established majority opinion about any given subject. But it seems that this tool might strengthen majority opinions beyond what is reasonable. If you happen to edit an article by adding valid but unpopular dissenting points of view, and the other contributors are sufficiently boneheaded, you lose karma (or whatever the tool calls it) for no good reason. This might then easily develop a life of its own, and you are screwed.
"When I first heard Daydream Nation it quite frankly scared the living shit out of me." -- Matthew Stearns
What if good truthful content is being removed by BAD (malicious) editors? How does the algorithm account for that?
We fairly urgently need to assess the reliability of the various sources on the net becaues the s/n ratio is getting so low that the net becomes harder and harder to use for real work.
:)
more addictive than crack, do not click
Although this method will certainly help filter pranks and cranks, it won't help if the "consensus" among wikipedia authors is wrong. If a true expert edits a page, but the masses don't agree with the edit, they will undo the expert's addition and give the expert a low reputation. Thus, the trust rating becomes a tool for maintaining erroneous, but popular ideas.
That said, I can't help but believe that this tool is a net positive because it makes points of debate more visible. One could even argue that it literally highlights the frontiers of human knowledge. That is, high-trust (white) text is well known material and highlighted (orange) text represents contentious or uncertain conclusions.
Two wrongs don't make a right, but three lefts do.
when things can be quantified and measurable. I've always wondered about the algorithm of a brand's worth. What is the logo's value, in relation to the slogan, and the consumer experience?
For instance, Google has a strong brand, despite their hideous logo and "Don't be evil" slogan, because the consumer experience is so good. Coca-Cola, on the other hand, score big with their logo's distinctive cursive script, despite ongoing critisms of its health effects and numerous allegations of wrongdoing by the company. And their product just isn't that good.
Man, I would loves me an algorithm for that.
SEO Copywriter. Just Say ON
...but call me when there's a tool to measure the truthiness of an article.
How did they pass up the chance to name this algorithm "Truthiness"?
Athletic Scholarships to universities make as much sense as academic scholarships to sports teams.
I might give a damn if Wikipedia editors had any actual interest in keeping articles truthful.
No algorithm, except maybe personally checking every single article yourself, will ever be perfect. I suspect that the stuff you talk about will be very rare exceptions, not the rule. In fact, one of the reasons that it is so rare is because people who know what the actual truth of a matter is can post it, cite it, and show it for all to see that some common misconception is, in fact, a misconception. This is much better than, say, a dead tree encyclopedia where, if something incorrect gets printed, it will likely stay that way forever in almost every copy that's out there. (And, incidentally, no such algorithm can exist, since dead tree encyclopedias generally don't include citations and/or articles' editing histories.)
The goal wasn't to create a 100% perfect algorithm, it was to create an algorithm that provides a relatively accurate model and that works in the vast majority of cases. I don't see any reason this shouldn't fit the bill just fine.
http://www.google.com/trends?q=slashdot%2C+digg&ct ab=0
pwned!
unless it is consistent with what I already know to be true or have had time to verify against other sources.
too many zealots rule certain categories and unfortunately too many of the same are the very powers that be.
* Winners compare their achievements to their goals, losers compare theirs to that of others.
Groupthink.
Knowledge is power. Knowledge shared is power lost.
I have seen much of this in science; flawed conclusions get credibility by an accelerating number of citations. This kind of software will have exactly the same effect :-(
Also, this software means that wikipedia is moving towards the kind of "elite control" over knowledge that wikipedia was supposed to oppose (or at least supposed to be an alternative to). This alternative now slowly merges with the very problem it was supposed to evade.
First, it will be too unwieldy for wide-spread use, and will fade into history. Second, it will be useful and will be incorporated into typical Wikipedia reference activity, and then the bitches of the world will start locking down articles in order to game the program.
What we really need is some sort of algorithm that compares new information to that which is already stored. It then could test hypotheses to gain further understanding. Unfortunately a machine with enough processing power to run this "critical thinking and understanding" algorithm would be impossible to build with today's technology. We would need a new type of processor that has maybe billions of "organic neurons", it would need to be equipped with highly sophisticated sensors, a method of self transportation, self-healing and even it's own energy production system which could harvest energy indirectly from the Sun. We can only dream of such technology being available to everyone.
No trees were harmed in the posting of this message. However, a great number of electrons were terribly inconvenienced.
Wikipedia (Jim Wales) once employed and expert who claimed to have a string of degrees but did not - the editor would say i have a degree in this and the victim would agree that the editor knew better.
But the expert got found out
Wikipedia and the algorithm needs to take account of persons (or organisations) who link to it get better seo - I state that 'the times of india' being good here for trashing content and putting in there own links to there site when in the 'news'.
The wiki item in question is now useless and tells you less than before the criminal was famous. Logs, examples and other pertinent information was trashed by these media staff.
Ideally one would re-edit the item, but why should i, and why should the seo for 'the times of india' be deemed better than my edit which was far more indepth.
Wiki is not perfect, but news organisations serve adds. When Jim Wales promotes the times of india i do hope he get paid something by them.
This is not sour grapes - but editors, idiots, and seo ops might make there own link farm ghetto - 'the times of india',the bbc, cnn etc. Journalists can write (say english) it does not mean they know much about the topic they discuss in english.
While the tool has good intentions my use of the wiki as a source has stopped and no i do not purchase the 'times of india' either.
The senario i discuss happens with the newly famous. That is when the profession seo'ers at the 'the times of india' rush in and mass delete.
Not my problem, but i do hope Jim gets paid by the 'times of india'.
"trustworthiness" doesn't enter into whether something gets edited out, for precisely the same reason a need for this is perceived at all: it can be edited by anyone!
We've secretly replaced Slashdot with new Folgers Crystals - let's see if it notices.
This definitely is a step in the right direction. A few years later, reputation calculation could become more complex based on profiles of the individual on the entire net. Who knows!
.. and it strikes me that every way in which a system like that could be gamed are quite horrible and damaging.
This is nothing more than a karma system where loud masses rule viewpoints.
The shark's in the water, and Wikipedia is up on their skis....
I've seen some very large, profanity-laden edits. Many of these more than double the size of the article. These are not the majority of vandalisms, but they are a significant fraction.
Ben Hocking
Need a professional organizer?
what if I am very trustworthy but can't spell or use proper grammar?
..about "trustworthy-ness" comes from articles and opinions like this.
"Bully for him, and Wikipedia nay-sayers be danged. Some of that massively democratic participatory media can get pretty funny, and teach us more about human nature than dull, non-participatory text."
Yes, those boring things like facts, who needs them, let alone an encyclopedia? What I really want is to be entertained. Let's invest in this huge experiment on "human nature" at the expense of truth and knowledge. Hell, it's not like anyone over 20 years old or who hasn't lived on a desert island doesn't already know about "human nature".
For a site that prides itself on being up their with announcing new things, this is really pretty much old news.
XML is like violence. If it doesn't solve the problem, use more.
That algorithm is a model that does not match real world data. It might be useful to measure who has protection from the bureaucracy, but it won't and can't decipher how true something is simply by how many times and at what frequency people scribble over it. This algorithm is psuedo-scientific, by assuming a premise without investigating the veracity of said premise, and then running away with it as if it were a proven one.
Slashdot: Playing Favorites Since 1997
One big problem with Wikipedia has been that editor status, and promotion to "adminship", is based on edit counts, the number of times someone has changed something. The editors with huge edit counts don't generally write much; it takes too long. Slashdot karma is a more useful metric than edit counts, but Wikipedia doesn't have anything like karma.
I'd suggested on Wikipedia that we needed a metric for editors like "amount of new text that lasted at least 90 days without deletion". This UCSC thing is a similar metric.
now the algorithm is publicly known, anyone can make pages that abuse it somehow ^^ and then they can update it to counter those abuses etc... let the war begin!
How trustworthy is this algorithm
So if the Wikipedia editors, US government, Fox news, spend all their time keeping pages they way they want it by editing user contributions, they automatically become the reliable source?
They might be somewhat correlated, on a statistical basis, over
many cases, but there are many individual cases and times
when the currently popular view is wrong and the lone
wolf opinions are later proven to have been correct.
This algorithm would seem to be more of a popularity contest
than a truth finder. I think we have to be very wary of
the truth by mass agreement theory.
Hint: Remember the "weapons of mass delusion" ?
I bet someone commenting that the US government is lying
through their teeth about it would have been re-edited
pretty quick.
Where are we going and why are we in a handbasket?
no matter how loosely related the subject is to Anime.
many have good information about references in the "Animaiacs" cartoon too, which will come in handy when writing your term paper.
Trustworthiness? NO.
Compliance with the dominant group view? YES.
We're well on the road to a society full of truthiness when people refer to passing groupthink tests as a measure of "trustworthiness".
... is even worse ...
For some reason, the phrase "tyranny of the majority" causes people to suspend disbelief. As is evidenced by your post being marked insightful...
Is ordinary admin (non-oversight) deletion used frequently compared to oversight deletion? I've seen articles where the entire edit history before a certain date containing several years' worth of edits was erased.
What could be causing some edit histories to get out of chronological order as mentioned in this post.
Why oil price increase equals economic trouble (Score: Interesti
The algorithms looks very similar to the Google's Pagerank. Take edition time as inverse of links to/from, and the whole concept looks very similar. The question is, PageRank was terribly biased once people started to automate cross linking, will this algorithm performs better against biased editors?
What's in a sig?
Your comment got me to thinkin'. (and on a Friday! Damn you!)
The big thing in academic research is peer review, and what is Wikipedia but the extension of peer review to the larger community? I'm certainly not a fanboi and don't use Wikipedia as a source for anything work related, but I'm not too quick to add "never" to the end of that statement.
When I go to a peer-review journal, either as a source for research or an outlet of publication, I am looking for two things. First I hope the community is right in trusting the journal. This is like the 'many eyes' program for weeding out bugs in open source software. And it's not just the lack of the community rising to refute the publication, but also the number of citations. Trust is demonstrated when members of the community continue to say, this is a source I trust and _use_.
Second I hope that trust transfers to me (or my research). The foo community trusts the Journal of Foo Letters, so when I reference that source, I least get the benefit of the doubt that I am starting from sound principles. Likewise, when my work is published in JFL, even those who disagree with my conclusions must either confer upon me a certain extent of respect or question the standing of JFL in the community.
The difference is, JFL has gained the trust of the foo community by being run members of that community. Its reviewers are foo experts in good standing who each have a history of original work. I go to Wikipedia for my foo research, and I don't get that level of trust.
However we've already seen a movement towards openness in the peer-review journal community. What's prevents Wikipedia setting up a foo area moderated by a panel of foo experts known to the foo community? If such a thing existed, and could still be called Wikipedia, I see no reason it could not be used as a source for academic research.
Does anyone else find this list hilarious? http://hemlock.knams.wikimedia.org/~leon/stats/wik icharts/index.php?wiki=enwiki&ns=articles&limit=10 0&month=08%2F2007&mode=view
WikiCharts -- Top 100 -- 08/2007
Views per day Percent Title
389 659 ± 1% 3.7114% 1. Main Page
17 773 ± 3% 0.1693% 2. Harry Potter and the Deathly Hallows
11 368 ± 4% 0.1083% 3. Wiki
10 995 ± 4% 0.1047% 4. Harry Potter
9 649 ± 4% 0.0919% 5. Transformers (film)
5 286 ± 6% 0.0504% 6. Naruto
5 173 ± 6% 0.0493% 7. Wikipedia
4 427 ± 6% 0.0422% 8. Deaths in 2007
4 119 ± 6% 0.0392% 9. United States
3 827 ± 7% 0.0365% 10. Harry Potter and the Order of the Phoenix (film)
3 714 ± 7% 0.0354% 11. Sex
3 616 ± 7% 0.0344% 12. List of sex positions
3 584 ± 7% 0.0341% 13. Hypertext Transfer Protocol
3 535 ± 7% 0.0337% 14. The Simpsons
3 519 ± 7% 0.0335% 15. YouTube
3 486 ± 7% 0.0332% 16. Bleach (manga)
3 422 ± 7% 0.0326% 17. Guitar Hero III: Legends of Rock
3 227 ± 7% 0.0307% 18. List of characters in the Harry Potter books
2 838 ± 8% 0.0270% 19. The Simpsons Movie
2 789 ± 8% 0.0266% 20. List of Konoha ninja
2 789 ± 8% 0.0266% 21. List of Akatsuki members
2 773 ± 8% 0.0264% 22. Optimus Prime
2 692 ± 8% 0.0256% 23. Harry Potter and the Half-Blood Prince
2 676 ± 8% 0.0255% 24. Seven Wonders of the World
2 514 ± 8% 0.0239% 25. Chris Benoit
2 449 ± 8% 0.0233% 26. Harry Potter (character)
2 416 ± 8% 0.0230% 27. 50 Cent
2 368 ± 8% 0.0226% 28. Megatron ...
so all wiki users are nerdy harry potter fans interested in sex? ha!
"sometimes he felt that his whole life was a dream, and he wondered whose it was and whether they were enjoying it."
I could submit nonsense on a variety of obscure topics, with low odds that anybody will find and correct them, thereby building up a great reputation. I wonder if their system accounts for that.
This is starting to sound like Karma for wikipedia.
If moderation could change anything, it would be illegal.
On reviewing the demo, it would seem that the untrusted i.e. frequently
changed sections are essentially the sections that people care about
or (in our present society) have more knowledge about or insight into,
so people want to tweak those sections.
So ironically, the algorithm will flag as untrustworthy the most relevant
sections of articles. The "don't know - don't care" parts will be virgin
white and pure.
Where are we going and why are we in a handbasket?
We will never make a 32-bit operating system, but I'll always love IBM. -Gates
What, sir, would you make a ship sail against the wind and currents by lighting a bonfire under her deck? I pray you, excuse me, I have not the time to listen to such nonsense. - Napoleon
I watched his countenance closely, to see if he was not deranged ... and I was assured by other senators after he left the room that they had no confidence in it. - U.S. Senator Smith of Indiana, after witnessing a demonstration of Samuel Morses's telegraph
Well-informed people know it is impossible to transmit the voice over wires and that were it possible to do so, the thing would be of no practical value. Boston Post
Louis Pasteur's theory of germs is ridiculous fiction.Pierre Pachet, professor of physiology at Toulouse
Radio has no future. - Lord Kelvin
The ordinary "horseless carriage" is at present a luxury for the wealthy; and although its price will probably fall in the future, it will never, of course, come into as common use as the bicycle. Literary Digest
I have traveled the length and breadth of this country and talked with the best people, and I can assure you that data processing is a fad that won't last out the year. The editor in charge of business books for Prentice Hall.
We don't like their sound. We don't think they will do anything in their market. Guitar groups are on their way out. Decca Recording Co., declining to sign the Beatles
"There is no reason anyone would want a computer in their home." Ken Olson
There are several other fun ones that I am sure people will post but the idea of saying never is a hard prophecy to live up to. It is also very hard to verify quotes without use of a wiki or a network. But I will include this excerpt to refute your "claims to be" remark Wikipedia's founder, Jimmy Wales, says he wants to get the message out to college students that they shouldn't use it for class projects or serious research. -http://chronicle.com/wiredcampus/article/1328/wik ipedia-founder-discourages-academic-use-of-his-cre ation
CS: It is all sink or swim...oh and did I mention there are sharks in that water?
Nothing but personal direct observation can be an authoritative source of anything, and even that can fail.
This algorithm is measuring compliance with the Wikipedia dispute processing norms -- not "trustworthiness". A better measure of "trustworthiness" of a passage is its consistency with the rest of the body of human knowledge -- which is most strictly measured by the degree to which it is not a special case within a compressed representation of that knowledge. This is the basis of the Hutter Prize for Lossless Compression of Human Knowledge. The Hutter Prize is currently using a 100M sample from Wikipedia as its corpus.
Seastead this.
> unless it is consistent with what I already know to be true
:)
:)
Absolutely. I keep trying to replace all their lies about quantum mechanics with my truth about the Electro-Flux Aether and Spiritual Gravitation, and I keep getting reverted.
> or have had time to verify against other sources
Ah, so you do understand how Wikipedia should be used. Good on yer, mate.
> too many zealots rule certain categories
Yeah, like those bastards who keep trying to insist that the Holocaust actually happened, that evolution is a scientific fact, and that the Earth goes around the Sun. I mean, have you ever heard anything so preposterous? I'd rather get my information from more reliable sources. Like Fox News and the New York Times. And Slashdot.
What I want to know is if it is smart enough to distinguish edits that correct spelling and grammar mistakes from those that change content.
In particular I'm worried that the system will undervalue the information from people whose edits are frequently cleaned up by others even if that content is left unchanged.
If you liked this thought maybe you would find my blog nice too:
#!/usr/bin/perl
open(WIKIPEDIA, "<", "wikipedia.dump") or die "Unable to find garbage file - $!\n";
while (<WIKIPEDIA>) {
++$crap;
++$nonsense;
++$spam;
}
print "Not very trustworthy.\n";
Wouldn't one find an entry, for example, about the early history of the WWW by the genuine Sir Tim Berners-Lee to be considerably more trustworthy than one by signed by some anonymous WikiWonderBoy?
I don't think the algorithm takes that into account.
It could also be useful for the trust worthiness of code from various authors! - Poorly written/designed code will probably have more faults and require more edits.