Slashdot Mirror
Competition Produces Vandalism Detection For Wikis
marpot writes "Recently, the 1st International Competition on Wikipedia Vandalism Detection (PDF) finished: 9 groups (5 from the USA, 1 affiliated with Google) tried their best in detecting all vandalism cases from a large-scale evaluation corpus. The winning approach (PDF) detects 20% of all vandalism cases without misclassifying regular edits; moreover, it can be adjusted to detect 95% of the vandalism edits while misclassifying only 30% of all regular edits. Thus, by applying both settings, manual double-checking would only be required on 34% of all edits. Nothing is known, yet, whether the rule-based bots on Wikipedia can compete with this machine learning-based strategy. Anyway, there is still a lot potential for improvements since the top 2 detectors use entirely different detection paradigms: the first analyzes an edit's content, whereas the second (PDF) analyzes an edit's context using WikiTrust."
here we are knocking at the friendly gates of wikipedia and now they want to throw us out.
Is there no hospitality in this world ?
ad: Searching for experienced pillagers and fortress busters. Exp 5 years min. Trebuchet skills a plus. Bring your own broadsword.
I'd like to hear more about MATT IS GAY LOL
If the algorithm can detect 20% with perfection then that must constitute extremely low hanging fruit. That type of vandalism is just annoyance. It is so obvious that the end user readily recognizes it as such and can skip over it or revert the edit.
The real issue is disinformation, which is vastly more subtle. The only defense is fact-checking or seeking out references. If the algorithm is capable of recognizing that kind of vandalism then the developers should have the software writing all the articles in the first place, because it'd have to be pretty spectacular to manage that.
Better known as 318230.
Also, what about more subtle vandalism, the kind that's hard to detect? A few edits that introduce bias in an article, for example. It's this reason that college students everywhere read wikipedia to get a general idea of a topic, and then go elsewhere to places they can actually cite in a paper.
The people who "own" a page with the assistance of powerful insiders and revert any changes to their "pet" pages, even spelling fixes or simple corrections to bad information?
Will edits of *those* insiders, who are ruining wikipedia for the rest of us, be flagged by the algorithm as vandalism?
The elephant^wvandalized article population in Africa^wWikipedia has tripled in the past six months.
That's good. start out with a blank page and run it through the algorithm a couple of thousand times til it becomes true.
Or, you know, just keep applying the first setting that always correctly detects 20% of vandalism on the 80% that's left over, until there's nothing left. Problem solved.
I don't know where that 34% figure comes from for the manual double checking. The test set contains about 60% vandalism and 40% real edits, so I'll assume this represents the rate of vandalism on wikipedia. Now, consider a set of 1000 edits. 600 would be vandalism while 400 would be real edits. The second filter would catch 570 instances of real vandalism along with 120 false positives. Even if you used the first filter to automatically remove the 120 instances of vandalism it finds, you would still be left with a set of 450 instances of vandalism + 120 false positives to check. This means that you would have to sort through about 57% of the original edits in order to identify the 120 false positives.
...because I am tired of my small edits here and there to be classified automatically as vandalism.
I do not believe in karma. "Funny"=-6. Do good and forbid evil. Yours, Oft-Offtopic Flamebaiting Troll.
This comes from personally maintaining some 200+ wikis on Wikidot.com.
There are two kinds of vandals: those in the community of contributors, and those outside it. The first class of vandals cannot easily be detected automatically but when a wiki is actively built, the community will easily and happily fix damage done by these. The second class are usually spammers and come along when the wiki is stale. They are easily detected by the fact that a long static page is suddenly edited by an unknown person. It's very rare to find a real edit happening late after a wiki has solidified. We handle the second type of vandalism trivially by getting email notifications on any edits.
Trick is, wikis (maybe not Wikipedia but then certainly individual pages) don't have random life cycles but go through growth and stasis.
My blog
This implies that the lower-scoring detectors are less valuable in terms of looking for sources of improvement. That's not true, and that wasn't stated in the paper's "Conclusions" section. If the lowest scoring detector finds 5% of the bad data, and it's a different slice from what the other detectors find, then that's quite valuable.
Wikipedia already has programs which detect most of the blatant vandalism. Page blanking and big deletions are caught immediately. Deletions that delete references generate warnings. Incoming text that duplicates other content on the Web is caught. That gets rid of most of the blatant vandalism. It's not a serious problem on Wikipedia.
The current headaches are mostly advertising, fancruft, and pushing of some political point of view. That's hard to deal with using what is, after all, a rather dumb machine learning algorithm that has no model of the content or subject matter.
They already compete to be the first to revert edits they disagree with.
It just characterizes all the edits to the conservapedia to be vandalism.
4chan and Somethingawful have been having Wikipedia vandalizing competitions for years. (Usually, whoever's edit or fake article stays the longest wins.)
As the owner of the first vandalism reverting bot in mainstream use - http://en.wikipedia.org/wiki/User:Tawkerbot2 I guess I have a bit of perspective on the whole problem. Originally the bot was designed / created to auto revert one very specific type of vandalism, a user who would put a picture of spongebob squarepants into pages while blinking them (or squidward or some cartoon character) - that was pretty easy to get. Next we went to stuff like full page blanking, ALL CAP LETTER UPDATES and additions of a tonne of bad words, based on common vandalism trends (ie, if a page had 0 profanity on it and someone added a few words it would be reverted, again, not too many false positives. That basically caught the "dumb kid" type of vandalism, and it was amazing how much lower a percentage it caught of total edits when students went back to school. The only problem, at the time, it was a resource pig. The bot was originally running on a P2 300MHz w/ a grand total of 256MB of RAM and the load got to be so high that we had to move it about 5 times. It's interesting to note that at first, many many people were opposed to the idea of automated vandalism revision, it was almost a contest to revert stuff first - and the bot would win a vast majority of the time. However, as time went on, my inbox started getting rather full whenever I had a power outage, cat knocked the cord out of the box hosting it etc. Community reaction to bots doing the grunt work in vandalism really changed. Anyways, just my 2c on it, and just for the heck of it to prove I'm actually the Tawker on wiki, http://en.wikipedia.org/w/index.php?title=User%3ATawker&action=historysubmit&diff=387163504&oldid=268687392
I suspect that, if the regular edits weren't misclassified by the algorithms then:
a) the reference classification was incorrect itself
b) they were much too convenient samples, compared to the kind of complex changes needed to improve pages in real life.
Detecting spam-like vandalism would seem to be fairly easy.
Far more insidious is politically spun issue-framing masquerading as objective description
of events or topics.
It is truly amazing what you can hide in there by using high-falutin', officious,
grammatically correct language to accomplish your spin. Oft' times you can even fool the
domain experts.
Physicists say that everything is either "spin-UP" or "spin-DOWN". Master spin-doctors
say the same thing.
Where are we going and why are we in a handbasket?
if $edit = EncyclopediaDramatica.match then $vandalism = TRUE;
Why vandalize articles in the first place?
Sure, stupid spammers think replacing an article with a badly spelled advert for ViAGRa is the way to go, and morons think that they gain something from inserting "I'M GAY!!!!!" into an article about someone they dislike, but why just do damage for no other purpose than destroying other people's hard work?
I just don't get it.
These trolls/vandals need to get their asses kicked - hard. Or maybe just have something of theirs broken, just for the fun of it, and see if they find that funny.
"For every complex problem, there is a solution that is simple, neat, and wrong." -- H.L. Mencken (1880-1956) --
...because I am tired of my small edits here and penis to be classified automatically as vagina.