Slashdot Mirror
Fake Scientific Paper Detector
moon_monkey writes "Ever wondered whether a scientific paper was actually written by a robot? A new program developed by researchers at Indiana University promises to tell you one way or the other. It was actually developed in response to a prank by MIT researchers who generated a paper from random bits of text and got it accepted for a conference."
I am always wondering what those damn robots are up to!
but I wonder if it can tell if a paper was written by a million monkeys pounding on typewriters?
Taking guns away from the 99% gives the 1% 100% of the power.
so can a robot write a paper and then decide whether the paper was written by a robot (itself)?
Jesus said to his disciples: "If you don't have a sword, sell your cloak and buy one" - Luke 22:36
RESULTS: FAKE
Yep, it works!
Dark Reflection
When will MIT modify this technology to filter all the spam from my mailbox?
I hope the ACLU will ensure that discrimination against metal people will not be allowed to continue.
Has anybody fed Dvorak's latest column to this program? I've often wondered if he actually writes his columns, or just generate verbiage at random.
I've taken a long posting that I wrote on my blog and dropped it into the site. And I am Inauthentic. Now I understand the "Bladerunner Moment" comment in the article. I shall begin to surround myself with oddly colored polaroids and snapshots of theoretically implanted ancestors.
The nice thing is that we've finally settled the argument if machines can be made to drink beer and like it !
This text had been classified as INAUTHENTIC with a 32.2% chance of being authentic text
Bearing in mind that text over 50% chance will be classified as authentic, this add credence to the theory that slashdot comments are generated by monkeys randomly typing on keyboards.
I've abandoned my search for truth; now I'm just looking for some useful delusions.
It seems like it wouldn't be too difficult to modify the MIT program to use this new anti-robot robot to write papers that this anti-robot robot would not be able to detect. Ideally, this would be done with a learning algorithm (so that it could easily be extended to other anti-robot robot programs), but reverse-engineering the anti-robot robot (by humans) should also provide a solution.
Now that Indiana U has thrown down the gauntlet, I wouldn't be surprised if MIT responds. Hopefully it will result in an even better paper-writing robot. Ideally, it will lead to dissertation-writing robots. :)
Ben Hocking
Need a professional organizer?
Apperantly I'm on average 49% artificial, based on school papers I wrote. I dub thee program: a failure.
I suspect that it is looking for the conventional thinking with conventional word structure. As such, it is NOT a good idea i
excitingthingstodo.blogspot.com
Do robots make typos? Do they make the same typos each time, or different ones?
Based on the slashdot articles that get posted. I would say YES.
Actually it's pretty easy to add random convincing misspellings to text, you could use a database from something like usenet, and a spell checker to map misspelled words to their real counterparts, and then have a straightforward algorithm for replacing some set of words with misspellings, and you could tune that for consistency. It would be easier than many other aspects of faking papers.
((lambda (x) (x x)) (lambda (x) (x x))) http://www.endpointcomputing.com a scientific approach to custom computing.
Maybe slashdot can start running it on their links for "cold fusion in 1 year!".......
E-mail spambots have been making typos for years.
Hey, if you don't like 1-ply you can always fold it in half.
And if you don't like 2-ply, you can separate the sheets. Keep in mind that this works best before you wipe.
There are 0x40000000 types of people: those who understand 32-bit IEEE 754 floating point, and those who don't.
So I go there, and I start shoving it text from my hard drive. I try:
A) Text of an article (Philosophy) I (native English speaker) wrote in Italian: 98.5 Authentic.
B) Text of an article I wrote in English (History): 87.8
C) Text of an article (History) written in French by a native French speaker and translated into English: 93.2
D) Critical edition of a 14th-century Latin text (Theology): 97.7 Authentic.
E) Documentation to a Field Artillery Simulation: 95.3
F) A completely bogus narrative for a monastic order that doesn't exist, written in a style that mimics A)-C): 16.8% Inauthentic
So in this case, we have a human written document that has superficial meaning, but is written as a "fake scientific paper", and registering as such.
And yes, I did read the "purpose" of the page; I know it's not supposed to detect it.
And yet it does, decisively.
Duplicating the first half of the sample fake paper after the end of the footnotes makes it go from inauthentic (17%) all the way up to 91% authentic. It seems to be looking for long-range n-gram repetition, but it doesn't have a ceiling on frequency or length or the repeated text.
It shouldn't be hard to compare the distribution of n-gram recurrence rates (or distances between recurrences) to the observed distribution for actual papers. Something like a KL divergence would capture deviations in either direction.
This raises a question... how do Wikipedia articles fare? --I'd guess that they should be at least *somewhat* scientific....
I just finished writing a scientific paper for publication. Apparently, this filter is very reliant on using long-term pattern recognition. When I fed this application my introduction only, it told me my work was INAUTHENTIC with a 35% chance of authenticity. When I fed it the first two sections, it said it was AUTHENTIC with a 66% chance of authenticity. And finally, when I fed it the entire paper, it said it was AUTHENTIC at the 87% level.
So apparently, all you need to do to beat this filter is insert the same buzzwords and phrases at many different points in a long article, and you should be fine.
I'm amazed too! It works!
Read the paper listed in the menu of the website. The system essentially compresses the text with different window sizes, and then looks at the compression factors. In other words, it is only looking for repetition of strings. This is absurdly easy to fool, and the MIT generator could be easily fixed to pass this filter. For example, try entering a random text once (your post, for example). Note that it fails. Then append a few copies of the same text, and run that through. Your post, when run once, is too short. When run with two copies, it is rejected as 41.2%. When run with three, it passes with 93%. There is a window of repetition level required in order to pass - papers that do not repeat enough are classified as fake, as well as papers that repeat too much (try entering twenty copies of your post).
It should be relatively simple to make a random paper generator that always passes this test with a higher probability than human-written papers.
I, for one, peruse the blogosphere. On my Powerbook, wearing a black turtle neck and beret. Stroking my goatee thoughtfully. Sipping a latté in a café
If I could just find a way to recharge my PowerBook from your hatred, I could stop carrying this ugly power adaptor.
echo -e 'global _start\n _start:\n mov eax, 2\n int 80h\n jmp _start' > a.asm; nasm a.asm -f elf; ld a.o -o a;
Hmmm, it's an interesting idea, but it seems to give a lot of false positives. (So naturally, it will detect fake papers, if it thinks every paper is fake.)
First thing I tried was some pages on computational oncology website, in particular, my cancer primer, which I wrote in not a short time. Everything I fed was determined to be inauthentic. Perhaps I just write like a robot. :-) I figured that perhaps the detector was more primed for real papers, so I figured it wasn't too big of a deal.
So, next I tried my most recent research paper, and it, too, was determined to be inauthentic, and in fact with less authenticity than my website. So much for the theory of being primed for scientific papers only. This thing is starting to look pretty bogus to me ... but an interesting idea, nonetheless. -- Paul
OpenSource.MathCancer.org: open source comp bio
Some variant on this thing might be useful as a new article filter in Wikipedia. We need more automation over there to stem the flow of incoming dreck.
Sir I regret to inform you that you are a ruffian. I for one sit not in a place so vile and common as a 'café', examining the flawed writings of others, but in a temple constructed purely out of my supercilious transcendent superiority. I consume nothing so plebeian as 'The Internet' but rather a rasterized, marked-up and projected form of my own rigourous, peerless stream of consciousness (with blue aqueous scroll-bars). I have no need for facial hair or indeed any of your corporeal trappings and hence know not the joy of stroking a 'goatee'.
And now I must mod you as Troll, for surely you must know that the PowerBook, created on the Seventh day, is immaculate in it's design and conception and therefore the only possibility is that you seek to trifle with the emotions of our brethren, in crudely ascribing to Our Power Adapters the property of ugliness. If you were truly one of Us you would know that Steve created all in his own beautiful image.
btw you haven't got a couple of rubber feet for an ibook going spare have you mate?