Fake Scientific Paper Detector

moon_monkey writes "Ever wondered whether a scientific paper was actually written by a robot? A new program developed by researchers at Indiana University promises to tell you one way or the other. It was actually developed in response to a prank by MIT researchers who generated a paper from random bits of text and got it accepted for a conference."

Comments

Yes!

[stupidfoo]

I am always wondering what those damn robots are up to!

That's good and all

[XxtraLarGe]

but I wonder if it can tell if a paper was written by a million monkeys pounding on typewriters?

Re:That's good and all

[denverradiosucks]

Obligatory Simpson's Quote

Monkey's typing on a typewriter as Mr. Burn's is working on the next great american novel:

Burns: This is a thousand monkeys working at a thousand typewriters. Soon they'll have written the greatest novel known to man.
(monkey smoking cigar typing on a typewriter)
Burns: Lets see. It was the best of times, it was the BLURST of times! You stupid monkey! (Smacks monkey upside his head)

Re:That's good and all

[visgoth]

Oh, I'm sure the work of monkeys is quite easily identifiable.

Turing test?

[Nesetril]

so can a robot write a paper and then decide whether the paper was written by a robot (itself)?

Discrimination

[hsmith]

I hope the ACLU will ensure that discrimination against metal people will not be allowed to continue.

An interesting experiment

[fm6]

Has anybody fed Dvorak's latest column to this program? I've often wondered if he actually writes his columns, or just generate verbiage at random.

Re:An interesting experiment

[irregular_hero]

"This text had been classified as
INAUTHENTIC
with a 24.9% chance of being authentic text"

No kidding.

Self defeating?

[benhocking]

It seems like it wouldn't be too difficult to modify the MIT program to use this new anti-robot robot to write papers that this anti-robot robot would not be able to detect. Ideally, this would be done with a learning algorithm (so that it could easily be extended to other anti-robot robot programs), but reverse-engineering the anti-robot robot (by humans) should also provide a solution.

Now that Indiana U has thrown down the gauntlet, I wouldn't be surprised if MIT responds. Hopefully it will result in an even better paper-writing robot. Ideally, it will lead to dissertation-writing robots. :)

Re:Self defeating?

[cp.tar]

I recently had to check out an essay-grading robot for my Introduction to Natural Language Processing class.

I'd fed it the introduction of a randomly generated essay. It got a 4/5 on all counts.

I figure, if teachers are going to use robots to grade essays, we should use robots to create them in the first place.

Re:Self defeating?

[mctk]

Eventually my students won't have to write papers and I won't have to grade them! Think of the potential application of this technology towards education!

Only works for scientific papers

[gurps_npc]

If you try to use it on any human written NON scientific paper, such as Lincoln's gettyburg address, it almost always considers it false.

I suspect that it is looking for the conventional thinking with conventional word structure. As such, it is NOT a good idea i

Re:Only works for scientific papers

[nasor]

No, it doesn't even seem to work on scientific papers. I submitted four papers from the latest issue of Inorganic Chemistry and it thought 2 out of 4 were false:

Inauthentic: Assembly of a Heterobinuclear 2-D Network: A Rare Example of Endo- and Exocyclic Coordination of PdII/AgI in a Single Macrocycle.

Inauthentic: Pyrazolate-Bridging Dinucleating Ligands Containing Hydrogen-Bond Donors: Synthesis and Structure of Their Cobalt Analogues

Authentic: Manganese Complexes of 1,3,5-Triaza-7-phosphaadamantane (PTA): The First Nitrogen-Bound Transition-Metal Complex of PTA

Authentic: Structure, Luminescence, and Adsorption Properties of Two Chiral Microporous Metal-Organic Frameworks

Based on this (small) sampling, the program doesn't appear to do any better than if it were to guess randomly. I wonder if this thing is even supposed to work, or if it just returns a random result based on a hash of the paper or something?

Re:Typos

[brian0918]

E-mail spambots have been making typos for years.

I am in awe

[DingerX]

So I go there, and I start shoving it text from my hard drive. I try:

A) Text of an article (Philosophy) I (native English speaker) wrote in Italian: 98.5 Authentic.
B) Text of an article I wrote in English (History): 87.8
C) Text of an article (History) written in French by a native French speaker and translated into English: 93.2
D) Critical edition of a 14th-century Latin text (Theology): 97.7 Authentic.
E) Documentation to a Field Artillery Simulation: 95.3
F) A completely bogus narrative for a monastic order that doesn't exist, written in a style that mimics A)-C): 16.8% Inauthentic

So in this case, we have a human written document that has superficial meaning, but is written as a "fake scientific paper", and registering as such.

And yes, I did read the "purpose" of the page; I know it's not supposed to detect it.


And yet it does, decisively.

Re:That's EASY!

[Hal_Porter]

I, for one, peruse the blogosphere. On my Powerbook, wearing a black turtle neck and beret. Stroking my goatee thoughtfully. Sipping a latté in a café

If I could just find a way to recharge my PowerBook from your hatred, I could stop carrying this ugly power adaptor.

Trying Wikipedia articles

[Animats]

I've been trying my own papers and articles from Wikipedia. My own papers all score around 90%. Wikipedia articles that I consider good ones seem to score in the 80% range. Badly written fancruft scores very low.

Some variant on this thing might be useful as a new article filter in Wikipedia. We need more automation over there to stem the flow of incoming dreck.