Slashdot Mirror


Fake Scientific Paper Detector

moon_monkey writes "Ever wondered whether a scientific paper was actually written by a robot? A new program developed by researchers at Indiana University promises to tell you one way or the other. It was actually developed in response to a prank by MIT researchers who generated a paper from random bits of text and got it accepted for a conference."

24 of 277 comments (clear)

  1. Yes! by stupidfoo · · Score: 4, Funny

    I am always wondering what those damn robots are up to!

  2. That's good and all by XxtraLarGe · · Score: 4, Funny

    but I wonder if it can tell if a paper was written by a million monkeys pounding on typewriters?

    --
    Taking guns away from the 99% gives the 1% 100% of the power.
    1. Re:That's good and all by denverradiosucks · · Score: 4, Funny

      Obligatory Simpson's Quote

      Monkey's typing on a typewriter as Mr. Burn's is working on the next great american novel:

      Burns: This is a thousand monkeys working at a thousand typewriters. Soon they'll have written the greatest novel known to man.
      (monkey smoking cigar typing on a typewriter)
      Burns: Lets see. It was the best of times, it was the BLURST of times! You stupid monkey! (Smacks monkey upside his head)

    2. Re:That's good and all by visgoth · · Score: 4, Informative

      Oh, I'm sure the work of monkeys is quite easily identifiable.

      --
      My patience is infinite, my time is not.
    3. Re:That's good and all by iNetRunner · · Score: 3, Funny

      Seems like it would be easier to develop a program that automatically detects /. dupes.. but no.

      *At least the million /. pounding monkeys detect it..*

      --
      Store with salt
  3. Turing test? by Nesetril · · Score: 5, Insightful

    so can a robot write a paper and then decide whether the paper was written by a robot (itself)?

    --
    Jesus said to his disciples: "If you don't have a sword, sell your cloak and buy one" - Luke 22:36
  4. Discrimination by hsmith · · Score: 5, Funny

    I hope the ACLU will ensure that discrimination against metal people will not be allowed to continue.

    1. Re:Discrimination by Iron+Condor · · Score: 3, Funny
      That is people of metal, you biologist

      I think the preferred term is "Ferro-Americans".

      --
      We're all born with nothing.
      If you die in debt, you're ahead.
  5. An interesting experiment by fm6 · · Score: 4, Funny

    Has anybody fed Dvorak's latest column to this program? I've often wondered if he actually writes his columns, or just generate verbiage at random.

    1. Re:An interesting experiment by irregular_hero · · Score: 5, Funny

      "This text had been classified as
      INAUTHENTIC
      with a 24.9% chance of being authentic text"

      No kidding.

    2. Re:An interesting experiment by Anonymous Coward · · Score: 3, Funny

      Yep, I tried that too.
      I also tried another article from ABC News about meat eaters contributing to global warming (http://abcnews.go.com/Technology/story?id=1856817 &page=1). It was inauthentic/28.8%.

      Looks like they have a crafty team of robots there at abc :)

  6. Sadly, It appears that I am a robot. by cbelt3 · · Score: 3, Interesting

    I've taken a long posting that I wrote on my blog and dropped it into the site. And I am Inauthentic. Now I understand the "Bladerunner Moment" comment in the article. I shall begin to surround myself with oddly colored polaroids and snapshots of theoretically implanted ancestors.

    The nice thing is that we've finally settled the argument if machines can be made to drink beer and like it !

  7. Self defeating? by benhocking · · Score: 5, Funny

    It seems like it wouldn't be too difficult to modify the MIT program to use this new anti-robot robot to write papers that this anti-robot robot would not be able to detect. Ideally, this would be done with a learning algorithm (so that it could easily be extended to other anti-robot robot programs), but reverse-engineering the anti-robot robot (by humans) should also provide a solution.

    Now that Indiana U has thrown down the gauntlet, I wouldn't be surprised if MIT responds. Hopefully it will result in an even better paper-writing robot. Ideally, it will lead to dissertation-writing robots. :)

    --
    Ben Hocking
    Need a professional organizer?
    1. Re:Self defeating? by cp.tar · · Score: 4, Interesting

      I recently had to check out an essay-grading robot for my Introduction to Natural Language Processing class.

      I'd fed it the introduction of a randomly generated essay. It got a 4/5 on all counts.

      I figure, if teachers are going to use robots to grade essays, we should use robots to create them in the first place.

      --
      Ignore this signature. By order.
    2. Re:Self defeating? by mctk · · Score: 5, Funny

      Eventually my students won't have to write papers and I won't have to grade them! Think of the potential application of this technology towards education!

      --
      Paul Grosfield - the quicker picker upper.
    3. Re:Self defeating? by BraksDad · · Score: 3, Insightful

      Maybe after a string of anti robot robots, MIT would come up with a robot that would generate a real scientific paper!

      next comes your anti robot robot
      then the anti anti robot robot robot
      and of course the anti anti anti robot robot robot robot
      and the anti anti anti anti robot robot robot robot robot
      ...
      I could go on since cut and paste is so easy ;-)

      Perhaps it would be a million anti's followed by a million and one robots before something useful came out of such an exercise, but wouldn't it be cool to witness?

      --
      Slowly waving my hand - "This is not the sig you are looking for."
  8. Only works for scientific papers by gurps_npc · · Score: 4, Informative
    If you try to use it on any human written NON scientific paper, such as Lincoln's gettyburg address, it almost always considers it false.

    I suspect that it is looking for the conventional thinking with conventional word structure. As such, it is NOT a good idea i

    --
    excitingthingstodo.blogspot.com
    1. Re:Only works for scientific papers by nasor · · Score: 4, Informative

      No, it doesn't even seem to work on scientific papers. I submitted four papers from the latest issue of Inorganic Chemistry and it thought 2 out of 4 were false:

      Inauthentic: Assembly of a Heterobinuclear 2-D Network: A Rare Example of Endo- and Exocyclic Coordination of PdII/AgI in a Single Macrocycle.

      Inauthentic: Pyrazolate-Bridging Dinucleating Ligands Containing Hydrogen-Bond Donors: Synthesis and Structure of Their Cobalt Analogues

      Authentic: Manganese Complexes of 1,3,5-Triaza-7-phosphaadamantane (PTA): The First Nitrogen-Bound Transition-Metal Complex of PTA

      Authentic: Structure, Luminescence, and Adsorption Properties of Two Chiral Microporous Metal-Organic Frameworks

      Based on this (small) sampling, the program doesn't appear to do any better than if it were to guess randomly. I wonder if this thing is even supposed to work, or if it just returns a random result based on a hash of the paper or something?

  9. Re:Typos by brian0918 · · Score: 4, Funny

    E-mail spambots have been making typos for years.

  10. I am in awe by DingerX · · Score: 4, Informative

    So I go there, and I start shoving it text from my hard drive. I try:

    A) Text of an article (Philosophy) I (native English speaker) wrote in Italian: 98.5 Authentic.
    B) Text of an article I wrote in English (History): 87.8
    C) Text of an article (History) written in French by a native French speaker and translated into English: 93.2
    D) Critical edition of a 14th-century Latin text (Theology): 97.7 Authentic.
    E) Documentation to a Field Artillery Simulation: 95.3
    F) A completely bogus narrative for a monastic order that doesn't exist, written in a style that mimics A)-C): 16.8% Inauthentic

    So in this case, we have a human written document that has superficial meaning, but is written as a "fake scientific paper", and registering as such.

    And yes, I did read the "purpose" of the page; I know it's not supposed to detect it.


    And yet it does, decisively.

  11. Read the Paper - Looks at Repetition by Constantine+Evans · · Score: 3, Informative

    Read the paper listed in the menu of the website. The system essentially compresses the text with different window sizes, and then looks at the compression factors. In other words, it is only looking for repetition of strings. This is absurdly easy to fool, and the MIT generator could be easily fixed to pass this filter. For example, try entering a random text once (your post, for example). Note that it fails. Then append a few copies of the same text, and run that through. Your post, when run once, is too short. When run with two copies, it is rejected as 41.2%. When run with three, it passes with 93%. There is a window of repetition level required in order to pass - papers that do not repeat enough are classified as fake, as well as papers that repeat too much (try entering twenty copies of your post).

    It should be relatively simple to make a random paper generator that always passes this test with a higher probability than human-written papers.

  12. Re:That's EASY! by Hal_Porter · · Score: 5, Funny

    I, for one, peruse the blogosphere. On my Powerbook, wearing a black turtle neck and beret. Stroking my goatee thoughtfully. Sipping a latté in a café

    If I could just find a way to recharge my PowerBook from your hatred, I could stop carrying this ugly power adaptor.

    --
    echo -e 'global _start\n _start:\n mov eax, 2\n int 80h\n jmp _start' > a.asm; nasm a.asm -f elf; ld a.o -o a;
  13. Trying Wikipedia articles by Animats · · Score: 4, Interesting
    I've been trying my own papers and articles from Wikipedia. My own papers all score around 90%. Wikipedia articles that I consider good ones seem to score in the 80% range. Badly written fancruft scores very low.

    Some variant on this thing might be useful as a new article filter in Wikipedia. We need more automation over there to stem the flow of incoming dreck.

  14. Re:That's EASY! by Unski · · Score: 3, Funny

    Sir I regret to inform you that you are a ruffian. I for one sit not in a place so vile and common as a 'café', examining the flawed writings of others, but in a temple constructed purely out of my supercilious transcendent superiority. I consume nothing so plebeian as 'The Internet' but rather a rasterized, marked-up and projected form of my own rigourous, peerless stream of consciousness (with blue aqueous scroll-bars). I have no need for facial hair or indeed any of your corporeal trappings and hence know not the joy of stroking a 'goatee'.

    And now I must mod you as Troll, for surely you must know that the PowerBook, created on the Seventh day, is immaculate in it's design and conception and therefore the only possibility is that you seek to trifle with the emotions of our brethren, in crudely ascribing to Our Power Adapters the property of ugliness. If you were truly one of Us you would know that Steve created all in his own beautiful image.

    btw you haven't got a couple of rubber feet for an ibook going spare have you mate?