Slashdot Mirror

← Back to Stories (view on slashdot.org)

Experiment On Public Pre-reviewing and Discussion of Workshop Paper Submissions (reddit.com)

Posted by timothy on from the wisdom-of-crowds dept.

An anonymous reader writes: The ADAPT workshop (6th international workshop on adaptive, self-tuning computing systems) is trying a new publication model: all papers have been submitted via Arxiv, are now publicly discussed via Reddit, and will then be selected by a Program Committee for a presentation at the workshop. The idea is to speed up dissemination of novel ideas while making reviews more fair and letting the authors actively engage in discussions, defend their techniques, fix mistakes and eventually improve their open articles.

41 comments

  1. So now the SJWs... by Anonymous Coward · · Score: 0

    want control of papers for a conference? This should be interesting.

    1. Re: So now the SJWs... by Anonymous Coward · · Score: 0

      Since they are in the lead against the fight against the Tepublicans, why shouldn't they?

    2. Re: So now the SJWs... by Anonymous Coward · · Score: 0

      Why do you hate justice?

    3. Re: So now the SJWs... by Anonymous Coward · · Score: 0

      Someone has to counter the Republican's control of academics in the US.

    4. Re: So now the SJWs... by Anonymous Coward · · Score: 0

      The Republicans want us to die and limiting information and education is their biggest weapon.

    5. Re: So now the SJWs... by Anonymous Coward · · Score: 0

      SJWs stand against rape so why shouldn't they also stand against academic fraud?

    6. Re:So now the SJWs... by Beck_Neard · · Score: 1

      Figures that this would be the first response on slashdot.

      Your two-bit brain is not capable of understanding the real world. Ask your parents for a genetic refund.

      --
      A fool and his hard drive are soon parted.
    7. Re:So now the SJWs... by Anonymous Coward · · Score: 0

      Go shave your beard, Mr. Three and a Half MILLION UID...

  2. Integrated Face System by Anonymous Coward · · Score: 0

    the integrated face system is a system that reviews faces and verifies them for submission.

    1. Re:Integrated Face System by Anomalyst · · Score: 1

      the integrated face system is a system that reviews faces and villafies them for submission.

      FTFY

      --
      There is no right to feel safe thru security vaudeville at the expense of everyone's freedom, privacy and tax money.
    2. Re:Integrated Face System by Anonymous Coward · · Score: 0

      "FTFY" should only use real words. Unless you really meant to turn something into a villa.

  3. Not new by Anonymous Coward · · Score: 0

    Other journals do exactly this, just not with arXiv or Reddit:

    www.nonlin-processes-geophys.net

    1. Re:Not new by guises · · Score: 2

      The Reddit thing is the notable part, and bizarre. A very odd choice. It seems likely they're doing it for publicity - no different from Facebook integration. It's just a workshop after all, and doing something weird like this can draw people in.

    2. Re:Not new by Anonymous Coward · · Score: 0

      Via Reddit? The SJW will get them.

    3. Re: Not new by Anonymous Coward · · Score: 0

      No, the CONservates have taken over that site.

    4. Re: Not new by Anonymous Coward · · Score: 0

      Allowing coontown to exist for so long proves that.

    5. Re:Not new by the_povinator · · Score: 1

      Agreed. The ICLR conference, which is a machine learning conference organized by Yann LeCun (who now heads Facebook's machine learning group), is a bit like this. They use their own site for discussions, not Reddit, though.

      --
      The .sig is dead, and I believe I had a hand in killing it.
    6. Re:Not new by serviscope_minor · · Score: 1

      The Reddit thing is the notable part, and bizarre.

      I interpret it differently.

      Reddit is basically a discussion forum and allows voting on stuff. I was assuming they they were going to reddit because that meant they don't have to set up their own forum, secondly, don't have to persuade everyone to sign up to their own forum (a probably nontrivial fraction of people will already have reddit logins).

      --
      SJW n. One who posts facts.
    7. Re:Not new by enriquevagu · · Score: 1

      No, they are not doing this for publicity. Their goal is to improve the review process by attracting more people interested in the topic to the review process and make it more open.

      I happened to attend the workshop last year and there was a very interesting discussion at the end about how to modify a review process to make it more open. While I didn't take part in the discussion, there were many aspects considered about the open peer-review process, both positive and negative. For example, some authors might be frightened to submit a paper when sending preliminary versions of their work. The selection of Reddit and ArXiv didn't have any publicity (or political) objective, they were just tools familiar to the people involved in the organization and the discussion.

      I am some skeptical to this model, but still is a very interesting experiment so it will be nice to see how it compares to the reviews from previous editions.

  4. What happens to the rejections? by Anonymous Coward · · Score: 0

    So, devil's advocate: Say I'm an author that submits my paper. If it's rejected, what prevents someone from stealing my idea and publishing it as their own at another conference? This is a very real concern. If the answer is "oh well", then I, personally, would never submit to such a conference.

    I honestly do like the "crowd-funded" reviewing, though - it reduces that chance of a few crummy reviewers having the ultimate say on your paper. But does the exposure come with a cost?

    1. Re: What happens to the rejections? by Anonymous Coward · · Score: 0

      What stops anyone from doing that with any paper, accepted or rejected for any journal or conference? You would still have documentation of your attempt on Arxiv. It is not like journals have a magic trick for catching people who republish stuff with slightly tweaked wording.

    2. Re:What happens to the rejections? by fph+il+quozientatore · · Score: 1

      The same reason why you can't steal an already-published idea: your paper is publicly available and timestamped on arxiv, so everyone provably knows that it was your idea first. Moreover, a google search will likely reveal your previous contribution.

      --
      My first program:

      Hell Segmentation fault

    3. Re:What happens to the rejections? by techno-vampire · · Score: 1

      One other thing I like about this form of reviewing articles is the fact that it can also prevent a small group of reviewers with an axe to grind rejecting everything that doesn't fit their world-view. I won't say that it's being done right now, but I do know that it's been claimed at least once.

      --
      Good, inexpensive web hosting
  5. Unexpected absence by Anonymous Coward · · Score: 0

    1) Organize the meeting via Reddit, each participant somehow getting himself[1] shadowbanned.
    2) Each one believes that he's[2] the only participant, consequently abandoning the meeting.
    3) ...
    4) Profit!

    [1, 2] Yes, I'm being intentionally non-PC.

    1. Re:Unexpected absence by Anonymous Coward · · Score: 0

      intentionally non-PC.

      Gay men all use Macs?

      captcha: sweating

  6. Ah, arXiv by mbone · · Score: 4, Interesting

    You do know that some people are blocked from arXiv, and at least in some cases there is no obvious reason why (and no real appeal)? (Opponents of string theory, for example, seem to get this, or at least complain about this, fairly often.) I have seen this in action, it is real and it is capricious.

    I do not think that arXiv is suitable for a filter for a public meeting as long as its internal filtering is opaque in this fashion.

    1. Re:Ah, arXiv by fph+il+quozientatore · · Score: 3, Interesting

      I have never heard of this, and I am interested. Can you name an example of a respectable scientist (not a "fringe" controversial person, I mean) who has been banned?

      --
      My first program:

      Hell Segmentation fault

    2. Re:Ah, arXiv by Anonymous Coward · · Score: 0

      If you are unable to contact the organizer of a workshop when something like that comes up and work out an alternative when needed, then you will likely have much bigger problems.

    3. Re:Ah, arXiv by Anonymous Coward · · Score: 2, Informative

      I have never heard of this, and I am interested. Can you name an example of a respectable scientist (not a "fringe" controversial person, I mean) who has been banned?

      Nobel Laureate Brian Josephson had this happen to him. He posted a record of his interactions with the arXiv moderators at

      http://www.tcm.phy.cam.ac.uk/~bdj10/articles/arxiv_correspondence.html

      Often the people who get their papers rejected by arXiv are just ordinary researchers who are trying to post ordinary papers. One such example is the following.

      http://blog.tanyakhovanova.com/2013/12/arxivs-police

    4. Re:Ah, arXiv by Anonymous Coward · · Score: 1

      Has Josephson been banned, or is he just upset that his paper, which seems to be more on the philosophy of biology and quantum mechanics, was deemed not appropriate for being listed as quantum physics? The paper is still on arxiv. His communications with the moderators and higher ups could maybe have been a little more professional. Not that it should matter. But the actual complaint is very questionable, and it is not a clear case of the moderators blatantly disregarding what they should be doing, as it is pretty easy to argue the the paper shouldn't have been cross-posted, even if the benefit of the doubt could have let it slip by...

    5. Re:Ah, arXiv by Anonymous Coward · · Score: 1

      Has Josephson been banned, or is he just upset that his paper, which seems to be more on the philosophy of biology and quantum mechanics, was deemed not appropriate for being listed as quantum physics?

      Josephson claims that he was temporarily "blacklisted" from posting to arXiv. See

      http://www.tcm.phy.cam.ac.uk/~bdj10/archivefreedom/main.html

    6. Re: Ah, arXiv by Anonymous Coward · · Score: 0

      The mod response wording was not appropriate, but the story starts off with saying he knew he was breaking the rules with what he did... isn't there anyone to champian this cause who isn't doing things half wrong?

    7. Re:Ah, arXiv by mbone · · Score: 2

      I have never heard of this, and I am interested. Can you name an example of a respectable scientist (not a "fringe" controversial person, I mean) who has been banned?

      Marni Sheppeard.
      Peter Woit.

      Note that they are not (as far as I can tell) banned, just blocked. Nothing is made public, it's just that certain things seem to happen consistently. And, in my experience, moderated papers are not available to the public.

      Note that the real problem here is not that papers are moderated. I understand the desire for moderation. It's the way it's being done that is problematic.

    8. Re:Ah, arXiv by Sibko · · Score: 1

      Can you explain to me why a "fringe" scientist (err... controversial person...?) shouldn't be allowed to speak on Arxiv? I just find it really curious that you immediately imply it's okay to censor a certain kind of speech you don't personally like. I mean, you're pretty much using the No True Scotsman fallacy right here; if the OP comes up with a name you can just declare him to not be a "real" scientist and you'll never be proven wrong.

      You either have an open forum and the idiots that come with that, or you don't. Arxiv would, frankly, do better with natural meritocratic filtering options - like science is meant to be based upon! - than to allow petty bureaucrats to control what is and is not considered science. We already HAVE methods to discard junk science: It's fundamental to how the scientific method works. Censoring people just because they present "controversial" ideas - that isn't science. That isn't just not science, that is pretty much the exact opposite of how science is supposed to work.

    9. Re:Ah, arXiv by Anonymous Coward · · Score: 0

      Peter Woit

      In the case of Peter Woit, a member of the arXiv advisory board eventually posted an explanation.

      https://golem.ph.utexas.edu/~distler/blog/archives/000760.html

      Apparently, despite the fact that Woit is a professor of mathematics at Columbia University, the arXiv moderators felt that he did not qualify as an "active researcher" because he only posted two papers within the last 5 years (at the time of the complaint).

    10. Re:Ah, arXiv by serviscope_minor · · Score: 1

      Can you explain to me why a "fringe" scientist (err... controversial person...?) shouldn't be allowed to speak on Arxiv?

      Because the actual scientists using arXiv don't want to have to manually filter out a crapflood of spam from whackadoodles. Make no mistake, if the normal channels get too much "fringe" science, then the real scientists will leave and it will be worthless for anything but timecube theories.

      I just find it really curious that you immediately imply it's okay to censor a certain kind of speech you don't personally like

      Aw jeez not this shit again. Yeah arXiv has great power and great responsibility blah blah but part of that is not crapflooding working scientists inboxes with loopy spam. The right to speak is not the same as the right to force others to listen. You are demanding the latter.

      No matter where you set the filters someone will complain. And if you have no filters, you'll get nothing but a massive flood of penis enlargement pill advertisements.

      Arxiv would, frankly, do better with natural meritocratic filtering options - like science is meant to be based upon!

      Hahah it's funny you believe that could actually happen.

      And, who gets to judge merit? Bonus points with coming up for a method which (a) is immune to criticism about cliques and (b) immune to spam.

      We already HAVE methods to discard junk science: It's fundamental to how the scientific method works.

      It's a trap! Our methods cannot repel spam floods of this magnitude! (you know if you just open the door and let anything in)

      --
      SJW n. One who posts facts.
    11. Re:Ah, arXiv by Anonymous Coward · · Score: 0

      Arxiv would, frankly, do better with natural meritocratic filtering options - like science is meant to be based upon!

      Hahah it's funny you believe that could actually happen. And, who gets to judge merit? Bonus points with coming up for a method which (a) is immune to criticism about cliques and (b) immune to spam.

      Fair points, but it's not that hard to come up with filters that could each -partially- help reduce the crackpot submissions in the search results. Some suggestions for filter options that could help one finding the papers with 'merit':
      - Is the author affiliated with any relevant institute? OR
      - Does the author hold a degree relevant to the topic of his paper? OR
      - Have any of the author's previous papers been widely cited?

      I think that rather few crackpot and/or spam submissions would get through filters like these. And although these filter suggestions are obviously not immune to criticism (because nothing is), accepting more papers while improving search filtering options could help prevent stuff like cliques suppressing alternative views/theories from getting published. Author positions, degrees and citation counts are fairly objective measures of merit, although not infallible ones.

    12. Re:Ah, arXiv by mbone · · Score: 1

      If you are unable to contact the organizer of a workshop when something like that comes up and work out an alternative when needed, then you will likely have much bigger problems.

      I was actually thinking more about the organizers than the submitters (I have organized scientific meetings). One reason why organizers like this is that the arxiv paper submission is part of their automated submission process (i.e., they don't have to set up a paper hosting service, arXiv does it for them). This means that there is a real risk that arXiv is involved in their Editorial process, and that they might not even know it, and I think most conference organizers would find that unacceptable.

    13. Re:Ah, arXiv by eric_harris_76 · · Score: 1

      Not banned, and not from arXiv, but a recent Nobel laureate was co-author of a paper that recently didn't survive peer-review at a couple of supposedly prestigious medical journals (JAMA and NEJM, as I recall). (Link below.)

      They took their paper elsewhere, where it was better received.

      How recent? The latest round of awards. How recently? The rejections happened before the awarding of the prize.

      Feather in the cap of the publishing journal, and a black eye for the two that rejected it.

      https://duckduckgo.com/?q=angus+deaton+paper+rejected&t=ffcm

      --
      There's no time like the present. Well, the past used to be.
    14. Re:Ah, arXiv by Anonymous Coward · · Score: 0

      Can you explain to me why a "fringe" scientist (err... controversial person...?) shouldn't be allowed to speak on Arxiv?

      Because the actual scientists using arXiv don't want to have to manually filter out a crapflood of spam from whackadoodles.

      ArXiv can be a repository of preprints and working papers, or it can be an electronic journal, but it can't be both. You, apparently, want it to be a carefully manicured journal that's free from "whackadoodles". But there are plenty of exclusive, carefully manicured journals out there. What there isn't is an alternative preprint server, because arXiv put them all out of business. The scientific community needs a preprint server that accepts papers before they've been peer reviewed because (1) sometimes authors want their colleagues' advice before they submit a paper for publication, (2) authors want to have a record of their submission to protect against unscrupulous reviewers, and (3) the arXiv helps protect the authors from copyright issues (because after the paper is submitted to a traditional journal, the author may lose the right to share his paper, depending on the nature of the copyright agreement he signs).

      Personally, I think what you want is a digest of arXiv submissions that's free of non-mainstream papers. And that's fine, but your desire should not interfere with the legitimate needs of others, enumerated above.

  7. Just Havta GNow by Anomalyst · · Score: 1

    is the program to be in Java and interpreted, C and compiled or specific to a CPU family and assembled, possibly in binary and toggled in via the front panel (don't reviews require a panel of ancients)?

    --
    There is no right to feel safe thru security vaudeville at the expense of everyone's freedom, privacy and tax money.