Web of Trust For Scientific Publications

An anonymous reader writes "PGP and GnuPG have been utilizing webs of trust to establish authenticity without a centralized certificate authority for a while. Now, a new tool seeks to extend the concept to include scientific publications. The idea is that researchers can review and sign each others' works with varying levels of endorsement, and display the signed reviews with their vitas. This creates a decentralized social network linking researchers, papers, and reviews that, in theory, represents the scientific community. It meshes seamlessly with traditional publication venues. One can publish a paper with an established journal, and still try to get more out of the paper by asking colleagues to review the work. The hope is that this will eventually provide an alternative method for researchers to establish credibility."

Comments

Wikipedia

[jgtg32a]

This is exactly what Wikipedia needs to implement.

This will allow it to overcome the credibility problems that it has.

Re:Wikipedia

[MoxFulder]

Indeed, the implementation of flagged revisions is currently being debated for the English Wikipedia, and was the subject of a recent ./ article.

A lot of the debate centers on exactly what the "signing" process will entail in terms of responsibilities and consequences for the articles subject to it.

I don't think a one-size-fits-all approach to trust networks is a good idea. Requirements for effective trust in key sharing, peer review, and wiki content may differ and I think it's appropriate for each to develop a fine-tuned approach, while borrowing good ideas from one another.

Re:Wikipedia

[Anonymous Coward]

Wikipedia [snip] credibility

Oh shush, you almost made me chuckle.

The last time I went to Wikipedia to look up something on Bellerophon it mentioned that he sucked balls. Now, knowing the Greeks of old, that might've been true, but as it turned out it was just plain old vandalism, and no actual ballsucking occurred.

I wouldn't trust wikipedia any further than I can throw it without external references, and no web of trust is going to change that, especially if that web depends on the edit-war-hungry denizens, sockpuppets, strawmen, and trolls that enjoy wikipedia. Great place to start looking for information though.

Re:Wikipedia

[ed.mps]

Great place to start looking for information though.

insightful

Re:Wikipedia

[mahadiga]

This will allow it to overcome the credibility problems that it has.

TRUTH is important than TRUST.

Re:Wikipedia

[puusism]

Half a year ago I wrote an article about how to implement Wikipedia peer review with digital signatures:

http://cameralovesyou.net/random/wikipedia-digital-signatures.html

Re:Wikipedia

[linhares]

Great place to start looking for information though.

[citation needed]

Re:Wikipedia

[puusism]

Answering to myself... here's the Wikipedia Village Pump discussison about the proposal. The proposal was rejected (or at least not implemented), since it was thought to produce a disincentive for editing signed articles.

Re:Wikipedia

[linhares]

Dormitory is important than Dirty Room

Evangelist is important than Evil's Agent

Desperation is important than A Rope Ends It

The Morse Code is important than Here Come Dots

Slot Machines is important than Cash Lost in 'em

Animosity is important than Is No Amity

Mother-in-law is important than Woman Hitler

Snooze Alarms is important than Alas! No More Zs

Alec Guinness is important than Genuine Class

Semolina is important than Is No Meal

The Public Art Galleries is important than Large Picture Halls, I Bet

A Decimal Point is important than I'm a Dot in Place

The Earthquakes is important than That Queer Shake

Eleven plus two is important than Twelve plus one

Contradiction is important than Accord not in it

Re:Wikipedia

[linhares]

Answering to myself... here's the Wikipedia Village Pump discussison about the proposal. The proposal was rejected (or at least not implemented), since it was thought to produce a disincentive for editing signed articles.

Yup, true, but it's flying NOW! Jimmy Wales just announced it today. (Probably on /. tomorrow.)

Re:Wikipedia

[Anonymous Coward]

We have the same problem in Computer Vision. Ever paper must be accompanied with a means to validate the results, like a Java program or a C++ program or Matlab code, preferably GPL or a licence respecting copyright, for others to reuse it (and not start afresh, this is the meaning of scientific comunity) public datasets and a section or recovering the published results.

Re:Wikipedia

[glittalogik]

President Clinton of the USA is more important than To copulate he finds interns.

Still needs a root

[orclevegam]

The problem of course is that at some level you still need to have a known good reference for the whole "web" to work. It doesn't help your credibility at all if you've got a paper signed by 100 of your closest crackpot buddies. What this does provide is the ability for someone in addition to established authorities to vet a work, such that a well respected member of the scientific community can easily and in a verifiable fashion signify his approval of a paper.

Re:Still needs a root

[644bd346996]

Wouldn't metrics like the Erdos number suffice? Calculate the weighted distance from known experts, such as Nobel laureates, Fields medalists, etc. It isn't that hard to notice a clique that is only weakly connected to the larger network.

Re:Still needs a root

[hobbit]

The root should be established by your opinion on papers, their authors, and their reviewers. If you see eye to eye with an author or reviewer about a paper, you should increase your trust of them. In other words, everyone has a separate view onto the web of trust.

Re:Still needs a root

[DancesWithBlowTorch]

Actually, it doesn't need a root. Quite on the contrary, the developing graph could give amazing insight into the structure of research communities. It would be possible to identify researchers forming links between otherwise almost disconnected areas of research, and to find the great minds at the centre of such blocks. There is no "root" to the web of scientists. Even people like Erdös were only ever local subroots.

I think this project is a great idea. Unfortunately, it currently seems to consist of only a command line tool to sign reviews with GPG. That's nowhere near enough if it is to thrive beyond the CS world. It needs a simple, rock-solid GUI, and most importantly, lots of eye-candy for the graph. It will need to look cool and work well to build up the momentum for this to work at all.

Re:Still needs a root

[Zerth]

Much like Page Rank, you don't need a "known good". Start with everyone on even footing, passing their value on to those they sign, receiving value from those who sign them, and then iterate until it reaches a reasonably steady state.

I don't recall if there is a general "scientist" number, like there is Erdos for mathematicians, but in the off chance a crackpot network was to form and become larger than any of the networks of actual scientists, then you might want a "known good", but it wouldn't matter who in the network was it, as long as there was connectivity.

If it is the case that biologist or material engineers, etc, don't co-publish as often as mathematicians or have smaller network densities, then you are screwed without an oracle that could distinguish good scientists from bad, as the need for "known goods" would increase rapidly as connectivity decreased.

Re:Still needs a root

[Anonymous Coward]

already exists - see dontdatehimgirl.com

Re:Still needs a root

[orclevegam]

Yes, but for the web of trust to have value to the casual observer certain respected authorities need to be established which is something people tend to do naturally on their own. If something like this is implemented it will most likely never have an official authority, but it will have several de-facto ones that people will come to associate as authority figures. Essentially someone not well entrenched in a particular field may not know if Dr. X who's work is signed by Dr. Y, is any good, but they have heard of Journal Z who signed for Dr. Y, and therefore provide credibility for Dr. X.

Re:Still needs a root

[orclevegam]

That's true, and would be an interesting use of the data, but you have to consider the primary application this would be applied to. Something like this holds little value for someone in the same field who travels in the same circles as they're already aware of the reputations and merit of other researchers in their field of study, or baring a recommendation from someone they know, they should be capable of reviewing the paper for themselves and deciding if it has merit. Where this does provide insight is to the outside observer who may not know who the crackpots are, who can be trusted, and who lacks the detailed knowledge of that field to be able to evaluate the merit of a paper. In this later case there must be certain organizations or individuals that are well established to the point of being discernible to the outsider and can act as a starting point for establishing credibility.

Re:Still needs a root

[PDAllen]

Journal publications are basically used to tell people who don't work in your area that you're doing decent work. If someone is doing decent work in an area you're a specialist in, you probably know them at least by sight and you probably hear about their results fairly soon after they prove them; the journal paper may well come a year or two later.

But if you want funding, or you want a job, you have to convince a bunch of people who know very little about your area that you are a valuable person. The easiest way to do that is to point at recent papers in good journals (which, really, isn't so different to the web of trust idea: I have a paper in CPC because someone thought my work was good enough to go there, that kind of thing).

There are lots of problems with the sort of metric you suggest; you need something relevant to now, you don't want it to discard people who do good work on their own or in tight groups (and there are quite a few of the latter), you don't want it to be distorted by the sort of mathematician who will publish every result they can get in any collaboration (there are quite a few, some of whom are very good and very well-connected but still publish some boring results along with the good ones).

Re:Still needs a root

[Sloppy]

Whoever happens to be reading a paper at the time, would be the root. If you haven't set your opinion of any of the 100 crackpots to greater than zero, then their signatures don't matter.

And yes, that means trust is subjective. Welcome to the real world. Anyone who tries (or has tried, and it happens a lot) to set up a trust system that isn't subjective, is deluded.

Re:Still needs a root

[edcheevy]

I wonder if subwebs will form around subroot voids? The clear "roots" I can think of for topics in my field would be less interested in using a tool like this because they're already known. It's us fresh-faced grad students and other unknowns who would be more likely to a) want the extra credibility and b) have the time to pursue it.

Re:Still needs a root

[hobbit]

Yes, but for the web of trust to have value to the casual observer certain respected authorities need to be established which is something people tend to do naturally on their own.

Indeed. I'd like to think that if I looked at the web of trust through, e.g., the IEEE website, I'd get something approximating the combined trust of their membership. Rent-an-opinion, if you will!

Re:Still needs a root

[Anonymous Coward]

Citations are probably the best way of assigning credibility to a publication. If you agree with it, and YOUR research depends on it (not necessarily solely, but if the premise is essential to the validity of your work), then you cite it.

Re:Still needs a root

[martin-boundary]

Where this does provide insight is to the outside observer

You mean like grad students?

Re:Still needs a root

[ralphbecket]

You've hit the nail on the head. For example, read the Wegman report into the Mann et al. "hockey stick" paper in the climate science debate. The right way to judge a result in science is not by the first paper on the topic, but by the number of independent follow-up studies, all of which support the result. The idea of independence is something you need to be stringent about: ideally it requires different (unconnected) researchers obtaining new data and performing their own analyses. People from the same research group rejigging old data is much less convincing.

Re:Still needs a root

[shadwstalkr]

You mean like grad students?

Who do you think is doing all the work? From my experience as a grad student, the faculty are more "outside observers" than grad students, especially in the student's thesis area. Faculty tend to deal a lot more with finances and administration than research.

Re:Still needs a root

[TapeCutter]

...read the Wegman report into the Mann et al. "hockey stick" paper...

Please don't encourage people to waste their time by conflating politics with science. We are talking about trusting scientific papers. Mann's "hockey stick" papers have been published in the journal science (2nd highest in academic journal rankings). The correct way for McIntyre and McKitrick to attack them is to publish contrary journal articles, asking a political committe devoted to 'energy' to act as judge and jury on a particular scientific result is idiotic in the extreme. It demonstrates just how desperate some politicians are to burry what they see as the "smoking gun".

Wegman's report was a request from a political committe to "evaluate whether the McIntyre and McKitrick (2005) (MM05) criticism of Mann, Bradley and Hughes (1998) (MBH) had statistical merit."

Re:Still needs a root

[deadzaphod]

I agree that this is a great idea, and with your criticism of the implementation. I would like to see this as a website, compatible with the existing tool, funded by Google, and written by someone other than me (although I am willing to help). It could be sort of like MySpace for science ;-)

Re:Still needs a root

[martin-boundary]

You must have gone to a top university. In my experience, grad students have no idea what their thesis is about or who's who in their chosen field until about 3/4 of the way through (but luckily there's always panic to smooth over the last 1/4 :)

Now, postdocs, that's another story.

Re:Still needs a root

[smallfries]

That already exists without a web of trust. There is a lot research already into analysing citation and co-citation graphs for the features that you mention. In network-analysis terms the links that you describe are called gateway nodes.

Re:Still needs a root

[ralphbecket]

You should read some statistics. Unlike you, I do recommend that people read key documents from different players in the debate rather than just take the party line from Real Climate (which I believe was started by Mann and his colleagues).

You might be interested in Wegman's CV before suggesting his findings on the Mann et al. hockey stick were constructed politically. There's this interesting entry in his CV, for example: Appointed Chair of the Committee on Applied and Theoretical Statistics, National Academy of Science, 2004. A separate NAS report also criticised the hockey stick, albeit in a more mealy mouthed fashion - not that you would know that if you only read Real Climate, which somehow translated the NAS panel's conclusions into a ringing endorsement of Mann et al.

Of course, I don't actually expect you to read any of this stuff, but other people following this discussion might be curious.

Re:Still needs a root

[Hal_Porter]

That reminds me of the funny epithet "graduate student algorithm" for things that only work with a vast amount of human intervention.

http://cotty.16x16.com/compress/fractcpr.txt

Barnsley, however, armed with his Collage Theorem, thought he had it
solved. He applied for and was granted a software patent and left acade-
mia to found Iterated Systems Incorporated (US patent 4,941,193. Alan
Sloan is the co-grantee of the patent and co-founder of Iterated Sys-
tems.) Barnsley announced his success to the world in the January 1988
issue of BYTE magazine. This article did not address the inverse problem
but it did exhibit several images purportedly compressed in excess of
10,000:1. Alas, it was a slight of hand. The images were given sugges-
tive names such as "Black Forest" and "Monterey Coast" and "Bolivian
Girl" but they were all manually constructed. Barnsley's patent has come
to be derisively referred to as the "graduate student algorithm."

Graduate Student Algorithm
      o Acquire a graduate student.
      o Give the student a picture.
      o And a room with a graphics workstation.
      o Lock the door.
      o Wait until the student has reverse engineered the picture.
      o Open the door.

Attempts to automate this process have continued to this day, but the
situation remains bleak. As Barnsley admitted in 1988: "Complex color
images require about 100 hours each to encode and 30 minutes to decode
on the Masscomp [dual processor workstation]." That's 100 hours with a
_person_ guiding the process.

Re:Still needs a root

[TapeCutter]

I did not say Wegman was political, I am saying he was appointed by a political commitee to give an OPINION. If you want to appeal to authority then by all means go ahead but I am kind of fond of the scientific method.

"Of course, I don't actually expect you to read any of this stuff, but other people following this discussion might be curious."

I have followed the science for at least 25yrs, I also followed the Wegman thing while it was happening. You don't have a clue, you are grasping at straws to try and debunk the hockey stick, perhaps because you think that will debunk AGW (it won't). Wegman, McKintrick & McKintyre all failed and so have you.

Re:Still needs a root

[ralphbecket]

You have me confused: are you saying Wegman's testimony on the hockey stick should be ignored not because of what he said or his evident qualifications as an independent expert, but because of who asked him for the report?

Personally I find it hard to believe you really respect the scientific method when you (a) prefer to accuse people of playing politics rather than addressing the content of their argument and (b) argue against argument by authority (I didn't: I posted a link to Wegman's CV to point out that he is eminently qualified to judge the quality of the hockey stick's statistical basis) *after* having claimed that Mann et al. was authoritative because it got published in Science. Oh, and for the record, M&M did publish their criticism in journals after peer review.

By the way, *I'm* not trying to debunk the hockey stick: it's already been done by M&M and checked by Wegman and endorsed by the NAS. And please do continue your ad hominem against me, too: it's very entertaining.

A new tool appears!

[Anonymous Coward]

Coming this fall. Jean Claude Van Damme in...Web of Trust.

Two certificates enter, one leaves.

But what about the new tool? Does Google have the favor?

Rated R for violence and sexual situations.

Weird objection

[nine-times]

I'm sometimes bothered by the stress on studies being "verified" by something like a peer-review process. Not that I don't understand why it makes sense. It's a pretty reasonable attempt to sort valid work from crap, but...

There's still a certain way in which it's just an appeal to authority. It's people saying, "We should accept what this scientist says because other scientists say that he's right." I guess what I'm saying is that I worry that, as a process like this becomes more technical, people will be more likely to confuse a statement like, "This study has been reviewed by other scientists and seems to have merit," with something more like, "This study is correct, infallible, and indisputable."

And I guess part of the reason I worry about this is that there may be cases where what "everyone thinks" (i.e. the common conception even among experts) is wrong, and some random nutcase is right. It almost never happens, but it happens sometimes. It seems to me that a technical method of assigning trustworthiness of ideas in a web of trust might possibly lead to having all the groundbreaking ideas go into a spam filter somewhere, never to be seen again.

Re:Weird objection

[orclevegam]

Interesting point. Something that occurs to me however is that any paper worth its salt really has two things that can be verified/approved independently of each other. The first, and easier of the two is the test procedures, and any math/established formula used. Assuming that no flaw can be found with that, you move on the second part, which is the theory being proposed to explain the results of the tests and/or how any discrepancies between the observed results and the theory are handled. It's entirely possible to have a paper that has excellent test results that raise interesting questions, but a completely nutjob theory attached to it. To ignore the results of the tests because the theory is crazy is to throw the baby out with the bath water. Likewise, just because the theory proposed in a paper is a well established and respected theory is no reason to sign off on flawed test results.

Re:Weird objection

[m50d]

Everything you say is true, but the comparison to a spam filter is apt. How many people read through and check all their emails carefully, in case they really are the heir of a deceased nigerian prince? Yes, some non-peer-reviewed papers might be valid; brilliant even. But it's simply not worth the cost of wading through the crap.

Re:Weird objection

[evanbd]

As a third party, there is no way I have the time to follow the chain of logic that results in a modern scientific paper from first principles. At some point, I have to accept some of the preconditions of the paper without verifying them, because doing otherwise implies that I am an expert in the particular field the paper is relevant to. And there are plenty of cases where I want to make use of a result from a field that is related to my work but in which I am not an expert.

Appeal to authority is the fundamental reasoning technique I apply in such cases. A respected expert says it is so, and so I will trust them until I have reason to believe otherwise. That trust should not be blind -- if I am presented with reason to, I will happily re-evaluate that trust. Perhaps the expert is mistaken. But, in the interest of actually getting something done myself, I will accept as a default position that the experts know what they're talking about.

Re:Weird objection

[fuzzyfuzzyfungus]

In science(and generally) the "appeal to authority" is complicated because there are actually several quite different flavors of appeal to authority, which all mean quite different things; but commonly blur together in ordinary use. The following is incomplete; but it hopefully gives a rough outline.

On the one hand, you have the appeal to authority as an argument in itself. This is the classic medieval "According to the philosopher..." stuff. When this happens in science, it is undesirable, since science is supposed to be about the world, not opinion(unless you are doing opinion polls, of course).

On the other hand, you have the appeal to authority as intellectual heuristic: If you don't know about subject X, it is generally most sensible to find somebody who does, and ask them about it. If you don't know who knows, then you ask about that. So, in effect, the statement "X is Y because Professor Z says so." is just (sloppy) shorthand for "I don't know about X; but people I believe to be familiar with the field of X say that Professor Z has done excellent research on X, and Professor Z says that X is Y." This is imperfect, to be sure; but barring the (generally recognized as impractical) strategy of being omniscient, it is more or less the best option.

The picture is further clouded by the way humans actually evaluate information. We didn't evolve our trust metrics to handle scientific papers, we evolved them to deal with social signalling in small hominid kin groups. So, it is often extremely difficult to avoid assigning or subtracting trust for scientifically irrelevant reasons. Again, though, this is something to watch out for, and it is part of why we have to use statistics and logic rather than hunches and feelings; but we don't really have a better option.

The trouble is, when somebody actually makes an argument from authority, they are likely to be mixing more than one flavor into the same statement. It might be a shorthand reference to X's excellent technique and scrupulous data gathering, it might be a sense of respect for X's character, based on personal interactions, it might be some creepy cult of personality thing. These are distinct phenomena; but they can show up together, and in very similar looking statements.

Re:Weird objection

[Khorniszon]

there may be cases where what "everyone thinks" (i.e. the common conception even among experts) is wrong, and some random nutcase is right. It almost never happens, but it happens sometimes.

We have a free (somewhat) market to verify that. If nutcase's idea is right, it WORKS. And if it's againt experts, even better - guy/gal is going to be innovative enough to make some serious buck and gain publicity.

Re:Weird objection

[migloo]

I'm sometimes bothered by the stress on studies being "verified" by something like a peer-review process. (...) there may be cases where what "everyone thinks" (i.e. the common conception even among experts) is wrong, and some random nutcase is right.

Absolutely, and that is exactly how Science progresses. The modern peer review process only adds viscosity to the mechanism of discovery.

Re:Weird objection

[nine-times]

Yeah, I acknowledge that my concern shouldn't really be the primary concern. There's a reason I wanted to call it a "weird" objection.

I just think it is possible to put too much faith in peer review, given that the "peers" reviewing it are also human beings, just as fallible as other human beings. Computers are arguably less fallible in other ways, but of course they can't really make judgements. So I'm just really trying to point out that, in the other cases where we mix fallible human beings with machine judgement, we tend to get very powerful systems that can work well in some ways, but we also tend to end up with important things getting lost in the shuffle.

I used the example of a spam filter. My Gmail spam filter still lets some spam through, and gets occasional false-positives. Still, I use it, and I'd hate to have to filter through all that spam myself.

Re:Weird objection

[ralphbecket]

Peer review is actually pretty weak. It's mainly effective at spotting obvious howling errors. Peer review is not the same as replication and, indeed, many reviewers don't bother to check the equations or data presented in a paper unless they are genuinely suspicious of the conclusion. Replication, not peer review, is the gold standard of science.

Re:Weird objection

[Sabz5150]

It's people saying, "We should accept what this scientist says because other scientists say that he's right." I guess what I'm saying is that I worry that, as a process like this becomes more technical, people will be more likely to confuse a statement like, "This study has been reviewed by other scientists and seems to have merit," with something more like, "This study is correct, infallible, and indisputable."

Getting a group of scientists to fully agree on something is akin to herding cats, or for a better analogy, getting all of Slashdot to agree on the best *nix text editor.

"This study is correct, infallible, and indisputable." is best replaced by "This study has made it through one of the must grueling gauntlets that modern civilization has to offer and made it out the other side, therefore it has merit."

Re:Weird objection

[shadwstalkr]

Appearing in a peer-reviewed journal isn't a stamp of authoritative correctness on a paper, just that a few people thought it was worthy of some space in the journal. Peer-review is (supposed to be) just a rough initial filter to cut down the noise; if a paper is actually good and useful it will be cited often.

Re:Weird objection

[nine-times]

I know it's not supposed to be a stamp of authoritative correctness. That was my point, that (some) people incorrectly treat it as one.

Re:Weird objection

[Sven Tuerpe]

I'm sometimes bothered by the stress on studies being "verified" by something like a peer-review process.

This is a misunderstanding. The role of peer review is not to verify anything. To the contrary, there are many situations where a reviewer will not be able to verify results with resonable effort. Think LHC experiments, Mars probes, etc.

Peer review is really just a spam filter. Reviewers can check whether a publication has novel aspects to it, whether it is relevant to the journal or conference, whether it is presented in a comprehensible manner, whether releated work is properly cited, and so on. A paper that has passed the peer review process is not verified, it is only deemed useful.

There are people who claim otherwise and unfortunately some of them are scientists. Overstating the capabilities of peer review makes sense if one attempts to use science in politics (which isn't wrong per se) and attempts to close political debates on the sole ground of scientific considerations (which is usually wrong).

Recommended reading:

• Three myths about scientific peer review (where one commenter adds a link to this article quoting reviewer comments on famous computer science papers)
• How Are the Mighty Fallen

in Michael Nielsen's blog.

Re:Weird objection

[MadMidnightBomber]

Things like relativity and plate tectonics have taken a while to be accepted because so many people thought they were crazy to start with.

However, it's better to have papers checked than not checked. The reviewers don't necessarily have to agree with the conclusions, but they do have to make sure the study's methodology is sound.

Other people will then go and do similar studies to see if they can reproduce or disprove the results.

Re:Weird objection

[TheTurtlesMoves]

We don't have the time. I have had 7 papers to review in the last 2 months. It takes a long time to do and you don't get paid. On top of that I am suppose to be getting out my own publications.

But peer review does get rid of a lot of rubbish. There are often papers are submitted that have not been finished or have some serious short coming especially wrt previous work. Its not a perfect system and it needs improving, but its not bad either.

Re:Weird objection

[darkwhite]

You should read "The structure of scientific revolutions". There is a large body of work associated with the phenomena you describe, and most scientific communities take a lot of care to prevent the negative effects.

a cumbersome solution in search of a problem

[SuperBanana]

The hash isn't necessary. If the trust relationship between two academic peers includes "worried about him modify the paper after I review it", there is no trust relationship.

In fact, the whole thing isn't necessary. Pubmed, anyone? All someone has to do is pick up the phone and call the reference on a CV and say, "So, what did you think of Dr. X's work on Y?", and they learn more than they will running a program that says "Hashes verified."

This system is also never going to fly with researchers. Most (but not all) of the (brilliant) bio people I've worked with are completely helpless when it comes to technical stuff. Even some of the bioinformatics people who can write amazing algorithms aren't clued in on stuff outside of their field.

Re:a cumbersome solution in search of a problem

[evanbd]

Yes, because the first thing I'd do on seeing a vaguely interesting paper is call up half a dozen random researchers, wait until they weren't busy in the lab to get a comment back, and then eventually have some clue what the consensus among those more directly involved in the field than myself is several hours later. Why not just have them publish their opinions? Then they don't have to answer the same questions repeatedly.

The question isn't "why should we include the hashes?" but more properly "Is there any reason not to use a properly designed digital signature?" The fact that I trust someone is a poor reason to deliberately design a weakness into the review system when it's so easy to avoid. What's that, you need a benefit as well? How about drafts of papers -- using hashes makes it easy to get someone to review the preprint of the paper, and make comments. A later draft could address those comments. Their signature should then only be applied to the first one, not the second, until they review it as well. Revision tracking is a useful feature.

Re:a cumbersome solution in search of a problem

[Sloppy]

The hash isn't necessary.

The hash is harmless (as in: there are no downsides), and a lot faster to sign (and verify) than a megabyte.

If the trust relationship between two academic peers includes "worried about him modify the paper after I review it", there is no trust relationship.

You can trust someone and still have a computer hardware failure, or accidentally hit a key after you accidentally load a document into an editor instead of a viewer. You can trust someone, but acknowledge that they might make a typo, and decide to not sign the first version of a paper that includes that typo, but them decide to sign the corrected version.

All someone has to do is pick up the phone and call the reference on a CV and say, "So, what did you think of Dr. X's work on Y?"

Swell, now you've got that person on the phone all the time with multiple callers, instead of entering their opinion just once. But tech people are still going to keep offering that person ways to make their life easier and save time, and eventually that person just might decide to take the free lunch.

Most (but not all) of the (brilliant) bio people I've worked with are completely helpless when it comes to technical stuff

Can they click a checkbox? Can they enter a passphrase? If not, then they're not working with computers anyway, so none of this will affect them.

Re:a cumbersome solution in search of a problem

[smallfries]

The hash isn't necessary. If the trust relationship between two academic peers includes "worried about him modify the paper after I review it", there is no trust relationship.

No. You could not be more wrong I'm afraid. When I sign a review of somebodies work I am not recommending / trusting them as a person. I am recommending that single piece of work. People who trust me can then trust that the piece of work is good.

The hash is completely necessary because the trust relationship is slightly more fine grained than you assumed. It is not about trust between academic peers, it is trust in the work of an academic peer.

Very poor idea

[littleghoti]

What is important is *anonymous* peer review. There needs to be a mechanism for new scientists to question established researchers without lasting detriment to their careers. On another note, what I thought this article might be about was CiteULike, which is great. Any academics should check it out

Re:Very poor idea

[TheSunborn]

I might have missed something, but I am pretty sure that most Peer review are anonymous. (The authors of the paper don't know who the reviewer are). The publisher does know, but he keeps it secret.

Re:Very poor idea

[rlseaman]

There have been numerous attempts to redefine peer review to bring it into the 21st century. There will be many more after this effort.

Peer review is typically anonymous. It represents a trust relationship between the editor and the referee, not directly between the author and the reviewer. If the journal - or rather, the editor - is removed from the equation, then some new mechanism is needed. It isn't obvious that the web of trust as described fits the bill, however.

An equivalent to a distributed certificate authority already exists and is widely used as a metric. The only certification that will be believed - even from professional peers - is to demonstrate a need and desire to actually use the results of prior publications. These are denoted (and trusted) by building a chain of publications by tracing back through the references embedded in subsequent publications themselves.

Re:Very poor idea

[Anonymous Coward]

What is important is *anonymous* peer review.

Er, yeah, because AC is so *amazingly* insightful.

Re:Very poor idea

[Anonymous Coward]

They are anonymous--the same way student reviews of a professor in a class with 5 students are anonymous. By the time you're doing anything original, innovative, and remotely interesting in a field, you're going to have 20 peers tops--very likely only 5 or so. Some of your reviewers will be unrelated and able to check basic mathematics for correctness but not much more.

Honestly, I think I haven't yet seen someone receive comments back where they couldn't take a good guess at who the originator was...

Re:Very poor idea

[slawekk]

Peer review is typically anonymous

Apart from a couple of experiments, peer review is never anonymous. It is typically half-anonymous - the author does not know the identity of reviewers, but the reviewers do know who is the author. I don't know in other disciplines but in mathematics this is a necessity from the practical point of view. Active researchers may get more than one paper to review per week. Sometimes one would need a couple of weeks to check all the details in proofs. So the reviewers have to rely on the credibility of the authors and the institutions the authors are affiliated with to make a judgment.

Re:Very poor idea

[rlseaman]

Yes, that is a more precise description. The main point I was making, however, is that the web of trust is currently between the referees and the editors on the one hand, and the editors (forwarding the referee's comments) and the authors on the other. Fairly often referees wave anonymity, but that is their choice, not the editors. As you say, referees have to know the identity of the authors, although this knowledge can be abused.

Re:Very poor idea

[godrik]

In Computer Sciences, there are several journal and conferences where the reviewing process is double blind.

Researchers that got more than one paper per week generally push the review to PhD student. Moreover, most of the time reviews are weighted with a confidence index so that reviewers can tell the editor 'it sounds correct but I did not checked in details'.

Re:Very poor idea

[godrik]

Honestly, I think I haven't yet seen someone receive comments back where they couldn't take a good guess at who the originator was...

Most of the time, I do not know who read my articles and I do not believe the review I wrote tells who I am. There is a bunch of "top-level" researcher in my field (parallel computing) and probably 10 times more "classical researcher" and PhD student. Perhaps it is not true in all field.

Even if your identity can be guessed, it is never entirely sure. And being anonymous prevent pressure such as "you rejected my last article, I will reject your". You won't risk it since you are not sure it is the same guy. And the editor knows you, you will lose credibility to him if you are proven wrong. And being proven wrong can be easy when the authors can answer to reviews.

In short, anonymity is a major property in the scientific peer reviewing process.

Re:Very poor idea

[chthonicdaemon]

Ideally, though, the reviewers wouldn't know who the authors are. This stops well-known researchers from slipping in bum papers and stops people from unknown institutions getting kicked without regard for their work. This would be the review equivalent of a double blind study, and if the goal of the process was to find valid research, it would already be like this. Unfortunately getting published these days is way more about being known and shaking hands than actually doing good work, because you know the guy you're sucking up to now is going to see your name on a paper he's reviewing and that will shift his opinion in your favour.

Re:Very poor idea

[PDAllen]

Depends on what area you're in, I think. I suppose there are some areas where really only a few people work and it's so different to anything else that no-one from outside can easily referee a paper - but if you're in that position, I'm hoping that you're working in a very new and growing subject. The alternative, bluntly, is that you're doing something the rest of the world thinks is boring.

I do combinatorics; if you give me a paper that doesn't use topological or heavy probabilistical methods, I could referee it with a bit of work; that covers about half of combinatorics (i.e. thousands of people). If you cut that back to the bits of combinatorics where I've actually published (so am likely to get referee requests in turn) then you still have a few hundred active researchers.

It's certainly true that sometimes you can guess who wrote a review - but mainly if they have a very distinctive writing style: Nash-Williams was famous for insisting on correctness in English to the point that his referee report could easily end up longer than the paper, for example. Since most referee reports are quite short - one paragraph about whether the result looks correct and is good enough for the journal, and a list of bullet points to change - it's usually hard to guess who wrote it.

Re:Very poor idea

[PDAllen]

That's true but sadly almost impossible to work. People routinely make preprints available as soon as they submit to a journal (a few weeks before it goes to a referee, often) and many people will give talks about their work when they're sure of the correctness of the result - which can be a very long time before it's written up. I gave a bunch of talks last summer on a couple of subjects, neither of which is yet written up; one because we think we can do something more, and the other because all of my coauthors are busy and we're just taking time to get it written down. When these papers go to a referee, the referee will probably have heard me or a coauthor talking about the stuff, and they'll recognise who wrote it whether or not our names are on the paper.

That said, I don't think there's so much discrimination against the guy you've never heard of; or at least, I would not write a bad report unless the paper was bad. What there is, is pressure not to reject a big name even if they are submitting something uninteresting (that should go to a minor journal) or very badly written (where normally I would want to list 20 problems with the first 2 pages then state that the paper must be fully rewritten before I accept it) to a major journal.

Re:Very poor idea

[jmv]

Honestly, I think I haven't yet seen someone receive comments back where they couldn't take a good guess at who the originator was...

Especially when the reviewer's comment says: "you should cite this great work by professor Foo..."

So we create a situation like slashdot.

[jellomizer]

Where popular Ideas get modded up and controversial ideas get modded down.
We still need to find a way to get Ego out of science. Without having every crackpot idea be seriously considered.

Re:So we create a situation like slashdot.

[Lendrick]

In addition, there will be an effect where more prominent scientists will get tons of links and favorable peer reviews, in exchange for being "friended" in this network.

Certainly this effect must exist already, and admittedly a bit of it is good (if someone repeatedly submits excellent papers, it stands to reason that their opinions should hold a bit more weight) but this may amplify the effect far past the point of usefulness. Ultimately, science needs to stand on its own merit, and not just the reputation of the person who published it.

Re:So we create a situation like slashdot.

[NonUniqueNickname]

popular Ideas get modded up and controversial ideas get modded down

Who would have modded up Galileo? No one.
Very bad for science.

Re:So we create a situation like slashdot.

[glwtta]

We still need to find a way to get Ego out of science. Without having every crackpot idea be seriously considered.

I love how people want science to be this pure ethereal thing, for some reason. Of course research is ruled by the egos of those involved, just like every single other human endeavor. Why would you expect it to be an exception?

Re:So we create a situation like slashdot.

[Sloppy]

No. Slashdot, unlike a reputation system based on a PGP WoT, has anonymous moderation. I can't look at how you have moderated, decide whether you've done a good job or not, and then set in my prefs to ignore or use your moderations. If you moderate something as insightful, then the system is going to present it to me as insightful. That is what causes the popular-goes-up, controversial-goes-down problem.

Re:So we create a situation like slashdot.

[iris-n]

No one? So, tell me, how have you heard about Galileo? Found an dusty tome in a shelf of an old monastery, translated it from latin and amazed yourself of how ingenious he was?

He is only known today because his people 'modded him up'. Some of his ideas were controversial, against the good ol' Aristotle, but he was a very respected teacher, that made brilliant insights in various aspects of physicis and mathematics, and only later reached his astounding conclusions. Read his biography.

If some paper is refused for publication, it's because it's plain bad, not controversial. In physics, you have objective criteria of quality, and is by that that they're judged.

It might be cool to imagine the lone crackpot that made revolutionary discoveries that are ignored by the scientific community, but that is just romance. Crackpots are just poor bastards that couldn't even get quantum mechanics right and went nuts. What they say may look cool for laymen but is just plain rubbish.

Re:So we create a situation like slashdot.

[Anonymous Coward]

No. Actually we do need to take every crackpot idea seriously. How many scientific discoveries and breakthroughs that we take for granted today would have been considered crackpot theories at one time? Every single one I'd imagine, provided you go back far enough.

We, today, do not have such a complete understanding of the world that some new hair-brained theory couldn't come along that blows everything we think we know out of the water. Heliocentrism, the theories of relativity, and quantum mechanics come to mind quite readily. In fact, I'll go out on a limb and say that's it's a certainty that we are totally screwed up in our understanding of the world today. We might have some pretty good approximations worked out, but it's only a matter of time before we understand that things are a lot weirder than we know.

People constantly accuse me of being closed-minded when I don't agree with them (which is a pet-peeve I'm really trying not to rant about right now), but the truth is that no matter how ridiculous they sound or how stupid I think they are, I still give their arguments a fair shake. I examine what they're saying, look for flaws, look for merits, and honestly try to sus-out the truth of the matter. Most of the time I end up deciding I'm right (because I've generally given my opinions quite a bit of thought before forming them in the first place), but sometimes I'll find a lot of wisdom coming from places I didn't expect, and would have missed if I hadn't taken the time to give those hair-brained ideas some thought.

The point is, we aren't so all-wise or all-knowing that we can afford to pass judgment without a really careful examination first. Most of those crackpot ideas really will turn out to be crackpot ideas, but it would be a real shame to miss out on an opportunity for greater understanding just cause we we're too arrogant in our own understanding to examine what others have to say.

Re:So we create a situation like slashdot.

[Orion Blastar]

Well we used to have such a system when Monasteries did the peer reviews before the Universities took over. After the Universities took over they ruined and abused the scientific method so that any crackpot idea be seriously considered as long as the scientist with that crackpot idea can get a few of his/her friends to conspire to rubber stamp the paper or create aliases to peer review his/her own work. Something about a system of morals and ethics that were not based on a religion and seriously bent to allow any crackpot idea so that all scientists had a equal right to be published and become notable.

My Astronomy professor published a series of papers that said the Hubble effect was wrong and that the universe is not expanding. He argued with our text books. Somehow his papers were peer reviewed even if it was a crackpot idea.

Getting Ego out of science? Impossible! It is like trying to get God out of religion, or violence out of hockey, or get all smokers to quit smoking, or put responsibility back into the media and news programs and newspapers and force them to verify the facts before they print or publish or report on something.

Re:So we create a situation like slashdot.

[TheTurtlesMoves]

If we took every one of them seriously. We would get zero actual useful work done. Period.

The vast majority can be dismissed in 3 seconds or less because they violate a huge about of experimental data that we already have. Why should I take the time when they won't even take the time to read the current results in the field or to write up the idea/experiments properly?

In God we trust. As for the rest of you, show me the data!

Even the patent office had to require a working prototype for perpetual motion machines in the end.

Already got one

[burris]

Scientific publications already have a web of trust in the list of cites at the bottom. Publications don't get cited unless they are notable in some way.

Re:Already got one

[Anonymous Coward]

This is so not true. You can negatively criticise a publication in your paper and you'll still be citing it. A good thing would be to add new fields to citations, such as "related", "agrees", "disagrees", "partially disagrees", etc. This would allow much richer search mechanisms. Nevertheless, the idea of a web of trust seems like a very good one.

Anonymous reviews are great for those who don't want to put themselves on the line, but posting anonymously (and in general, anonymous publications) will not help in general.

However, anonymous reviews might allow some people to state a negative opinion (or an actual fact) about a publication his/her close colleagues have posted without damaging their relation. I know some people will say that you have to be able to say something negative about your peer's work and live to tell the tale, but we all know that, as corrupt scientific-relations are, this is not always possible and you might end up losing your job if you publicly criticise your boss' work.

Best regards,
Anonymous Coward.

Re:Already got one

[LargeMythicalReptile]

Indeed. Google's PageRank algorithm started off as citation analysis for academic papers--one could find out which papers were notable in a given field by the quantity and notability of the papers citing it. Then they realized that the same approach could work for the Web, treating links as citations.

As a sibling post points out, this says nothing about the correctness of the paper, only its notability--but ideally if a paper is shown to be faulty, then the paper exposing the faults will get many citations too.

The proposed system might give a more detailed granularity than a purely citation-based system, so in that sense might have a reasonable benefit. However, as a "social network" of sorts, it will tend to have a life of its own, and consequently could very easily be subject to failings at the social/political layer (as other commenters have noted).

Re:Already got one

[tucuxi]

When building its databases, Google does not treat all links as equal. You have the 'rel=nofollow' link attribute to indicate that you don't want to attribute trust to the destination of the link, and Google can discount link weight based on the outgoing anchor text.

I would be very interested in seeing that done in scientific publications. As you said, notability can come in many flavors, and they are not equally yummy.

arXiv leads the way

[MoxFulder]

arXiv, the pioneering online preprint archive, already does something like this, though not as sophisticated. They have an endorsement system, wherein more established users endorse newer ones. It's fairly rudimentary and ad-hoc, but seems to keep out crackpots and spam fairly well in practice.

Re:arXiv leads the way

[PDAllen]

Spam yes. Crackpots piped to math.GM for the amusement of all (e.g. the guy whose 'proof' of the Riemann hypothesis was 20 pages of verbiage boiling down to 'the universe is built on maths and maths is built on primes, so they must behave naturally and therefore the result is true..').

Re:arXiv leads the way

[tehgnome]

I feel the arXiv system is a bit weak, but when I RTFAed, this is precisely what I thought of. I would love to see this be implemented with arXiv.

Re:arXiv leads the way

[physburn]

ArXiv endorsement, indeed prevent outsides getting posting, when i wrote my paper on neutrino based dark energy, i couldn't get anyone to endorse me, dispite being on the builtin boards, and posting out to scientist in the field. I guess there were to busy to take a look at my work.

that's true of normal publication too

[Trepidity]

Normal peer review is basically a grouped version of the same thing. A paper being published in a journal tells you that the journal's editors/reviewers thought it was a legitimate contribution. What that tells you depends on what you think of the journal's editors/reviewers. It's basically an endorsement mechanism: publication venue X, meaning set of people Y, says that paper Z is good. This is a somewhat more decentralized version, where individual researchers can say that paper Z is good.

Re:that's true of normal publication too

[orclevegam]

That is essentially the point I was trying to make.

Endorsements stink.

[Anonymous Coward]

Everyone loves you, but then some nut wants to climb the ladder. They rake through your relationships looking for pot-smokers and communists. Suddenly you lose your security clearance to the knowledge in your head about to how to make the atomic bomb that you developed.

Industry Standard

[spydabyte]

If this works, it'll hopefully create an industry standard. Education and Research often lead technological standards, like the Internet, and this sounds like exactly what the entire 'net needs.

Could this be incorporated into article references

[Anonymous Coward]

So if an article is referenced by other articles from trusted sources, the reference is like a trusted "trackback"?

Gettin' yer hands dirty

[mcrbids]

Passing a peer review doesn't provide assurance of the accuracy of a scientific paper. All a peer review does is filter out stuff that we are already pretty sure is bogus.

But what makes a scientific concept meaningful is review by experimentation. Ultimately, it's important for people to drum up experiments that could be used to confirm or reject the theory. And really, your concern is an artifact of the lousy job being done by the public school system in teaching students what the Scientific Method is really all about.

If schools taught what the scientific method actually was, peer review reinforced by experimentation, and the process of learning rather than "teh skyz she is blue because of ze refraction of ze vater vapour", this wouldn't be such a worry...

Faculty of 1000

[xplenumx]

The Web of Trust proposal sounds a lot like the website Faculty of 1000. I also would suggest looking at the website for the Web of Science. If I cite another's work, I've reviewed it and, at least on some level, agree with the findings. I'm certainly not going to cite that I think are bogus.

A more accurate statement

[harlows_monkeys]

PGP and GnuPG have been utilizing webs of trust to establish authenticity without a centralized certificate authority for a while.

A more accurate statement would be "PGP and GnuPG's web of trust system has been mostly ignored pretty much since it was created".

what journals still offer

[dalhamir]

Journals still offer a critical service by at least partially blinding the review process. If I know your reviewing the paper, and you know that I know that you are reviewing my paper then large social pressures can come into play. For this reason, the idea of decentralized publishing seems less appealing to me than modifying the metric by which scientific impact is measured.

The concept can be applied elsewhere

[gznork26]

The idea of having varying levels of endorsement can be used in other contexts as well. For example, when someone offers indirect information, you would be wise to consider how much you trust the person relaying the information when assessing its validity. The more remote the source is, the more you want to suspect the credibility of the report. That's why hearsay evidence is inadmissible in court. But how would this be implemented in daily life? I took a whack at exploring the idea in a short story called "Business Decision". It starts like this...

+ + +

Evan studied the portly man standing in front of the curved dais for a moment before answering.

Jason Sweeney had attended Council meetings before, a silent but imposing presence brooding in the far corner. A curious glance was enough to influence the more convivial constituents in the room, causing them to stay well away lest they become enamored of whatever unsavory business had paid for the custom woven fabrics of his business suit, and led him to wear such uncomfortable-looking shoes. But something was different today. Something had driven him to exchange the shadows at the edge of the room for a brightly lit moment at the center of attention.

"I can offer this Council the means to retire its debts," he had said. "It's just a simple business transaction. What harm can that do?"

+ + +

You can read the whole story here:

klurgsheld.wordpress.com/2007/06/14/short-story-business-decision/

P. Orin Zack

Anonymous Coward

[Anonymous Coward]

I've established careers in aerospace engineering, broadcast engineering, chemical engineering, nuclear engineering, and biophyics, with degrees in physics and more, at where I consider the premier research University. I would hope that any publication is reviewed, and by peers, and that all results are viewed. The issue seems to be by whom; I've never bothered with the peer-review process, to the detriment of my career, but to the benefit of my work. Maybe it's time to bring this together. Physics gets dangerous at times. Some of us must take the risks.

Pointless Application of Social Networking

[solanum]

Firstly, I am a scientist, I have a number of papers published in journals significant to my field (plant biology) and have 200+ citations to date. This scheme is pointless, it amazes me how many stories along the lines of 'we can make scientific publishing work better' that get on Slashdot.

In general the peer review process works extremely well: a journal gets multiple reviews on a submitted manuscript, the authors don't know who they are by, the editor makes a decision based on those reviews, if the authors have a legitimate grievance about a rejected manuscript they are able to make that to the editor, if the manuscript is accepted it is almost always in the form of 'accept on minor revision', whereby the authors have to address the reviewers comments prior to publication. The paper gets published and it sinks or swims. If it is a good paper then it gets citations and is seen as such, if it isn't a good paper it doesn't gets citations and is seen as such. Of course some good papers don't get citations because they are of interest to very few and some bad papers get citations from people refuting it. In the former case, no social networking process is going to help, in the latter everyone knows the reason why it has citations and they don't tend to last for long anyway.

Adding a social networking layer over the top will do nothing, simply show which researcher has the most mates. In general, scientists don't care whether a paper is by someone known to them or not, though of course a reputation will colour the way you approach a paper.

The main problem scientific publishing has is reconciling the increasing desire for open publishing with the need to maintain some funding, without publishing everything that comes across their desks willy nilly. There is no prblem of trust.

Re:Pointless Application of Social Networking

[tucuxi]

it amazes me how many stories along the lines of 'we can make scientific publishing work better' that get on Slashdot.

So, in your humble opinion, science publishing should continue essentially as it is, perhaps with a shift towards open publishing. And the current 'science accounting' methods (adding up citation sum-totals, relying on journal weights, or combining both) are perfectly adequate. And there the pressure from these flawless accounting systems does not drive scientists to publish un-matured, partial results in a frenzy to score more papers than their peers.

No, I do not have a silver bullet, and yes, the system sort-of, kind-of works. But I am sure that it can be improved. Funding depends very heavily on accounting.

A social network, if divorced from editors and anonymous reviewers, may not be so bad as you paint it. I, for one, would not mind to see rated and commented papers...

Re:Pointless Application of Social Networking

[Anonymous Coward]

As a scientist myself, I have to disagree. The peer-review process is very susceptible to manipulation by groups of upward mobile researchers. They cite each other's works to build up citation lists without necessarily having a strong scientific leg to stand on.

That said, the problem with peer review is the people ... social networking will only make it worse.

just adds more politics

[Anonymous Coward]

This will just add more politics to the already political process of peer-review. People will set up deals - you sign mine and I sign yours - probably as part of a collection.

Re:GNAA

[A nonymous Coward]

There, fixed that for you.

There, fixed that for you. Grammer nazi's suck!

Credibility and Noobs

[Hordeking]

Great. So now those new scientists with few/none papers or unpopular theories are going to have to fight even harder to establish credibility and get published.

Every rose has it's thorn.

This is a fallacy

[Orion Blastar]

sure it will show who signed it.

What is to stop one scientist to create an "alias" with a different PGP/GPG key and publish to a different scientific journal and then use that "alias" or series of aliases to peer review his/her own work. Ward Churchill did this with his works, and I have known many who published articles under aliases in the scientific community as well.

A key sign is lack of such things as margin of error calculation to show that the data was randomly selected and not "cherry picked" to prove the hypothesis as true. Does it strike anyone as odd that many such scientifc papers get peer reviewed and skip such steps by the people doing the peer reviews just like the original scientist did?

The days of the margin of error is too hard to calculate is a poor excuse when we have spreadsheets with that function built in or statistic software that can crunch the numbers for the scientist.

This does not improve credibility, but instead improves fraud and the opportunity for fraud. People will be more easily mislead by someone who peer reviews their own works using several aliases because each alias has a trusted PGP/GPG key that signed it. Thinking that third parties and different people peer reviewed the work, when in reality it was the same person or a club of scientists who peer review each other's work like a rubber stamp as some sort of conspiracy.

Re:GNAA

[palegray.net]

There, fixed that for you. Grammer Nazis suck!

There, fixed that for you. You somehow managed to make yourself look like an idiot and invoke Godwin's (note the possessive case) Law simultaneously. I'm impressed.

Re:GNAA

[Anonymous Coward]

It's Grammar. Not Grammer. LOLZ at you.

Woulditbe too crass to tag this story circlejerk?

[epine]

Isn't "web of trust" in the same synosphere as Greenspan's failed notion of counter party surveillance? Wasn't it a "web of trust" which allowed the Catholic church to conceal deeply entrenched violations of trust while delaying its apology to Galileo for 400 years? Wasn't "web of trust" what allowed Madoff to dig a $50b crater? What percentage of novel endorsements from one genre author to another come equipped with a set of kneepads?

Why is it that so many people are allured by this concept?

The big problem here

[chthonicdaemon]

From the gpeerreview page: "If the author likes the review, he/she will include it with his/her list of published works."

I honestly believe that the future of scientific publication is in a system where everyone publishes whatever they like and generate a hash of the article (or register some kind of unique ID). People then review these articles when they read them, with researchers in the field having unique IDs of their own. The score of the paper then works using something like the pagerank eigeinvector approach. Someone whos own work is rated highly adds more to your paper's score than a nobody. This allows people to post negative reviews (not just the positive ones you like).

For extra credit, there should be a window during which the paper and reviewers are anonymous, allowing the paper's merit to be assessed objectively.

Re:GNAA

[A nonymous Coward]

Whoosh!

GPG?

[Beetle B.]

Use GPeerReview to sign the review. (It will add a hash of the paper to your review, then it will use GPG to digitally sign the review.)

Here's where everything will fall apart. When almost all faculty members I know (except the math and some CS ones) act like this, I can hardly see how they won't bungle it up.

Getting back to his Why's:

Peer reviews give credibility to an author's work.

We already have it.

Journals and conferences can use this tool to indicate acceptance of a paper.

Bad idea. This will easily devolve into a numbers game. Paper X has 20 signatures approving it, with 5 of them at Level Zen. Paper Y has only 10 signatures approving it, with most being at Level Neophyte. We'll take X and reject Y.

Think I'm exaggerating? Go observe people talk about impact factors. In some disciplines, they've begun to take this quite seriously when hiring ("Sure he had 8 papers during his PhD - but all published in papers with IF less than 4. Reject!").

Researchers can also give credibility to each other by reviewing each others' works.

I can see this being useful. If there's a central repository where people can submit and sign their reviews for the world to see, it could be great. Realistically, though, faculty members won't do it without incentives - their lives are busy enough.

Besides, a lot of academia is back scratching. Friends will give positive reviews frequently. Enemies will trash it. There is a reason current peer review is anonymous (at least one way).

This enables researchers to publish first, and review later.

Eh? You mean just to get feedback? Isn't that what Arxiv.org is for?

It meshes seamlessly with existing publication venues. Even the credibility of works that have already been published can be enhanced by obtaining additional peer reviews.

Same complaint as above. "Candidate A has a number of publications in good journals, but candidate B has more online hashed reviews. B wins!". Unless a mechanism can come up where B won't bribe people to provide friendly reviews, this will fall apart. Academics are already over aggressive about getting citations, and this will just be the next stage.

I know I'm cynical, but I advocate less reliance on numerics in judging quality than there is in the current system. Not more.

The good, the bad, and the ugly.

[Randy Savage]

The good: A. This reduces the key man risk of getting rejected by a single biased reviewer. That is really, really annoying. B. Studies that show no statistically signficant effect are often left out of the literature. Having a more comprehensive database could improve meta-analyses (studies of lots of studies). The bad: A. You probably cannot publish to a good journal with copyright and also to this system. B. Cheating, but let's assume cheating can be minimised. C. I cannot envision how this would look on my academic cv. "So, everybody can publish on this?" "Erm, but I ranked highly..." Is that really going to fly? The ugly: Democratising science? Never gonna work.

Number of citations received is far from ideal

[tucuxi]

[the number of references] already exists and is widely used as a metric

Citing a paper does not necessarily mean endorsement for its contents. Only that the paper you are referencing was relevant to a part of your discussion. References can be used to provide counter-examples or denounce bad research; but they are counted as a citation anyway. In scientific citation number-crunching, any publicity is good publicity.

A crazy idea would be to add metadata to references, describing the type and relative importance of the source. That would make 'paper A, main inspiration, very important' actually count more than 'paper B, cited in an off-hand comment to exemplify bad research in this field'. The crazy part is changing the established format of scientific papers to accomodate this metadata, standarizing the metadata, and convincing authors to adhere to standards and editors to enforce them.

Re:Number of citations received is far from ideal

[rlseaman]

Well, the real mark of success is when the paper is recognized merely by the cite, e.g., "Shannon, 1948".

If significant metadata have to be added to whatever the web of trust technology, one suspects a successful result will more closely resemble slashdot, rather than a formal ontology. It's hard enough to get the computer scientists who are building a knowledgebase as a step toward writing their own paper to enter information into the required ontology :-)

Either a root, or critical mass

[gwolf]

If this gets enough traction/critical mass, this problem will be solved - Crackpots can mutually sign as being topz 1337, but they will be inevitably signed as being below the mark for most other, serious (and better punctuated) users.
The site linked describes the process in too low detail to make sense of what they are proposing, but I have thought through a very similar process with some colelagues of mine - This problem is solved when not only each article, but each reviewer gets scored according to both their contributions' (i.e. their articles') and their reviews' value - in a way possibly akin to /.'s metamoderation. And, of course, a very well reviewed article should raise your "karma" more than a great review.
That can even lead the system towards being interesting for some people usually regarded as semi-sacred inside their academic communities: Anybody can submit a paper for review. The paper will be initially ranked at a value based on the author's karma - and will be presented for review initially to people at this person's own level. If the paper gets "modded up" (again, using /. terminology), it will enter the radar of possible things to review for higher-ranked people.
The influence can go on a semi-logarithmic scale - If my /. karma is 2, then each review I make can weigh the same as (say) 3 people with karma 1. 3 reviews of people in my rank will be equivalent to one review from a higher rank.
Yes, I am describing a system somewhat different than the linked article's, but still, it might be an interesting system - And it does solve (at least to a certain degree) the problem you describe - of course, after playing a bit with the factors.
And, yes, 100 crackpot buddies can still push the paper up quite a bit. Still, a couple of well-ranked people will sink it back.

Re:Woulditbe too crass to tag this story circlejer

[skeeto]

From a technical perspective, being decentralized is a major feature for a system. Decentralized systems are generally robust, reliable, and scalable. There is no single point of failure, and the system will be much more resistant to attacks than a centralized system. The cost of the system is also shared by its nodes, so there is no expensive central point, like SSL authorities.

From a human perspective, a web of trust is very similar to human social behavior. For most people, there are a handful of people we absolutely trust. They might be parents, family, spouse, or close friends. This is how the web of trust works in PGP, but you only need to trust them as far as being competent with PGP (properly checking and exchanging keys, etc). The web of trust maps perfectly to our normal behavior and mindset.

Yes, there are trade-offs. For example, centralized systems, at certain scales, will be more efficient than decentralized ones. Each will be vulnerable to different kinds of attacks, like the ones you mention. However, many people think the trade-off favors the web of trust over the alternative.

Double-blind means _doubly_ anonymous

[gwolf]

Usually, when you review an article, it is under the double-blind method - You don't know either who the reviewee is. You must judge based only on the merits of the paper, without taking care on whether the author was a student of Einstein himself.

Which is also a reason to expand this

[gwolf]

If peer-review is done only at the top of each discipline, you might know who reviewed you. However, people write at all levels of the academic scale. Having a broad peer review process, where even people from a different specialization level than yours, can comment (and the opinions are weighed, of course, according to their respective ranking) can further improve this. Sometimes even lead you to understanding several points you didn't even consider originally, as they are outside your usual radar.

Re:GNAA

[palegray.net]

All this whooshing is really blowing my mind.

This keeps the essential problems of "peer review"

[Yoram Gat]

The cure for the "peer review" malaise has to be very different: The self review system.

NAS Testimony to the energy committe

[TapeCutter]

NAS testimony to the energy commitee (pdf).
NAS climate homepage
As you say "other people following this discussion might be curious", even if you are not.