Slashdot Mirror

Bennett Haselton writes "The U.S. government recently announced that academic papers on federally-funded research should become freely available online within one year of publication in a journal. But the real question is why academics don't simply publish most papers freely anyway. If the problem is that traditional journals have a monopoly on the kind of prestige that can only be conferred by having your paper appear in their hallowed pages, that monopoly can easily be broken, because there's no reason why open-access journals can't confer the same imprimatur of quality." Read on for the rest of Bennett's thoughts on the great free-access debate.

Around the time of the tragic suicide of Aaron Swartz, who lobbied tirelessly for free access to academic articles (in his sometimes grey-hat manner, which ultimately got him in trouble), I admitted to some friends that I didn't understand how this became a problem. Why aren't all journal articles free, all the time?

I don't mean that I didn't know why the journal publishers charged exorbitant fees for their subscriptions. If academic researchers have to have access to journal articles in order to do their jobs, then you can expect the journals to gouge academic libraries on the prices. What I didn't understand was: Why do academics even publish in journals that demand exclusive publishing rights for their work, and then charge readers huge fees to read it?

Well actually, we know the answer to that too: academics want the prestige of publishing in big-name journals that have established reputations, and as a result, those well-known journals are in a position to dictate the terms of the contract. A professor might genuinely want to publish their paper in a journal where it can be read for free by all, but they can hardly be blamed for thinking of their own career path first.

Here's the question I really wanted answered: If "prestige" only exists in the minds of other academics within a field, then why don't the academics within a given field just agree to confer "prestige" on papers published in open-access journals, if they can see for themselves that the quality is equivalent to what would be published in the old-guard journals that charge an arm and a leg? And then make hiring, promotion, and tenure decisions accordingly?

I don't mean that the papers published in an open-access journal would bypass the peer-review process, and that everyone in the field would have to judge the papers for themselves without any prior certification of their quality. One of the points that Peter Suber makes repeatedly in his book Open Access is that open access is not about skipping peer review and dumping papers directly onto the web. Rather, the process would work similarly to peer review for a traditional journal:

1. Author submits a paper to journal XYZ.

2. Journal XYZ selects one or more peer reviewers from among their list of people they consider qualified to review the paper. The peer reviewers send back their usual suggestions and some consensus is reached as to whether or not to publish.

3. If Journal XYZ publishes the paper, then they have certified that the paper passed the quality controls in step #2, and the author can now legitimately claim that they had a paper published in Journal XYZ.

4. If people in the field know that Journal XYZ is not skimping on the quality controls in step #2 — that Journal XYZ is sending the papers to the same academics who would do peer review for one of the old-guard journals, and who are holding the papers to the same standard — then they should respect the paper just as much as if it were published in a traditional journal. If a person has never heard of Journal XYZ, then it should only take a minute to explain to them how it works (and crucially, that Journal XYZ is just as strict about quality as the old-guard journals that everybody has heard of).

Each step in this process should cost the journal virtually nothing. The "hard cost," the part that consumes the time of people with unique skills, is the peer review step, but peer reviewers are usually paid by universities and consider peer review for academic journals to be part of their job description. At a minimum, all the editors really have to do is maintain the list of people they consider qualified to do the peer review, and send the submitted papers off to them.

Moreover, the entire process should be fast. Again, the "hard cost" in time is the peer review, but there's no reason that the delays between submission and publication should be in the range of months or years.

(I'm assuming that the article authors would want their writings to be widely read, or at least would not be opposed to it. That may not be the case if, for example, the authors were commissioned by a pharmaceutical company for a study that cast their drug in a favorable light, but the authors realize that their research methods contained errors and want to minimize the number of eyes on their paper, to reduce the chances of their chicanery being caught. Ben Goldacre's Bad Pharma documents these types of problems very thoroughly, but I'm sidestepping that issue for now.)

So, with that in mind as the ideal, I asked my friends, including many current and former academics, why this essentially wasn't the model that was used. Several mentioned the Public Library of Science, which publishes all articles in its journals under a Creative Commons Attribution License (free for anyone to read and reproduce in full, as long as the original author is cited), and finances its operations through publication fees. These fees are in the $2,000-$3,000 range, heavily discounted for low-income countries and authors, and in any case most academic authors pay the fees out of their research grants and not out of their own pockets. That sounded much better than the traditional model, I thought, but I still didn't understand why the costs weren't even closer to zero. Another friend pointed out that PLOS costs cover the expenses for many of their other activities — which are all noble goals, to be sure, but at the same time, why isn't anybody operating a more bare-bones model which minimizes all expenses, and charges almost nothing for publication or subscription?

This, it turns out, appears to be the approach of the PeerJ project, which aims to let authors pay a one-time fee of $99 at article submission time for the right to publish one article per year — or, if you prefer to pay only if your article is accepted for publication, you can pay $129 "on acceptance" (explained here). And the author of the Techdirt piece mentions that he submitted a paper which was published in the inaugural edition of one of PeerJ's journals, 10 weeks after the submission date. This is cheap and fast enough that I'd call it a validation of the theoretical model which predicts the whole process should be able to be done for almost no cost in almost no time. In other words, I think PeerJ will succeed, but even if it does fail, it will only be because of some anomalous business snafu, not because the hard costs of the service they're providing are greater than the dirt-cheap price they're charging for it. If for any reason PeerJ doesn't happen to get it right the first time, they or some other company should keep trying until someone makes it work.

The basic algorithm at work here — taking a piece of content, submitting it to one or more suitably qualified reviewers, and then certifying the content based on the feedback of the reviewers — is something I've advocated in many contexts over the years, for many different types of problems. In one article I argued that we could make success in the music industry into much more of a meritocracy, with far less arbitrariness in determining who succeeds and fails, if a suitably popular site like Pandora simply took new submissions from artists, had the content "rated" by a random sample of listeners interested in that type of music, and if enough of them liked it, push the content out to all of the fans of that genre. In "Crowdsourcing the Censors" I suggested that Facebook's complaint review process should use the same principle: If a given page received enough complaints, have the page contents reviewed by a random subset of Facebook users who had signed up to be "abusive content" reviewers, and then only flag the page for removal if a high enough percentage of those users voted that the page had indeed violated Facebook's guidelines. This year I argued that "We The People", the White House's online petition-drive-organizing website, should rate ideas based on what a random subset of users think of each idea, rather than allowing users to organize mobs of their friends and followers to vote their own ideas to the top of the pile (which, in case you missed it, is how 4chan gave us this). Or, if you think the general public is not qualified to rate ideas according to how they should be prioritized by the White House (and I'd be inclined to agree), you could have the ideas rated by a random subset of, say, the nation's economics professors.

Of course, I haven't heard of any plans to implement this algorithm in any of those contexts. Not that I expected the key power players to be reading my articles, but it's a little surprising that none of them ever came up with this idea independently, either. (To this day, the only website I'm aware of that ever implemented random-sample voting correctly, was HotOrNot.com, where users could rate members' pictures by attractiveness — but each picture's rating was determined by showing it to a random subset of the site's visitors. That system is gone, since the site has made itself over into a date-finding service.)

But academia in general, and science specifically, is different from other arenas in a number of key ways which could help this algorithm succeed:

• Academia, uniquely, is comprised of many professionals whose love of knowledge and intellectual inquiry, is greater than their desire for money. That's not to say that I don't think the same algorithm could work just as well in a business like the music industry, where most of the stakeholders are in it for the money. But even if Pandora did successfully implement the algorithm, it would meet a lot of resistance from entrenched interests in the music industry, who make their money by finding and promoting and managing talent and would not be happy about a new system that threatened to make them irrelevant. In academia, by contrast, it's quite plausible that even the "entrenched interests" — the people who had become superstars under the old system — would see the new system's great potential for disseminating free knowledge, and would welcome it even if it gave scrappy new upstart academics a chance to dethrone them. Not everybody in academia loves knowledge more than they love their own prestige, but I know more people like that in academia than anywhere else.

• In academia, even among people who do care primarily about their own prestige, many of them have tenure and guaranteed job security, a situation that does not exist in most other industries. This gives them the freedom to experiment with new models, such as submitting papers to upstart PeerJ journals. But more importantly for our purposes, it means they can announce that in their department's hiring and promotion decisions, they will count PeerJ-published papers as legitimate professional accomplishments, for the benefit of non-tenured faculty members who do have to worry about their resume.

• Academics, particularly in maths and sciences, are more prone to the kind of thinking that would lead a person naturally in the direction of the kind of system that PeerJ embodies. First, think of a theoretical model (like the kind I described near the beginning of the article). This model predicts that, ideally, it should be possible to publish papers at very low cost with quick turnaround times, without sacrificing peer-review quality assurance. Now, try to approximate that model as closely as possible in the real world. (In most other industries that I've worked in, there's much more inertia around the existing way of doing things, and far less willingness to entertain any discussion about whether a theoretical model can show how we could accomplish the same thing with vastly less overhead.)

And that, in the end, is the real reason journal articles should be free. Not because the U.S. government is making it a condition for taxpayer-funded research, although that is a welcome development. But because there's no part of the process that should cost very much to begin with, if article authors and peer reviewers are already being paid by their employers. The last piece of the puzzle is that enough academics and faculty departments have to agree to confer "prestige" on articles published in open-access journals, equivalent to the level of prestige that they would accord for an article published in a traditional journal of the same quality. If they won't do that, then the old-guard journals will maintain their monopoly on conferring "prestige", and don't be surprised if journal prices keep growing to the point where even Harvard can't pay for them.

Frequent contributor Bennett Haselton writes this week with his explanation of how an improved algorithm on the White House's petition-creation site could do away with Death Star petitions and even improve on the existing serious ones. Read on below for his modest proposal on that front.

With a little boost from 4chan, a petition for the U.S. government to build a working Death Star has reached 30,000 signatures and counting, over on the White House's Department Of Let's See How Fast We Can Get 75,000 Signatures To Legalize Pot (or as it's officially known, "We The People"). This is the website where any of the member of the public can create a petition that other users can sign, and if the petition receives 25,000 signatures in 30 days, the White House will issue an official response. (Alan Boyle is taking suggestions on how the White House should respond to the Death Star request. How about: "4chan. You will never find a more wretched hive of scum and villainy.")

Cynics will say that the whole process was already a joke anyway. Even looking at the most popular non-Death-Star related petitions on We The People, most of them express standard left- or right-wing positions on hot-button topics in a manner that's extremely unlikely to convert anyone who doesn't already agree. Since everyone already knows that those some large segment of the population holds those positions, nobody would be surprised that any one of those petitions would be able to gather 25,000 signatures, and so there would be no pressure on the White House to change any of their official positions as a result.

On the other hand, I don't think this means that online petitioning can't work. Rather, I think a sligthly different algorithm could greatly improve the quality of the suggestions that get filtered to the top and trigger a response from the White House. At least one algorithm exists that (a) would prevent the system from being "gamed" by any large, organized group (whether 4chan or the NRA); and (b) would reward the petitions that were supported by the highest percentage of the general user population (or, if you prefer, the petitions that were supported by the highest percentage of credentialed experts in a given field).

The algorithm is the same one that I've advocated for preventing cheating on digg, or identifying the best "hidden gems" among newly released songs (and political arguments), or adjudicating Facebook abuse complaints -- have each petition voted on by a random subset of users registered on the We The People site. Based on this random sampling method, the petitions that have the highest percentage of "yes" votes, are assumed to be the ones with the broadest level of support among registered users, and the ones most deserving of a response from the White House.

Example: Suppose there are 250,000 registered users on the We The People site. A user creates a new petition, and somehow manages to pass some "threshold" that is implemented to screen out blatant time-wasters. (Perhaps you have to gain 100 signatures to pass the first threshold. I'd prefer it if you could clear the first hurdle just by paying $5 with a credit card, but this might anger purists who say that petitioning the government should always be free.) Your petition then gets emailed out to 100 randomly selected other users on the site, who vote to either Agree or Disagree. (In practice, in order to get 100 votes cast, you'd have to email more than 100 people, taking into account their response rate. So if only 50% of users respond to an email request for votes, email it to 200 randomly selected users to ensure you get about 100 votes cast.) Then petitions are sorted according to the percentage of users in their sample who voted to Agree. Petitions that got a high percentage of yes-votes, could be forwarded out to a wider audience (say, 1,000 users), to ensure that the initial high percentages of yes-votes wasn't just a fluke. Users in each random sample could also include comments about why they were voting a particular proposal up or down.

This sounds deceptively simple, but it makes it much harder for an organized online movement to hack the system. Say that 4chan manages to get 25,000 registered users in an attempt to push their favored petition to the top. This still means that, on average, their voters will comprise only about 10% of the randomly selected voters in any online poll - possibly enough to give an extra boost to a petition that already had broad support from regular users, but not enough to achieve a coup all by themselves.

Perhaps you'd object that even if such a system could not be manipulated by organized mobs, it would still leave the approval rating in the hands of non-expert ordinary citizens (even if citizens registered on We The People are slightly more informed than average). Whether you think this is a good thing, depends on whether you think the purpose of the site is to reflect the will of the people, or to provide informed advice to the President.

But if you want to get a random sampling of expert opinions, that's pretty easy as well. For petitions on, say, economic matters, just have a subset of users consisting of economics professors from accredited universities across the country. (These credentials would have to be confirmed manually by White House staff, but it's not that hard to verify that someone owns an .edu address and that their university webpage identifies them as an econ professor.) Then any petition on an economic matter could be submitted to a random sample of economics professors to be rated by them. If a petition gets a rating from economics experts that is wildly different from the rating it gets from the general user population, that suggests something interesting is going on (either econ professors are out of touch, or the general public is misinformed). But if a petition gets high levels of support from the public and the relevant expert group, that would seem to justify a response from the White House, much more so than some of the idiotic petitions currently pulling 65,000+ votes on We The People.

Something almost like this has actually been done by the IGM Economic Experts Panel in Chicago, which surveyed a group of 41 economists that the IGM believed to be among the best in the world, representative of the political left, right, and center. The survey found a high degree of consensus on questions that the general public is divided on, such as the fact that 40 out of 41 experts agreed with the statement:

All else equal, permanently raising the federal marginal tax rate on ordinary income by 1 percentage point for those in the top (i.e., currently 35%) tax bracket would increase federal tax revenue over the next 10 years.

To people who have heard celebrity conservative economists claiming that raising marginal tax rates lowers tax revenue, it might come as a surprise that virtually all expert economists in the IGM's sample, including a representative number of self-described conservatives, agreed that it does not. But don't just soak the rich and call it a day; most economists in the IGM's sample also disagreed that:

The cumulative budget shortfalls in the US over the next 10 years can be reduced by half (or more) purely by increasing the federal marginal tax rate on ordinary income for those in the top tax bracket.

Of course those were questions of fact (what economists call positive economics), while petitions address questions of what should be done (what economists call normative economics, and which varies according to your values and goals). But even economists with diverse political leanings often advocate similar policies; NPR interviewed 5 economists spanning the spectrum from left to right, and found across-the-board consensus in favor of 6 proposals, which you can read here. And hey, one of them is legalizing pot!

If We The People implements a system for polling a random sample of economics experts, I think their first order of business should be to have them rate the ideas in that 6-point platform. The five-person panel claimed that all of these ideas have broad support from economists across the political spectrum, but it would be good to know for sure. And for any of those six points that has broad consensus support from experts, it should be incumbent on the White House to declare whether they agree, and if not, why not.

More generally, random-sample voting will always reveal more useful information -- whether about the opinion of the public, or about the opinions of experts -- than a petition site that lets passionate users self-organize into signature mobs. As I've been saying ever since my first story advocating this algorithm, the only site I'm aware of that currently implements random-sample voting correctly, is HotOrNot, which shows users a random series of pictures and lets users rate the picture's hotness on a scale of 1 to 10. Can we not make at least that much effort to design a working system, when it comes to deciding which petitions get a response from the White House?

Slashdot regular Bennett Haselton writes in with an essay on a subject we've dealt with internally at Slashdot for years: user abuses of social news... this time at Digg. He starts "Alternet uncovers evidence of a 'bury brigade' coordinating efforts to 'bury' left-leaning stories on Digg. Digg had previously announced that the 'bury' button will be removed from the next version of their site, to prevent these types of abuses, but that won't fix the real underlying issue — you can show mathematically that artificially promoting stories is just as harmful in the long run. Here's a simple fix that would address the real problem."

Even if you just arrived from Mars and have never heard of Digg, that description of the service should make it obvious how easy it is to game the system, by rounding up groups of friends to vote on stories that you want to promote, or to bury stories that you want to kill. The former type of abuse (and it is abuse, under Digg's Terms of Use; search for "organized effort") is far more common, since people usually have more incentive (commercial or otherwise) to promote their own work than to bury someone else's. And in fact, Digg has announced that the next version of the service will remove the "bury" button, replacing it with a "Report" button for reporting bona fide cases of abuse, not just to bury boring stories.

The thinking seems to be that abusive "digging" to promote a story, is less harmful than abusive "burying", and this has the ring of plausibility — that a creative effort is better than a destructive one. After all, Alternet had previously highlighted several artificial right-wing "digg brigades" mentioned in their story (Diggs And Buries, theliberalheretic, etc.), but they didn't blow the lid off of the situation until their report on the Digg Patriots bury brigade, as if to say, "Now we've found something really scandalous!" Annalee Newitz cheekily reported on how she bought votes to boost a story to the front page of Digg, but probably would have felt guilty if she'd hired a service to bury someone else's story. And when a Digg user organized an effort to bury Ron Paul stories that he thought were "spamming" the system, Ron Paul supporters protested that they were merely organizing to vote up stories they agreed with — the clear implication being that this was more honorable than organizing to vote stories down.

But this, I think, is a fallacy. If a story's ranking is artificially inflated, then the extra eyeballs for that story have to come from somewhere, and they come from users paying less attention to the other stories that the phony up-and-comer pushed out of the way. Artificially bumping a story up is just as harmful as artificially burying a story, but the harm is distributed among many innocent victims, not just one. (By the same reasoning, in fact, you could argue that burying a story does no net harm to other users of the Digg site, because the harm done to one story is cancelled out by the benefit to all the other stories that rise in prominence when the victimized story is pushed out of the way. So by strict economic logic, recruiting friends to boost your own story at the expense of everyone else's, is actually more harmful than organizing a bury brigade!)

So I don't think that Digg's replacing the "bury" button with a "report" button will fix the problem. For one thing, obviously groups could abuse the "report" button in the same way — issuing calls to action to report a story for violating the TOU. Since a flurry of bona fide abuse reports is presumably what Digg uses to identify and remove truly abusive stories like MLM spam, how are they going to tell the difference between these cases and cases of abusive "reporting"? (My suggestion: See if there is a sudden change in the percentage of users who view a story and make an abuse report. For stories that are genuine TOU violations, the percentage of users who "report" it should remain steady; for stories that are victimized by a "report brigade," you'll see a sudden spike in viewers and in the percentage of those viewers who report the story for abuse. This might have worked for detecting and stopping the bury brigades as well, although we'll never know now.)

But more fundamentally, even if this change does stop the "bury/report brigades" from killing stories at will, that only fixes the most obvious symptom of the underlying problem, which is that the system can be gamed by recruiting your friends to vote either way. It won't stop "brigades" from artificially promoting shallow stories that agree with their opinions, which does the same net harm overall.

Indeed, the most long-term harm that the DiggPatriots Yahoo Group might have done is that their cheating was so egregious that it makes other examples of cheating look benign by comparison, and might prevent people from realizing that "benign cheating" is just as harmful. As detailed in the Alternet report, the DiggPatriots group talked openly about cycling through different Digg accounts and circumventing bans on their IP addresses. The welcome message to the Yahoo Group told new users that the group was operating "under the radar." The group leader, a woman with the handle "bettverboten," talked about how to prevent Digg from monitoring their actions. And of course the vast majority of posts were calls to bury stories. But what if all of that had been inverted? If the group had operated in the open, while still focusing on recruiting conservative members? If each user limited to themselves to only one Digg account like they were supposed to? And if they focused not on burying stories, but on digging stories that promoted their viewpoints? Just as bad. It just doesn't sound as bad.

I still think the only way to make Digg a true meritocracy, would be to use some version of an algorithm I outlined in an earlier article, inauspiciously titled "How to Stop Digg-cheating, Forever." The gist of it is that in addition to collecting votes from friends, stories should be shown to a random subset of users on the site (perhaps in a box that occasionally appears at the top of the screen when they're logged in), who are asked to vote it up or down. The votes of a random sampling of users would be more representative of how much value the story would have to the Digg community as a whole. Even if most users who are asked to vote on a "random story" simply ignore the request, all you need is to show the story to a large enough sample that you can measure the difference in responses to a truly good story vs. one that has been promoted by digg-cheaters. You don't necessarily have to run this procedure for every story, only the ones that are about to gain some benefit from a large number of diggs (such as being pushed to the front page), and you need to decide whether the story really deserves that big boost. The only way to game that system would be to organize a group of dedicated Digg users so enormous that they constituted a significant percentage of all users on the system — something pretty hard to do without getting caught.

Still, the only site that I know of, that uses a version of this "random sampling" algorithm is HotOrNot.com, which lets you recruit your friends to vote on the "hotness" of your picture on a scale of 1 to 10 (by sending them a link to that specific picture), but also shows a stream of random pictures to visitors, so that your picture can collect votes from strangers. If the votes from the users who visit your picture via the link are significantly different from the votes from users who see your picture via the random stream, then HotOrNot discounts the votes from users who view your page via the link. This prevents digg-style gaming from people who want all their friends to give them a 10. (Note that if you think about it, this is essentially the same as always throwing out the votes from people who visit your picture via the link. If you collect votes from group A and B, but you only count the votes from group A if they agree with the votes from group B, then you're really only counting votes from group B! All the extra votes really give you is the ability to brag that X many people voted on your picture.)

This seems like the simplest way to prevent Digg-cheating, although there may be others. Still unresolved is how to solve the general problem of "gaming" in traditional media and the blogosphere. For the foreseeable future, it's going to be the simple truth that if a major media outlet wants to run a story, it will be heard, and if no media outlet wants to run it, it won't be heard, regardless of how many viewers or readers would have voted in some hypothetical poll that, yes, they want to read that story, and yes, they liked it afterward. That's true for Internet articles as well, except to the extent that a deserving article might be rescued from obscurity by Digg, but the more that system can be gamed, the less it will reward articles that really deserve it. Digg is gameable because power users can recruit votes from their friends; the media and the blogosphere are so obviously "gameable" that we don't even call it "gameable," because "power users" — media outlets and A-list bloggers — can run whatever they want. Right now, the only way I can think of to change this situation that is even logically possible, would be for a site like Digg to adopt some version of the random-sampling algorithm, and to continue growing in power until a significant percentage of the public (not just Internet users, but everybody) relied on it for information. Then, if you had something important to say, people would hear it, but you wouldn't be able to cheat your way to the top.

The ultimate irony is that Alternet's story may never have seen the light of day, if it hadn't been the beneficiary of the same gameable, non-meritocratic inefficiencies that exist in the media-blogo-outrage-o-sphere, just as they exist on Digg. Yes, the Alternet story deserved to be heard, but you don't get the publicity you deserve, you get the publicity that you organize, and Alternet had the organizational publicity structure in place to get their voice heard. If a kid blogging from his bedroom had infiltrated the Digg Patriots group and made essentially the same discovery, would anybody ever have heard about it? (Well, maybe, because of the political hot-button factor — but even then, only after the story had been picked up by a major site like Alternet.) A truly meritocratic Digg algorithm could make it possible to get a good story out without a lot of organizational support behind it — and to ensure that an organized effort can't kill a good story either.

theodp writes "In its just-disclosed patent application for the Online Personal Appearance Advisor, Microsoft describes the 'invention' of its three Microsoft Research employees in these words: 'The contributor uploads self images for viewing and rating (or voting) by viewers who choose provide an opinion on different fashion and/or cosmetic looks of the contributor.' So what do you think — is Microsoft's invention really Hot or Not?"

Frequent Slashdot contributor Bennett Haselton sends in his latest, which begins like this and continues behind the link. "When Google's VP of Engineering announced their proposed Knol project, where users can submit articles on different subjects and share in the AdSense revenue from the article pages, he didn't mention "Wikipedia," but practically everyone else did who blogged about it. Here's what I think will happen, if Knol is implemented according to the plan: Even though it won't technically be a "Wikipedia fork," it will quickly become equivalent to one, with a "gold rush" of users copying content from Wikipedia to Knol articles hoping for a piece of the AdSense dollars. But I submit this will be a good thing, especially if bona fide experts in different fields join the gold rush as well and start signing their names to articles that they've vetted."
First, I've been saying for a while that someone should fork Wikipedia and start assigning "ownership" of articles to credentialed experts where possible, so that an article can be cited as a source that has been vetted by a recognized individual, and to guard against vandalism. Citizendium does something like this, but started from the ground up rather than fork Wikipedia. I argued that they should fork as much as possible from Wikipedia (having experts "bless" the content in the process); the project's official reason for not doing this was that authors are more motivated when starting with a clean slate than when taking over someone else's article. True, we all know the energizing feeling of a clean slate compared to the sluggish feeling of taking over a 50%-completed project with all of its flaws and compromises, but the "energizing feeling" often doesn't make up for the advantages of having 50% of the work already done for you (which is, in a nutshell, the only reason people ever finish 50%-completed projects instead of starting over!).

So could some other Wikipedia fork achieve the same thing? Programmer/blogger and Guardian columnist Seth Finkelstein, a frequent Wikipedia critic, has pointed out that other sites such as http://veropedia.com/ have tried to build a "verified" version of Wikipedia. "But," he writes, "it doesn't work for many reasons:

1) Maintenance
2) Nobody knows the site exists, or uses it.
3) Google will kill the site's ranking, because of "duplicate content"
4) Roughly 99% of Wikipedia's value is the Google-rank it has, and sites trying to copy its content don't have — or get — that Google-rank.

All true. But Knol has a shot at solving all of these problems. #1 should be mitigated if users earn money for maintaining articles — and besides, many articles like "Abraham Lincoln" won't need much maintenance anyway. #2 should not be an issue since it's a Google project. #3 and #4 depend on how Google lists Knol pages in its search results. The VP's blog post says only,

"Our job in Search Quality will be to rank the knols appropriately when they appear in Google search results. We are quite experienced with ranking web pages, and we feel confident that we will be up to the challenge."

Of course the question on everyone's minds, not answered directly by those sentences, is whether Knol pages will get any special treatment in search rankings. Google would probably be criticized if they manipulated the results outright. But they might achieve the same result indirectly — for example, having a tab across the top of their search results page for "Knol results," along with the tabs for Web, images, and news. Or if Knol results get killed for "duplicate content," Google might (legitimately) consider this a bug and tweak their duplicate-detection algorithm. Thus Knol would have the same advantage in Google that Microsoft's Media Player has on Windows: The operating system doesn't favor Media Player directly, but compatibility problems with Media Player will always get fixed first (while the RealPlayer people have to watch their programs get broken by Windows upgrades). One way or another, it's pretty certain that Knol results are not going to be "unfindable" on Google.

Now, I'm sure Knol will not formally fork Wikipedia. I wouldn't see any problem with them doing that, but it would be too controversial, after the VP announced it without ever mentioning "Wikipedia," and with Google already dealing with speculation that they're only creating Knol to complete with Wikipedia in their own search results. But with users having cash incentives to copy content from Wikipedia, probably most of the content would get replicated very quickly, and I would be surprised if many users didn't start writing scripts to robo-copy as much content from Wikipedia as possible.

Then you get to the point where experts start improving it. If the first couple of entries on "Physics" are just the robo-copied Wikipedia version, "signed" by users that nobody has ever heard of, this is barely an improvement over the unsigned article on Wikipedia itself. But then only one Physics professor in the entire world has to think it's worth their while to read the standard Wikipedia article, make any necessary corrections, and sign their name to it on Knol — and now you have a version that has been vetted by a credentialed expert, increasing its value many times to people who want to cite it as a source, or who want a higher degree of confidence that it's accurate. (Hopefully Knol will allow authors to confirm their e-mail addresses and display them — in an image, presumably, to stop them being scraped by spammers. This will allow professors to prove that they really have faculty .edu addresses and enhance the credibility of their articles, something I suggested for Citizendium.)

So, some criticisms of Wikipedia would not apply to Knol. Author Nick Carr has written of Wikipedia,

"Certainly, it's useful - I regularly consult it to get a quick gloss on a subject. But at a factual level it's unreliable, and the writing is often appalling. I wouldn't depend on it as a source, and I certainly wouldn't recommend it to a student writing a research paper."

When I asked if he would recommend Knol for the same purpose, he was more optimistic:

"Probably. Since a Knol would be written by an identifiable person at an identifiable point in time, I don't see why you wouldn't treat it, in doing research, in a similar way that you'd treat, say, an article by that person. Obviously, you'd need to judge the writer's expertise and authority when deciding whether or not to draw on his or her work, which becomes somewhat more problematic where no editorial or peer-review system applies, as in Knol."

This is where I think the value of a professor's .edu e-mail address comes in, which can at least establish a writer's authority in their subject. I asked Seth Finkelstein whether he would recommend Knol in those same circumstances (verified professor's .edu address, etc.), and his take was, "Of course I would, but you loaded the question in a way so as to remove any problem from it." Well, yeah. I just happen to think Knol actually could remove those problems.

Then there was a little-noticed phrase in Google's blog post that suggests another area where Knol could improve over Wikipedia: the inclusion of "how-to-fix-it instructions." Given that people often need how-to instructions a lot more badly than they need encyclopedia articles, it's surprising that there hasn't been an attempt to standardize around a "Wikipedia of how-to's." Perhaps it's because the Web itself actually does pretty well for that — type in the text of some error message, and you'll usually get some hits on support forums where people ran into the same problem. The trouble is that the ranking of search results depends on the popularity of the site, not on whether the thread ended with someone posting a solution to the problem, so you might have to read through a lot of search results to find an answer. And if you're an expert who happens to know how to do or fix something, there's not much incentive for you to post a page about it (even with AdSense ads), because your page will get buried in the search results beneath all the support forums discussing the same question, even if your post is more concise and useful. Some gurus like Dave Taylor and Leo Notenboom have written so many how-to articles that their own sites have risen through the Google rankings, so if they write a how-to article about something, it will get read (which, of course, creates an incentive for them to write more of them). But for a new expert just starting to write how-to articles, it would take a long time to reach that critical mass.

Knol, however, creates an incentive for experts to start posting how-to-fix-it advice and start reaping the rewards right away, since your how-to articles are just as easy to find under a given subject as anyone else's. Your earnings would start out small as you began to write articles, but they would rise in proportion to the number of articles you wrote, and you wouldn't have to slog along writing for no reward like a typical blogger or site creator, hoping to hit "critical mass" some day. You'd find out early on what the reward would be (financial and otherwise) for the work you were doing, and could decide if you wanted to continue.

Actually, the possibility of "instant rewards" does depend on how Knol articles are ranked against each other within a given topic. The Google blog post says, "For many topics, there will likely be competing knols on the same subject," and, "Knols will include strong community tools. People will be able to submit comments, questions, edits, additional content, and so on. Anyone will be able to rate a knol or write a review of it." Presumably the top-rated articles on a given topic will be displayed first by default, so I'm making the assumption that good articles really will get sorted to the top. But if there are already 50 articles on some topic, even if you know you can write a better one, how do you know it would rise to the top of the pile? If 10 people rate your article a 9, and all the other articles have been rated by 100 people each and got an average rating of a 7, then yours should still be listed as the highest in terms of average rating. But how do you get even those first 10 people to see your article? You could invite your friends, but then how do you stop anyone from gaming the system by inviting their friends and asking them all to rate their article a 10?

I'd written about this in the context of whether Wikia Search might try to solve these problems by allowing users to vote on search results, if they could prevent people from gaming the system. That was basically just me thinking out loud that something like that would be cool, before Wikia Search announced any specifics, and I haven't heard that Wikia is trying anything like that. But now that Google has stated that they will use voting and ranking systems in Knol, the question is how to reward new authors while preventing cheating. I suggested some ideas in an article about how to stop cheating and vote-buying on digg. One idea was that you could have a section on the page that showed people links to different articles at random, so that users couldn't self-select on what articles were shown to them, and if those randomly selected users followed the links and voted on the articles, count only those votes in determining the true "rating" of an article. (This is what HotOrNot does for people's picture ratings.) Even if few people rated those articles, the ratings that were collected, would be representative of real users, and not the horde of friends that you'd sent to rate your article. If that did not prove popular enough, then you could give authors the option to pay random users to rate their articles — as long as there was no way for authors to tie payment to higher ratings, the ratings would average out to reflect the article quality, and could be used to sort articles based on their actual merits.

So, I'd like to think that someone in the Googleplex is reading everything I write, but it's probably just a case where great nerds think alike. I wrote in Feb 07 that I thought Citizendium should allow authors to put their name next to articles, both for the "name up in lights" incentives factor and to enhance the article's credibility, and now Knol is going to do that (not to mention throwing in money as well). The same month I wrote that someone should build a search engine that groups together user-submitted articles under different topics, and provides a means for newly submitted articles to rise through the ranks as a result of user votes, and it sounds like Knol will attempt that too. Then in April 07 I wrote about the ways that you could prevent cheating in such a system, and even though Knol hasn't talked about what they will do to address that problem, they're almost certainly thinking about it, and have probably come up with some of the same ideas.

So let's do a test to find out if Google is reading these articles. There's one area where Wikipedia would beat Knol, and that is that everything on Wikipedia can be redistributed for free. That's something really special, and it's the one part of the Wikipedia hype that I actually buy into. I don't really care that Wikipedia articles were created as part of a "worldwide collaborative effort" unless that helps to achieve the goal of being useful. But Wikipedia, for all its flaws, represents the first time in human history that we have a compendium of a huge amount of human knowledge that can be copied freely, that literally belongs to the world, and because it's duplicated in so many places, it can literally never be taken away. That part of the hype really is true, and is quite heady when you think about it.

Google Knol has not declared this as one of their goals; a Knol article might not be freely distributable. When a proprietary project is hosted on a private site, there's always the risk that the company will pull the plug on it. They probably won't pull off the content offline, but they might shut the service down to stop new content from being added, the way Google did with Google Answers. Yes, Knol authors will retain ownership of their writings, so they could try to regroup and continue the project somewhere else, but it would be a huge mess to try and contact all of the authors and get their permission to copy all of their articles to the new location. As currently planned, Knol doesn't "belong the world," and Google never promised not to take it away.

So, I think that Google Knol should include a feature whereby authors can flag their articles as being freely distributable under the same terms as Wikipedia articles. (Any author who copied an article from Wikipedia and submitted it, would be required to set this flag, because under the terms of "copyleft", you can't copy something that's freely copyable and then try to stop others from copying it!) Then a user who wanted a copylefted, freely distributable article, could limit their search to articles that have this flag set. This would give Knol the best of both worlds: if the author of the top-ranked article did not wish for it to be freely redistributable, then they wouldn't put it on Wikipedia, but they could make it available on Knol, and users could choose either the top-ranked copylefted article or the top-ranked article overall, depending on what they wanted. If the best article on a given subject also happened to be flagged freely distributable, then so much the better.

Maybe the Knol people have had this idea already. But even so, if they end up implementing it, then I'm starting right away on articles about how Google should implement Google Anti-Censorware, Google Site Hijack Prevention, Google Security Compensation, and Google Sergey And Larry Give Bennett Their Airplane.

The following was written by frequent Slashdot editorial contributor Bennett Haselton. He writes "Recently author Annalee Newitz created a bit of a stir with the revelation that she had bought her way to the front page of the story-ranking site Digg. Since Digg allows any registered user to go to a story's URL and "digg it" in order to push it upward through the story-ranking system, it was inevitable that services like User/Submitter would come along, where a Digg user can pay for other users to cast votes to push their story up to the top. User/Submitter says they are currently backlogged and not taking new orders, but they say the service will return and will soon feature services for manipulating similar sites like Digg competitor reddit. Even if the new U/S features are vaporware, it probably won't be long before other companies offer similar services. But it seems like all of these story-ranking sites could prevent the manipulation by making one simple change to their voting algorithm."

Before getting to that though, what's at stake? The revelation that Digg could be trivially manipulated did not cause the site to be overrun with bogus stories all at once -- most of the links on the front page still look interesting. Newitz said that her story, which was deliberately chosen to be as lame as possible, got buried by users soon after it hit the front page, which is how Digg cleans spam stories out of the system. However, she also said that in the time that the story was on the front page, the story got about 35,000 hits, whereupon her server crashed and the traffic was thereafter divided with two other mirror sites; presumably if the server had stayed up, she would have gotten about 100,000 hits, all for an initial expenditure of $100, which is orders of magnitude cheaper than buying advertising any other way. (If she had done the same thing with a good story instead of a deliberately lame one, presumably the traffic gains resulting from word-of-mouth and repeat visitors would have been even higher.) As long as the benefits outweigh the cost, more and more unscrupulous users are likely to pay for such services, and since the service provided by User/Submitter is easy to copy, probably similar services will spring up to drive the price down even further. If nothing changes, then eventually sites like Digg and reddit will be flooded with nothing but paid stories. Most of the stories on the front page will probably still be interesting (why would you pay to promote a link, unless it was good enough to draw repeat visitors and get the most value for your money?), but everybody who didn't pay for votes would eventually get crowded out.

One Good Samaritan, Jim Messenger, managed to shut down one Digg manipulation service called Spike The Vote, by buying it out (for a paltry $1,275 - they must have wanted to get out fast) and then turning over to Digg. He warned people that the moral was: Don't sign up for Digg manipulation services, since Digg might get your information from them and then you'll be banned. Actually, I think the moral is simpler: if you're going to try anything like that, do it from a throwaway account that you don't care about losing if you get caught. (Or, only sign up with manipulation services which publish a privacy policy promising never to share your information, especially not with sites like Digg. Then if Digg buys them out, then the site has violated their privacy policy and Digg as the new owner inherits the liability for that, so you can sue them, right?) But as the idea spreads, it will probably become impractical to play whack-a-mole by shutting down manipulation services as they keep springing up. Any time the cost of providing a service (clicking on a few buttons) is small compared to the benefits of receiving the service (100,000 hits in 24 hours), a market will exist for it one way or another, whether you're talking about drug-smuggling, prostitution, or selling Digg votes.

However, I think there's a way to fix it, and here it is. Have you ever seen people put a link in their profile to their HotOrNot picture, saying "Go here and vote me a 10!!"? Similar to the people who send links to their friends and say, "I just posted this, please Digg this for me!" The difference is that on HotOrNot, it doesn't work. On HotOrNot, you can cast votes for a picture in one of two ways. The first way is to go directly to the URL for someone's picture; the second way is to load the front page, where a random picture from the database is selected at random, and vote for whatever picture comes up. The catch is that the votes that you cast by going directly to someone's picture, are simply ignored in calculating the average score for that photo. The only votes that are counted are the votes cast for random pictures displayed on the front page. So if you want to manipulate the voting for your own photo, you'd have to load the front page hundreds of thousands of times waiting for your own picture to come up repeatedly, which is hard to do without being detected.

To enable an algorithm like this on Digg and reddit, the sites could present users with a sidebar box that displays random stories from the pool of recent submissions. (reddit already has a serendipity feature that users can use to select a random story from the available pool, which could be leveraged for this purpose.) Once a story has collected, say, 100 votes -- or whatever number is considered sufficient to provide a representative random sample of how the story appeals to people -- then on that basis the story can either be buried or promoted to the top, where it would be seen by, say, 100,000 people. The elegance of this system is that bad content would only be seen by 100 people on average before it's buried, whereas good content would be seen by all the 100,000 people who view it on the front page, so the average user sees 1,000 pieces of good content for every 1 piece of crap. Even if 75% of users ignore the random story box completely, that just means you have to display it to 400 users instead of 100 before you have enough data points for a good random sample.

I suggested essentially the same algorithm for how an open-source search engine could work without being vulnerable to gaming even by those who understood all of its inner workings. The main difference, of course, is that Digg and reddit actually exist now. Digg declined to comment on the possible merits of such an algorithm; reddit's Steve Huffman said that the idea sounded interesting, although even if the idea got full buy-in, naturally any proposed change would take a long time to bring to fruition.

But it seems that an algorithm similar to this one would be the only way to prevent cheating on sites like Digg that sort content based on user votes. So it's ironic that HotOrNot, the only site I know of that is using a variation of this algorithm and hence is probably the most secure against cheating, is also the one where cheating is least likely to be a problem. Getting a high placement on Digg might enable you to make some money, but getting a highly rated picture on HotOrNot isn't going to make you rich (unless it helps you meet a millionaire who is using the site to find his third wife). Also, making HotOrNot meritocratic doesn't give people an incentive to improve the "content" that they submit, because up to the limits of what can be done with hair and wardrobe, you can't make yourself that much more attractive. With Digg and reddit, on the other hand, I might work harder at submitting a good story, if I knew that it worked in a perfectly meritocratic fashion that pushed good stories right to the top.

If you do this, you don't need any of the other countermeasures listed in Annalee Newitz's follow-up piece "Herding the Mob", such as analyzing user account history for suspicious behavior. As long as most users in the system are legitimate, most of the users in your random sample will be legitimate as well, and their voting will be representative of what most of the community would think. A story could also get a high score within a specific sub-area of the site like the sports page, but kept off of the main site front page, if the story got a high score from a random sampling of sports-oriented users but a low score from a sample of everyone else.

You could even sub-divide the topical areas further, down to a level of granularity like "Would Barack Obama make a good president?" A site called Helium is currently trying something like this -- users can submit essays on subjects like "Racial inequality or oppression: Do they truly exist in todays society?", and vote on how to rank other essays against each other. The voting works on the random selection principle that I'm advocating here -- users are presented with a pair of randomly chosen essays from a given category (not necessarily the same category for which you submitted an essay) and told to vote for the better one, so there's no way to tell all your friends to go to the link for your essay and give it a high rating. The main limitation though is that while the votes can push you to the top of a particular sub-category, that won't cause your article to "break out" and get to the front page of the site -- Helium says that those front-page articles are chosen at random by employees from the among those articles that are highly rated within their narrow category, so just being good is not enough. And if you want to write something that doesn't fit into any existing categories, you have to create a new category for your essay like I did, which will then be a category containing one essay that nobody else ever sees. Perhaps both of these limitations could be overcome by adding the option to rate randomly selected essays on a scale of 1 to 10 -- thus providing a way to rate essays that exist alone in their own category, and also a way to find the best essays across the entire site, rated against each other.

If Digg or reddit adopts a model that uses the random-voter-selection method, then there's the issue of how to handle the votes cast by users under the current system -- the ones who go to a story link and click "digg it", which is what makes the existing system vulnerable to gaming. Digg could do what HotOrNot does, and just ignore those votes outright, but users would probably view this as deceptive. Perhaps Digg could say that votes cast by self-selected users (the ones who go straight to the story link) are counted along with votes from randomly-selected users, unless the average of the self-selected votes is significantly different from the average from the randomly-selected votes, in which case the self-selected votes are ignored. Hopefully this would satisfy most users and preserve the "community" feel of the site, and only a spoilsport would point out that counting the self-selected votes only if they agree with the randomly-selected votes, is exactly the same thing as ignoring the self-selected votes entirely.

I asked the owner of User/Submitter what he thought about this. He was willing to talk with surprising candor (except about things like his real name) and spoke as if he'd like nothing better than for Digg to make changes to their service that would block his system from working. To both Annalee Newitz and me, he said, "We find it interesting that Digg still allows anybody to view any user's diggs. By way of this 'feature,' User/Submitter is able to verify that our users actually digg the stories they're given. Without this feature, Digg users are given complete digging privacy, and User/Submitter cannot exist." Some have expressed skepticism that the Digg cheaters really want Digg to fix the problem. But as a security tester, I can understand that mentality. If you report a problem, and a company doesn't fix it, eventually you get tempted to publicize the problem to draw attention to it. And if they still don't fix it, and it's a fairly benign security hole that merely enables some pranksters to get some undeserved attention, why not build a service around exploiting the hole, if will highlight the problem and encourage it to get fixed?

So I'm going to go out on a limb and say the U/S guy sincerely wants Digg to be more secure. However I disagree with him about his proposed fix, that of hiding a user's digg history. First of all, it won't stop anyone who creates a multitude of accounts all under their control -- you can use Tor to make it appear that you're coming from many different IP addresses, and build up a history of "legitimate" votes before using your votes to push sites deliberately. (Be sure to use different browsers, or vary your User-Agent header if you know how to do that, so that a series of votes from identical browser types doesn't give you away.) If your service does work by paying other users to cast votes, then you could still audit whether they're casting their votes honestly -- for example, create a test story, use 5 sockpuppet accounts to digg it 5 times, then tell your confederate to digg it. If the number of diggs doesn't go up to 6, then you know they're not honoring their end of the deal, and kick them out of the system. As long as most confederates think there might be some chance of getting caught if they don't play along, most of them would probably cast the votes that they were paid for, since it costs them nothing to do so and they wouldn't want to jeopardize their stream of easy money.

I asked the owner of User/Submitter if his service could defeat the random-sampling algorithm I described. "It would slow down our service," he answered, "but certainly wouldn't eliminate it because eventually a U/S User will have an opportunity to vote on a U/S Submission by way of chance." But I don't see how this would beat the algorithm -- some U/S voters would still get to vote on the story, but as long as there are far more legitimate voters than U/S voters, then a random sampling will almost always contain far more legitimate voters. The U/S owner also said, "Randomized voting privileges would be unnecessarily confusing, frustrating, and fragmenting. Not to forget: unfair and undemocratic." Well, you could keep it from being "confusing" or "frustrating" by keeping the existing interface (with the possible addition of a randomly-selected-story box), so that the only changes would be in how the votes are handled under the hood. "Fragmenting"? If anything, it seems to me that the existing Digg/reddit algorithms would be more fragmenting, keeping users within their existing communities of friend who vote for each others' stories; a random-selection box would give stories with "crossover appeal" a greater chance of success, bringing them to the attention of users who might otherwise never have seen them. As for "unfair and undemocratic", presumably this is a reaction to the fact that the votes of 100 users decide what everyone else sees. But it's already the case with Digg that the votes of a small number of users decide what content becomes popular. At least with a random sample of users, it would be the case that the vast majority of the time, the voting outcome would be the same as it would have been if the entire site had voted, due to the magic of representative sampling.

So, I'm putting this suggestion out there for the same reason that Jim Messenger bought out Spike The Vote -- because I don't want sites like Digg and reddit to be manipulated by the abusers. In fact, if they used this algorithm, they would become more meritocratic than they are now, because the systems would strictly favor the highest-rated content, instead of content written by people who have informal networks of friends who can all go digg their stories for them. If I were to design the user rating system to make it cheat-proof, these are the exact details of what I would do:

• Wherever they decide to post the "random story sampling" box (on the front page, or on a link off to a separate page, etc.), have it work so that as soon as new stories are submitted, they can be rotated into that box and displayed to a random set of users, until it's reached its total of 100 votes or however many are required to get a random sample.
• You can have "shutout voting" to kill off stories early that are obvious spam or otherwise really useless, without going through the full 100 votes. (For example, if 90% of the first 10 votes are negative, then stop collecting votes.) This decreases the number of users "inconvenienced" by really obvious spam and other garbage.
• For someone to submit content that gets rotated into that voting process, have them submit a Turing test (read numbers off of a graphic and type them in), or something similar. This prevents spammers from submitting spam content over and over just to have it viewed by those initial 10 voters. If they have to type in a number each time, it's not worth it.
• When users give votes to a story, give them the option to say why they voted the way that they did. (This is especially valuable if they're giving negative votes, then the submitter would know what to improve.) Personally I think the comments would be more valuable if each user can't see other users' comments, at the time they submit their own comments; this prevents the "me too" effect where everybody echoes the first two commenters. (When I ask for independent comments from people, and they almost all say the same thing without seeing each other's comments, that's when I know they have a point!)
• To prevent an attacker from having their own username hit the random-voting page over and over in hopes of voting up their own content, make sure that each user account is only allowed to vote on a given piece of content once (even if they found the content through the random-story page).
• Require a Turing test for new user signups. This would prevent an attacker from registering a huge number of accounts just to hit the random voting page with different users over and over, in hopes getting to vote on their own submitted content eventually.

Then after running this system for a while, look through some collected data to determine if the system could be more efficient. For example, do you really need a sample of 100 votes every time? Suppose you determine that in 99% of cases, you get the same result just from tabulating the first 50 votes, as you would have gotten from tabulating all 100 votes. Then you could modify the system to collect only the first 50 votes, and then make a decision.

Suggestions for improvement? Flaws (hopefully not fatal)? Everyone who cares about keeping community sites like Digg free from abuse, and who wants to create a path for the best content to rise to the top, let's put our heads together and see what we can think of. The above is intended merely as a jumping-off point, and although I've worked it over and I can't see any specific points to improve efficiency, that's probably just because I've been looking at it too long. And if you Digg this story for me I'll give you 1,000 times as much cash as I gave my Mom last Mother's Day.

Slashdot contributor Bennett Haselton writes "When Jimmy Wales recently announced the Search Wikia project, an attempt to build an open-source search engine around the user-driven model that gave birth to Wikipedia, he said his goal was to create "the search engine that changes everything", as he underscored in a February 5 talk at New York University. I think it could, although not for the same main reasons that Wales has put forth -- I think that for a search engine to be truly meritocratic would be more of a revolution than for a search engine to be open-source, although both would be large steps forward. Indeed, if a search engine could be built that really returned results in order of average desirability to users, and resisted efforts by companies to "game" the system (even if everyone knew precisely how the ranking algorithm worked), it's hard to overstate how much that would change things both for businesses and consumers. The key question is whether such an algorithm could be created that wouldn't be vulnerable to non-merit-based manipulation. Regardless of what algorithms may be currently under consideration by thinkers within the Wikia company, I want to argue logically for some necessary properties that such an algorithm should have in order to be effective. Because if their search engine becomes popular, they will face such huge efforts from companies trying to manipulate the search results, that it will make Wikipedia vandalism look like a cakewalk." The rest of his essay follows.

This will be a trip into theory-land, so it may be frustrating to users who dislike talk about "vaporware" and want to see how something works in practice. I understand where you're coming from, but I submit it's valuable to raise these questions early. This is in any case not intended to supplant discussion about how things are things are currently progressing.

First, though, consider the benefits that such a search engine could bring, both to content consumers and content providers, if it really did return results sorted according to average community preferences. Suppose you wanted to find out if you had a knack for publishing recipes online and getting some AdSense revenue on the side. You take a recipe that you know, like apple pie, and check out the current results for "apple pie". There are some pretty straightforward recipes online, but you believe you can create a more complete and user-friendly one. So you write up your own recipe, complete with photographs of the process showing how ingredients should be chopped and what the crust mixture should look like, so that the steps are easier to follow. (Don't you hate it when a recipe says "cut into cubes" and you want to throttle the author and shout, "HOW BIG??" It drove me crazy until I found CookingForEngineers.com.) Anyway, you submit your recipe to the search engine to be included in the results for "apple pie", and if the sorting process is truly meritocratic, your recipe page rises to the top. Until, that is, someone decides to surpass you, and publishes an even more user-friendly recipe, perhaps with a link to a YouTube video of them showing how to make the pie, which they shot with a tripod video camera and a clip-on mike in their well-lit kitchen. In a world of perfect competition, content providers would be constantly leapfrogging each other with better and better content within each category (even a highly specific one like apple pie recipes), until further efforts would no longer pay for themselves with increased traffic revenue. (The more popular search terms, of course, would bring greater rewards for those listed at the top, and would be able to pay for greater efforts to improve the content within that category.) But this constant leapfrogging of better and better content requires efficient and speedy sorting of search results in order to work. It doesn't work if the search results can be gamed by someone willing to spend effort and money (not worth it for the author of a single apple pie recipe, but worth it for a big money-making recipe site), and it doesn't work if it's impossible for new entrants to get hits when the established players already dominate search results.

Efficient competition benefits consumers even more for results that are sorted by price (assuming that among comparable goods and services, the community promotes the cheapest-selling ones to the top of the search results, as "most desirable"). If you were a company selling dedicated Web hosting, for example, you would submit your site to the engine to be included in results for "dedicated hosting". If you could demonstrate to the community that your prices and services were superior to your competitors', and if the ranking algorithm really did rank sites according to the preferences of the average user, your site could quickly rise to the top, and you'd make a bundle on new sales -- until, of course, someone else had the same idea and knocked you out of the top spot by lowering their prices or improving their services. The more efficient the marketplace, the faster prices fall and service levels rise, until the prices just covered the cost of providing the service and compensating the business owner for their time. It would be a pure buyer's market.

It's important to precisely answer the question: Why would this system be better than a system like Google's search algorithm, which can be "gamed" by enterprising businesses and which doesn't always return the results first that the user would like the most? You might be tempted to answer that in an inefficient marketplace created by an inefficient search result sorting algorithm, a user sometimes ends up paying $79/month for hosting, instead of the $29/month that they might pay if the marketplace were perfectly efficient. But this by itself is not necessarily wasteful. The extra $50 that the user pays is the user's loss, but it's also the hosting company's gain. If we consider costs and benefits across all parties, the two cancel out. The world as a whole is not poorer because someone overpaid for hosting.

The real losses caused by an inefficient search algorithm, are the efforts spent by companies to game the search results (e.g. paying search engine optimization firms to try and get them to the top Google spot), and the reluctance of new players to enter that market if they don't have the resources to play those games. If two companies each spend $5,000 trying to knock each other off of the top spot for a search like "weddings", that's $5,000 worth of effort that gets burned up with no offsetting amount of goods and services added to the world. This is what economists call a deadweight loss, with no corresponding benefit to any party. The two wedding planners might as well have smashed their pastel cars into each other. Even if a single company spends the effort and money to move from position #50 to position #1, that gain to them is offset by the loss to the other 49 companies that each moved down by one position, so the net benefit across all parties is zero, and the effort that the company spent to raise their position would still be a deadweight loss.

On the other hand, if search engine results were sorted according to a true meritocracy, then companies that wanted to raise their rankings would have to spend effort improving their services instead. This is not a deadweight loss, since these efforts result in benefits or savings to the consumer.

I've been a member of several online entrepreneur communities, and I'd conservatively estimate that members spend less than 10% of the time talking about actually improving products and services, and more than 90% of the time talking about how to "game" the various systems that people use to find them, such as search engines and the media. I don't blame them, of course; they're just doing what's best for their company, in the inefficient marketplace that we live in. But I feel almost lethargic thinking of that 90% of effort that gets spent on activities that produce no new goods and services. What if the information marketplace really were efficient, and business owners spent nearly 100% of their efforts improving goods and services, so that every ounce of effort added new value to the world?

Think of how differently we'd approach the problem of creating a new Web site and driving traffic to it. A good programmer with a good idea could literally become an overnight success. If you had more modest goals, you could shoot a video of yourself preparing a recipe or teaching a magic trick, and just throw it out there and watch it bubble its way up the meritocracy to see if it was any good. You wouldn't have to spend any time networking or trying to rig the results, you just create good stuff and put it out there. No, despite whatever cheer-leading you may have heard, it doesn't quite work that way yet -- good online businessmen still talk about the importance of networking, advertising, and all the other components of gaming the system that don't relate to actually improving products and services. But there is no reason, in principle, why a perfectly meritocratic content-sorting engine couldn't be built. Would it revolutionize content on the Internet? And, could Search Wikia be the project to do it, or play a part in it?

Whatever search engine the Wikia company produced, it would probably have such a large following among the built-in open-source and Wikipedia fan base, that traffic wouldn't be a problem -- companies at the top of popular search results would definitely benefit. The question is whether the system can be designed so that it cannot be gamed. I agree with Jimmy Wales's stated intention to make the algorithm completely open, since this makes it easier for helpful third parties to find weaknesses and get them fixed, but of course it also makes it easier for attackers to find those weaknesses and exploit them. If you think Microsoft paying a blogger to edit Wikipedia is a problem, imagine what companies will do to try and manipulate the search results for a term like "mortgage". So what can be done?

The basic problem with any community that makes important decisions by "consensus" is that it can be manipulated by someone who creates multiple phantom accounts all under their control. Then if a decision is influenced by voting -- for example, the relative position of a given site in a list of search results -- then the attacker can have the phantom accounts all vote for one preferred site. You can look for large numbers of accounts created from the same IP address, but the attacker could use Tor and similar systems to appear to be coming from different IPs. You could attempt to verify the unique identity of each account holder, by phone for example, but this requires a lot of effort and would alienate privacy-conscious users. You could require a Turing test for each new account, but all this means is that an attacker couldn't use a script to create their 1,000 accounts -- an attacker could still create the accounts if they had enough time, or if they paid some kid in India to create the accounts. You could give users voting power in proportion to some kind of "karma" that they had built up over time by using the site, but this gives new users little influence and little incentive to participate; it also does nothing to stop influential users from "selling out" their votes (either because they became disillusioned, or because they signed up with that as their intent from the beginning!).

So, any algorithm designed to protect the integrity of the Search Wikia results would have to deal with this type of attack. In a recent article about Citizendium, a proposed Wikipedia alternative, I argued that you could deal with conventional wiki vandalism by having identity-verified experts sign off on the accuracy of an article at different stages. That's practical for a subject like biology, where you could have a group of experts whose collective knowledge covers the subject at the depth expected in an encyclopedia, but probably not for a topic like "dedicated hosting" where the task is to sift through tens of thousands of potential matches and find the best ones to list first. You need a new algorithm to harness the power of the community. I don't know how many possible solutions there are, but here is one way in which it could be done.

Suppose a user submits a requested change to the search results -- the addition of their new Site A, or the proposal that Site A should be ranked higher. This decision could be reviewed by a small subset of registered users, selected at random from the entire user population. If a majority of the users rate the new site highly enough as a relevant result for a particular term, then the site gets a high ranking. If not, then the site is given a low ranking, possibly with feedback being sent to the submitter as to why the site was not rated highly. The key is that the users who vote on the site have to be selected at random from among all users, instead of letting users self-select to vote on a particular decision.

The nice property of this system is that an attacker can't manipulate the voting simply by having a large number of accounts at their control -- they would have to control a significant proportion of accounts across the entire user population, in order to ensure that when the voters were selected randomly from the user population, the attacker controlled enough of those accounts to influence the outcome. (If an attacker ever really did spend the resources to reach that threshold point, and it became apparent that they were manipulating the votes, those votes could be challenged and overridden by a vote of users whose identities were known to the system. This would allow the verified-identity users to be used as an appeal of last resort to block abuse by a very dedicated adversary, while not requiring most users to verify their identity. This is basically what Jimmy Wales does when he steps in and arbitrates a Wikipedia dispute, acting as his own "user whose identity is known".)

This algorithm for an "automated meritocracy" (automeritocracy? still not very catchy at 7 syllables) could be extended to other types of user-built content sites as well. Musicians could submit songs to a peer review site, and the songs would be pushed out to a random subset of users interested in that genre, who would then vote on the songs. (If most users were too apathetic to vote, the site could tabulate the number of people who heard the song and then proceeded to buy or download it, and count those as "votes" in favor.) If the votes for the song are high enough, it gets pushed out to all users interested in that genre; if not, then the song doesn't make it past the first stage. If there are 100,000 users subscribed to a particular genre, but it only takes ratings from 100 users to determine whether or not a song is worth pushing out to everybody, that means that when "good" content is sent out to all 100,000 people but "bad" content only wastes the time of 100 users, the average user gets 1,000 pieces of "good" content for every 1 piece of "bad" content. New musicians wouldn't have to spend any time networking, promoting, recruiting friends to vote for them -- all of which have nothing to do with making the music better, and which fall into the category of deadweight losses described above.

An automeritocracy-like system could even be used as a spam filter for a large e-mail site. Suppose you want to send your newsletter to 100,000 Hotmail users (who really have signed up to receive it). Hotmail could allow your IP to send mail to 100,000 users the first time, and then if they receive too many spam complaints, block your future mailings as junk mail. But if that's their practice, there's nothing to stop you from moving to a new, unblocked IP and repeating the process from there. So instead, suppose that Hotmail stores your 100,000 received messages temporarily into users' "Junk Mail" folders, but selectively releases a randomly selected subset of 100 messages into users' inboxes. Suppose for arguments' sake that when a message is spam, 20% of users click the "This is spam" button, but if not, then only 1% of users click it. Out of the 100 users who see the message, if the number who click "This is spam" looks close to 1%, then since those 100 users were selected as a representative sample of the whole population, Hotmail concludes that the rest of the 100,000 messages are not spam, and moves them retroactively to users' inboxes. If the percentage of those 100 users who click "This is spam" is closer to 20%, then the rest of the 100,000 messages stay in Junk Mail. A spammer could only rig this system if they controlled a significant proportion of the 100,000 addresses on their list -- not impossible, but difficult, since you have to pass a Turing test to create each new Hotmail account.

The problem is, there's a huge difference between systems that implement this algorithm, and systems that implement something that looks superficially like this algorithm but actually isn't. Specifically, any site like HotOrNot, Digg, or Gather that lets users decide what to vote on, is vulnerable to the attack of using friends or phantom users to vote yourself up (or to vote someone else down). In a recent thread on Gather about a new contest that relied on peer ratings, many users lamented the fact that it was essentially rigged in favor of people with lots of friends who could give them a high score (or that ratings could be offset unfairly in the other direction by "revenge raters" giving you a 1 as payback for some low rating you gave them). I assume that the reason such sites were designed that way is that it just seemed natural that if your site is driven by user ratings, and if people can see a specific piece of content by visiting a URL, they should have the option on that page to vote on that content. But this unfortunately makes the system vulnerable to the phantom-users attack.

(Spam filters on sites like Hotmail also probably have the same problem. We don't know for sure what happens when the user clicks "This is spam" on a piece of mail, but it's likely that if a high enough percentage of users click "This is spam" for mail coming from a particular IP address, then future mails from that IP are blocked as spam. This means you could get your arch-rival Joe's newsletter blacklisted, by creating multiple accounts, signing them up for Joe's newsletter, and clicking "This is spam" when his newsletters come in. This is an example of the same basic flaw -- letting users choose what they want to vote on.)

So if the Wikia search site uses something like this "automeritocracy" algorithm to guard the integrity of its results, it's imperative not to use an algorithm vulnerable to the hordes-of-phantom-users attack. Some variation of selecting random voters from a large population of users would be one way to handle that.

Finally, there is a reason why it's important to pay attention to getting the algorithm right, rather than hoping that the best algorithm will just naturally "emerge" from the "marketplace of ideas" that results from different wiki-driven search sites competing with each other. The problem is that competition between such sites is itself highly inefficient -- a given user may take a long time to discover which site provides better search results on average, and in any case, it may be that Wiki-Search Site "B" has a better design but Wiki-Search Site "A" had first-mover advantage and got a larger number of registered users. When I wrote earlier about why I thought the Citizendium model was better than Wikipedia, several users pointed out that it may be a moot point, for two main reasons. First, most users will not switch to a better alternative if it never occurs to them. Second, for sites that are powered by a user community, it's very hard for a new competitor to gain ground, even with a superior design, if the success of your community depends on lots of people starting to use it all at once. You could write a better eBay or a better Match.com, but who would use it? Your target market will go to the others because that's where everybody else is. Citizendium is, I think, a special case, since they can fork articles that started life on Wikipedia, so Wikipedia doesn't have as huge of an advantage over them as they would if Citizendium had to start from scratch. But the general rule about imperfect competition still applies.

It's a chicken-and-egg problem: You can have Site A that works as a pure meritocracy, and Site B that works as an almost-meritocracy but can be gamed with some effort. But Site B may still win because the larger environment in which they compete with each other, is not itself a meritocracy. So we just have to cross our fingers and hope that Search Wikia gets it right, because if they don't, there's no guarantee that a better alternative will rise to take its place. But if they get it right, I can hardly wait to see what changes it would bring about.

WhiskerBiscuit writes "The boys at Am I Hot or Not have started a sweepstakes to encourage people to register to vote. According to this blogger's analysis, the contest should encourage disempowered people to register (subject to the constraint that poor people don't have computers). The organizers have cleverly split the prize between a lucky winner and whoever happens to have referred them, providing a selection advantage for viral dispersal of the meme."

macdaddypunk writes "We all remember last December's grim news: MP3.com closed its doors, warning thousands of musicians that 'all your content will be deleted from our servers.' However, as the Wall Street Journal reports today, most of the original MP3.com archive was never deleted! Two companies, GarageBand and Trusonic, claiming to have a legitimate copy of the archive, are now enabling former MP3.com artists to visit www.MP3isBack.com and recover their MP3.com music, instantly re-generating their artist pages with just a few clicks. Trusonic, itself a Vivendi spin-out, focuses on licensing music to retailers for in-store airplay. GarageBand, like a HOTorNOT for music, offers free mp3 downloads and claims to host the definitive charts of independent music."

webword writes: "Web Techniques is running an interesting article written by James Hong, one of the masterminds behind AmIHotOrNot.com (now known as HOT or NOT?). Before you decide to skip over this, consider that Hong and company used Apache, PHP and MySQL to build their site. They found that these open source tools ran much better on a 700-MHz Pentium III than a quad processor Sun E220. Hong also covers their moderation system, advertising arrangements, and how they were able to scale to handle 1.8 million page views per day after being in operation for a mere 8 days."