Slashdot Mirror

Rowling confirmed (after the information leaked accidentally) that she had authored a new book, The Cuckoo's Calling, under the male pseudonym Robert Galbraith, which went on to sell only about 1,500 copies before she announced that she was the real author and sales of the book spiked 150,000%.

Stephen King actually tried something similar in the 1970s, publishing a series of books under the pseudonym "Richard Bachman," which he later said was partly an attempt to answer the question of whether his success was due to talent or luck. (The Richard Bachman books sold 10 times as many copies after King was revealed as the author.) Rowling has not said whether she was attempting a similar experiment, having issued a statement that before the revelation, it had been "wonderful to publish without hype or expectation, and pure pleasure to get feedback under a different name."

But if either J.K. Rowling or Stephen King really wants to find the answer to the question of talent vs. luck, the solution lies in the random-sample voting algorithm that I've been advocating in occasional articles for years now, going back to "Censorship By Glut" in 2006. Here's how the experiment could work, for evaluating the quality of fiction writing:

1. Rowling or King could approach a pre-established amateur fiction hosting site with a large number of registered users. Or they could create their own fiction hosting site and announce it to the world for the purpose of running the experiment, which would almost certainly attract a large number of users to sign up. (The experiment only works if the site has a large number of users, for reasons that will become clear.)

2. When a user submits a new short story to the site, the site randomly selects a small subset of other users on the site (say, 20 other users), emails them a link to the new story, and invites them to read it and rate its content. There are several ways you could incentivize those users to read the link and rate the story on a scale of 1 to 10. You could bill it as the "civic duty" of registered users of the site (in the same way that it's the civic duty of registered Wikipedia editors to maintain the quality of articles, even though the editors are working for free). You could require registered users to read and rate any stories that are emailed to them (although of course there'd be no way to stop someone from lazily submitting a rating without even reading the story). You could actually require payments from users who submit stories, and then use that money to distribute small payments to the raters as compensation for reading the story (although that seems like it would be the biggest headache, since you'd have to jump through legal and logistical hoops to set it up, and it would attract cheaters who would try to abuse the system just for the free small payments). But in any case, you don't need every user who gets emailed a story, to actually click through to read the story and rate it. All that matters is that out of those 20 users, enough of them click through to read the story, that you get a statistically representative sample of what users think of the quality.

   Optionally, the story raters could also submit written feedback about why they liked or did not like a story. But the important part is collecting the numeric ratings so that they can be averaged into a single overall rating for the piece of content.

3. If a story gets a high enough average rating in the first round of voting, then it gets emailed out to a larger random sample of voters, say, 200. The ratings given by this larger sample can be used to distinguish the very best stories from the merely good. (We expect that for good stories, the ratings would tend to cluster around the high end of the scale, so with that smaller variance, it would take a larger sample size to find a statistically significant difference between the quality of two stories.)

4. The stories that get the highest ratings can be featured on the front page of the site, so that everybody can have the benefit of enjoying the "best" stories. Meanwhile, J.K. Rowling and Stephen King have the benefit of finding out how their stories compare against stories written by unpublished amateur writers.

It all sounds deceptively simple, but the important feature is that you've taken the arbitrariness out of the outcome. As long as your sample sizes are large enough, the rating that a story obtains in this system, will be approximately equal to the average rating it would get from all users across the site. "Luck" is no longer a factor, because you could re-run the experiment twice with the same set of stories, and get approximately the same outcome.

This is important, because numerous experiments and real-world studies have shown that in any environment where users can recommend content to each other and browse content that is already known to be popular — in other words, how most of us discover content in the real world — luck plays a much greater role in which content becomes wildly successful. The generally accepted explanation is that an initial stroke of luck can have a self-reinforcing snowball effect — if a few key influencers happen to discover and recommend a piece of content at the same time, their friends and followers will be drawn to that content as well, and once it crosses that threshold, the content has now become "popular" enough that even more users will be drawn to it just because it's popular.

This is also why any of the existing fiction-rating sites would not work for this experiment — because most such sites allow authors to invite their friends to sign up and give high ratings to their stories, or to form cliques that all give high ratings to each other's writings. It's usually in the site's best interests to allow these tricks, because it gives authors the incentive to promote the site to their friends in order to get them to sign up. But it also means that (a) authors can easily game the system, and the highest-rated stories may not be the highest-quality ones but the ones whose authors simply play the game the best, and (b) even without "gaming the system", the fact that users can see other users' ratings and can seek out "most popular" or "trending" stories, creates the snowball effects discussed above, and introduces a huge amount of arbitrariness into the process.

Duncan Watts' excellent book Everything Is Obvious (Once You Know The Answer) is an excellent introduction to the arbitrariness phenomenon, but if you don't have time to read the whole book, just read about the Matthew Salganik 'many-worlds' experiment", which Watts co-authored and which I've linked to in pretty much every other article I've written about the random-sample-voting algorithm. The gist was that if you divide users into multiple artificial "worlds," where users can recommend content only to other users within those worlds, and seed all artificial worlds with the same content (in this case, songs), then songs which become wildly popular in some worlds will become duds in others.

The whole of Everything is Obvious is at least as insightful as anything ever written by Malcolm Gladwell, and would appeal to the same people, but it never became a bestseller, because — well, probably because we live in one of the many possible worlds of a Salganik experiment, and in the world we happen to live in, the luck of the draw meant that book didn't take off.

But back to the proposed experiment. It is true that the votes of the average users would not tell us anything about whether the winning stories were "artistically" good, however you define that. But in King's case, he was not trying to answer questions about artistic merit. he was trying to find out if his bestselling-author status was due to talent or luck, so the average rating from regular readers would be quite on point. Rowling said that she wanted to write without any hype and receive honest feedback, and it's hard to imagine a better place to do that than writing under a pseudonym for a fiction site that distributes your content directly to the public.

Both King and Rowling deserve some credit for even addressing the question of whether their success was due to talent or luck. It would have been easy for them to assume that their global success was due to their innate skill and hard work, and 99% of the world would have accepted that explanation, so it took no small amount of courage to even raise the question of how luck might have played a role. (We all know plenty of successful people who take umbrage if you even mention "the L word".)

But King did say that he thought he was outed too early to obtain any conclusive results from the experiment (and Rowling also said she wished she could have kept writing under the pseudonym, although she didn't say whether she had any similar "experiment" in mind). The random-sample-voting algorithm would provide instant feedback, not just to King and Rowling, but to any other writer who wanted to see how their writings would stack up against others in the field, from unpublished amateurs to worldwide bestselling authors.

My prediction, if such an experiment is ever conducted: King's and Rowling's writings would be rated very good, but so would many other writers' stories, including struggling writers who have never been published. Or as economist Daniel Kahneman put it: "success = talent + luck; great success = a little more talent + a lot of luck." (That took a certain amount of modesty on his part too, having achieved "great success" himself in the form of a Nobel Prize.) If J.K. Rowling or Stephen King ever launched such an experiment, the biggest favor they'd be doing for the world would not be to boost the egos of a few struggling writers, but to call more attention to the role that luck plays the world.

It's not as if their own egos would have to be bruised in the process. Donald Trump, the last person in the world that I would have guessed to have uttered these words, actually said that "Everything in life is luck," but it didn't seem to deflate his opinion of himself. You don't have to be a jerk like Trump, but just because some unpublished author's story gets a higher rating than yours, doesn't mean you have to let him come live in your mansion.