J.K. Rowling Should Try the Voting Algorithm

Frequent contributor Bennett Haselton proposes a new use for online, anonymous voting: helping sort skill from luck in the cheek-by-jowl world of best-selling (and would-be best-selling) authors: "J.K. Rowling recently confirmed that she was the author of a book she had published under a pseudonym, which spiked in sales after she was outed as the true author. Perhaps she was doing an experiment to see how much luck had played a role in propelling her to worldwide success, and whether she could recreate anything close to that success when starting from scratch. But a better way to answer that question would be to strike a deal with an amateur-fiction-hosting site and use the random-sample-voting algorithm that I've written so much about, to test how her writing stacks up against other writers in the same genre." Read on for more. Update: 07/20 01:23 GMT by T : Note: An editorial goof (mine) swapped out the word "confirmed" for "revealed" (above) in an earlier rendering of this story.

Rowling confirmed (after the information leaked accidentally) that she had authored a new book, The Cuckoo's Calling, under the male pseudonym Robert Galbraith, which went on to sell only about 1,500 copies before she announced that she was the real author and sales of the book spiked 150,000%.

Stephen King actually tried something similar in the 1970s, publishing a series of books under the pseudonym "Richard Bachman," which he later said was partly an attempt to answer the question of whether his success was due to talent or luck. (The Richard Bachman books sold 10 times as many copies after King was revealed as the author.) Rowling has not said whether she was attempting a similar experiment, having issued a statement that before the revelation, it had been "wonderful to publish without hype or expectation, and pure pleasure to get feedback under a different name."

But if either J.K. Rowling or Stephen King really wants to find the answer to the question of talent vs. luck, the solution lies in the random-sample voting algorithm that I've been advocating in occasional articles for years now, going back to "Censorship By Glut" in 2006. Here's how the experiment could work, for evaluating the quality of fiction writing:

1. Rowling or King could approach a pre-established amateur fiction hosting site with a large number of registered users. Or they could create their own fiction hosting site and announce it to the world for the purpose of running the experiment, which would almost certainly attract a large number of users to sign up. (The experiment only works if the site has a large number of users, for reasons that will become clear.)

2. When a user submits a new short story to the site, the site randomly selects a small subset of other users on the site (say, 20 other users), emails them a link to the new story, and invites them to read it and rate its content. There are several ways you could incentivize those users to read the link and rate the story on a scale of 1 to 10. You could bill it as the "civic duty" of registered users of the site (in the same way that it's the civic duty of registered Wikipedia editors to maintain the quality of articles, even though the editors are working for free). You could require registered users to read and rate any stories that are emailed to them (although of course there'd be no way to stop someone from lazily submitting a rating without even reading the story). You could actually require payments from users who submit stories, and then use that money to distribute small payments to the raters as compensation for reading the story (although that seems like it would be the biggest headache, since you'd have to jump through legal and logistical hoops to set it up, and it would attract cheaters who would try to abuse the system just for the free small payments). But in any case, you don't need every user who gets emailed a story, to actually click through to read the story and rate it. All that matters is that out of those 20 users, enough of them click through to read the story, that you get a statistically representative sample of what users think of the quality.

   Optionally, the story raters could also submit written feedback about why they liked or did not like a story. But the important part is collecting the numeric ratings so that they can be averaged into a single overall rating for the piece of content.

3. If a story gets a high enough average rating in the first round of voting, then it gets emailed out to a larger random sample of voters, say, 200. The ratings given by this larger sample can be used to distinguish the very best stories from the merely good. (We expect that for good stories, the ratings would tend to cluster around the high end of the scale, so with that smaller variance, it would take a larger sample size to find a statistically significant difference between the quality of two stories.)

4. The stories that get the highest ratings can be featured on the front page of the site, so that everybody can have the benefit of enjoying the "best" stories. Meanwhile, J.K. Rowling and Stephen King have the benefit of finding out how their stories compare against stories written by unpublished amateur writers.

It all sounds deceptively simple, but the important feature is that you've taken the arbitrariness out of the outcome. As long as your sample sizes are large enough, the rating that a story obtains in this system, will be approximately equal to the average rating it would get from all users across the site. "Luck" is no longer a factor, because you could re-run the experiment twice with the same set of stories, and get approximately the same outcome.

This is important, because numerous experiments and real-world studies have shown that in any environment where users can recommend content to each other and browse content that is already known to be popular — in other words, how most of us discover content in the real world — luck plays a much greater role in which content becomes wildly successful. The generally accepted explanation is that an initial stroke of luck can have a self-reinforcing snowball effect — if a few key influencers happen to discover and recommend a piece of content at the same time, their friends and followers will be drawn to that content as well, and once it crosses that threshold, the content has now become "popular" enough that even more users will be drawn to it just because it's popular.

This is also why any of the existing fiction-rating sites would not work for this experiment — because most such sites allow authors to invite their friends to sign up and give high ratings to their stories, or to form cliques that all give high ratings to each other's writings. It's usually in the site's best interests to allow these tricks, because it gives authors the incentive to promote the site to their friends in order to get them to sign up. But it also means that (a) authors can easily game the system, and the highest-rated stories may not be the highest-quality ones but the ones whose authors simply play the game the best, and (b) even without "gaming the system", the fact that users can see other users' ratings and can seek out "most popular" or "trending" stories, creates the snowball effects discussed above, and introduces a huge amount of arbitrariness into the process.

Duncan Watts' excellent book Everything Is Obvious (Once You Know The Answer) is an excellent introduction to the arbitrariness phenomenon, but if you don't have time to read the whole book, just read about the Matthew Salganik 'many-worlds' experiment", which Watts co-authored and which I've linked to in pretty much every other article I've written about the random-sample-voting algorithm. The gist was that if you divide users into multiple artificial "worlds," where users can recommend content only to other users within those worlds, and seed all artificial worlds with the same content (in this case, songs), then songs which become wildly popular in some worlds will become duds in others.

The whole of Everything is Obvious is at least as insightful as anything ever written by Malcolm Gladwell, and would appeal to the same people, but it never became a bestseller, because — well, probably because we live in one of the many possible worlds of a Salganik experiment, and in the world we happen to live in, the luck of the draw meant that book didn't take off.

But back to the proposed experiment. It is true that the votes of the average users would not tell us anything about whether the winning stories were "artistically" good, however you define that. But in King's case, he was not trying to answer questions about artistic merit. he was trying to find out if his bestselling-author status was due to talent or luck, so the average rating from regular readers would be quite on point. Rowling said that she wanted to write without any hype and receive honest feedback, and it's hard to imagine a better place to do that than writing under a pseudonym for a fiction site that distributes your content directly to the public.

Both King and Rowling deserve some credit for even addressing the question of whether their success was due to talent or luck. It would have been easy for them to assume that their global success was due to their innate skill and hard work, and 99% of the world would have accepted that explanation, so it took no small amount of courage to even raise the question of how luck might have played a role. (We all know plenty of successful people who take umbrage if you even mention "the L word".)

But King did say that he thought he was outed too early to obtain any conclusive results from the experiment (and Rowling also said she wished she could have kept writing under the pseudonym, although she didn't say whether she had any similar "experiment" in mind). The random-sample-voting algorithm would provide instant feedback, not just to King and Rowling, but to any other writer who wanted to see how their writings would stack up against others in the field, from unpublished amateurs to worldwide bestselling authors.

My prediction, if such an experiment is ever conducted: King's and Rowling's writings would be rated very good, but so would many other writers' stories, including struggling writers who have never been published. Or as economist Daniel Kahneman put it: "success = talent + luck; great success = a little more talent + a lot of luck." (That took a certain amount of modesty on his part too, having achieved "great success" himself in the form of a Nobel Prize.) If J.K. Rowling or Stephen King ever launched such an experiment, the biggest favor they'd be doing for the world would not be to boost the egos of a few struggling writers, but to call more attention to the role that luck plays the world.

It's not as if their own egos would have to be bruised in the process. Donald Trump, the last person in the world that I would have guessed to have uttered these words, actually said that "Everything in life is luck," but it didn't seem to deflate his opinion of himself. You don't have to be a jerk like Trump, but just because some unpublished author's story gets a higher rating than yours, doesn't mean you have to let him come live in your mansion.

Comments

JK Rowling!

[i kan reed]

Now that a famous person's name has gotten your attention, please use our website and give us money!

Re:JK Rowling!

[girlintraining]

Now that a famous person's name has gotten your attention, please use our website and give us money!

More or less, yeah. It's "Hey, here's a famous artist who could have used something I wrote my thesis about, but didn't." And we're supposed to gloss over the why they didn't. Here's the thing... algorithms and voting mechanics are fine for thesis projects, but this is an author. She makes her living selling books. And the best litmus test for whether it's the name, or the work itself, that people are buying, is to put it on the market under a pseudonym and find out.

Which is what she did. Shame on her for doing the same thing so many famous authors have done for thousands of years instead of opting for little college boy's pet thesis project!

Re:JK Rowling!

[bennetthaselton]

But my argument is that releasing a second book under a pseudonym is *not* a good test of whether your success was due to "the work itself", because there's so much luck involved either way. If your first book is a hit but your second book (under a pseudonym) is not, it could be that you're a bad writer who got really lucky the first time, or it could be that you're a good writer who got unlucky the second time.

The random-sample-voting algorithm takes the luck out of the equation and actually measures what the average reader thinks of your writing, when they're not biased by hype or expectations.

tl;dr

[Anonymous Coward]

no ones going to read all this you huge nerd

Fud

[Anonymous Coward]

"But a better way to answer that question would be to strike a deal with an amateur-fiction-hosting site and use the random-sample-voting algorithm that I've written so much about"

If you've developed a method to distinguish the difference between luck and skill then you should really be talking to some scientists, not trying to get Barry Potter and the Sea of Bees to join your amateur-fiction-hosting-hour.

Name dropping BS

[Anonymous Coward]

J.K. Rowling was doing nothing of the sort.

http://www.bbc.co.uk/news/entertainment-arts-23366660

I appreciate the the autors may havea neat Idea they want to push but name dropping like this and creating false image of proceedings suggests they're not the brightest or most observant bunch.

Re:Name dropping BS

[tnk1]

And it would be a stupid idea to begin with.

People buying your books isn't all about your talent. It's also having built a reputation.

Why would I read Stephen King over Richard Bachman? Because while Bachman could be a genius, I already know that I like most of what King puts out. There is still a risk that this latest venture by King is not one of his best or even sucks, but when it comes to actually spending money, it is less of a risk than buying something that J. Random Author has put out. The King books are also more likely to have more reviews so I can do a little checking if I want to risk spoilers.

The fact is you do need to be good to be picked up by a publisher, but even if you have talent, it can still be about luck. You need the luck to get the break and then the talent to not become a one-off.

You'll notice a lot of writers and poets were editors or contributors to college or even high school publications. These publications are not very influential and have a limited readership, but what they do is put you in contact with people who are actually going into the editorial field. If you want to be a writer, it helps to know writers and editors. Even other writers either become editors on the side or they know editors.

Obviously, the publishing field is always looking for talent, but that talent can often be generated from the friends of the people they are already in the same social circles with.

And if you look at Rowling, she might have published under a pseudonym, but the publisher knew who she was, and publishers can often do more for a writer's popularity than even the quality of the writing can. Look at the crap like Twilight and Fifty Shades of Gray. Do you believe it became popular because the authors were talented writers? Instead, they wrote fiction in an area that the publisher could sell them.

Re:Name dropping BS

[bennetthaselton]

Well this one *didn't* sell, so apparently even the influence of a smart publisher wasn't enough to break the "luck" barrier.

But back to the question of "Why would I read Stephen King over Richard Bachman?" -- this is actually quite an insightful point and highlights the difference between scenarios where the random-sample-voting algorithm would be beneficial and scenarios where it wouldn't. The key point is, do you already feel like you have enough good books in your life, more than you have time to read?

If the answer is Yes, then you might as well just stick with the authors you like, like Stephen King, if they collectively write more than enough books to fill your needs.

On the other hand, if you want more books to read, then the random-sample-voting algorithm can be useful for identifying more authors who have something good to contribute.

There is also a second-order effect: As long as luck dominates the publishing industry, then some potentially good authors might not even bother writing books because they know (correctly) that the odds of hitting it big are so slim. But if a process existed to take the arbitrariness out of it, then some of those authors might try their hand at it after all.

quality vs popularity

[minstrelmike]

I don't care about voting, blind or not.
Socially, the New York Times is supposed to be representative of American media and literacy.
In reality, the National Enquirer is the most _popular_ 'news' paper in America.

imo, you should decide what you want to measure first. Not measure (vote) and then say that is the highest quality because quality is not what is being measured when you count popularity.

Short Experiment (Rowling's)

[bill_mcgonigle]

I went looking for the older reviews on Amazon, and while they were generally positive, it looks like this 'experiment' lasted a month and a half or so. If it were a year or more, that would seem like an experiment. A month and a half seems like a publicity stunt. The reviews there seem to indicate that everybody knew it was a pseudonym and suspected the author had significant publisher backing, so it's hard to even call it a fair experiment in the first place.

Re:Short Experiment (Rowling's)

[rhyder128k]

I don't really like the way that Rowling is held up as an example of a rags to riches success against the odds. According to Wikipedia she completed her university education and gained a degree in French and Classics. She then travelled the world a bit, got married, had a kid and then got divorced. When she returned to the UK, she moved into a flat and had all of her bills paid by the state. She then lived the (expensive) coffee shop lifestyle while she was writing her book. At one point, she received a massive grant from an arts council.

It was cushy. I wish I'd had it as hard her when I was struggling to get my first bit of paid writing work.

A practical algorithm for manuscript quality.

[slimdave]

Speaking as someone who shares a sofa of an evening with a publisher, I can vouch that almost every manuscript submitted to any publisher will be dreadful in almost every single department. It starts with embarrassingly poor spelling and punctuation, and moves on through dreadful grammar, choice of paragraph size, layout, and on to issues with plot, characterisation, and general readability.

The average quality hovers somewhere between execrable and toe-curlingly awful, and they get dismissed after a glance through the first page. Sometimes the covering letter is enough and the manuscript can be spared its cursory eyeballing, because if you cannot correctly spell and punctuate in your covering letter then you're wasting everyone's time -- thankfully only 15 seconds of it was the publishers, and two years of it was yours.

Based on that alone, a very useful algorithm with a high degree of accuracy in judging a manuscript's quality is to just throw it straight in the bin -- you'd only be wrong one time in a couple of hundred, which is a pretty good average.

Re:A practical algorithm for manuscript quality.

[MickLinux]

Actually, that's a lousy average. If the standard is 1 good, money-making book per three-hundred, then you can expect to waste 250 hours sorting through trash to find a good moneymaker. Then, assuming that you don't miss it -- which I suspect happens at least 3 times in 4, putting you up at perhaps 1000 hours now -- you can capitalize on it, and make some money.

That 1000 hours has to be split with other duties, so you're talking about one new find every 3 or four years, per reviewer.

Throw each and every one in the waste bin, unopened, eand you're down to one new find every, what, 100 years? 1000 years? 10000 years? You haven't given a criteria of acceptance, and therefore it'll take however long it takes for you to learn that it was you, yourself, who was stupid.

Here's the criteria that publishers use, according to one respected, published, source:

A book publisher is more likely to publish something by someone who has already written a couple years' of articles for magazines.

A magazine publisher is more likely to publish magazine articles if the topic meets the criteria of interest of their special magazine.

There are over a million special interest catered to by magazines. Write for those.

A magazine publisher is more likely to publish something by a person with a higher degree (MS or PHD) in the field of study, or who is earning that higher degree.

They are also more likely to publish something by someone who has a venue in another media, like radio. Nowadays, possibly a blogger with a huge twitter following might also get published.

There's your algorithm. It works pretty well.
It increases the hit rate.

The parent's post only increases the hit rate of egos. Which, admittedly, in this day and age is great for getting government funding, which in turn is more and more critical to the appearance of success, since the economy is going away.

Out of date info

[ZephyrXero]

She did not purposefully release that this was her pseudonym, so kind of a bad example. There have been numerous news posts today about how she's mad at the PR firm that leaked the info... http://www.npr.org/blogs/thetwo-way/2013/07/19/203548818/book-news-j-k-rowling-very-angry-that-law-firm-leaked-her-name

Re:So propose it to her?

[CanHasDIY]

Why are you mentioning this here? As you say, you've already posted this idea many times to /., and gotten a good amount of feedback.

I'm developing the impression that Bennett Haselton is one of those people who talks, at length, but seldom says anything worth listening to.

After his anti-Fifth Amendment diatribe, taking this pontificating narcissist seriously has become impossible.

There is no way to test what you are proposing

[m00sh]

This is silly. If they had luck back then, it would be wrong to assume they haven't improved their skills. Remember the Matthew effect. By getting that luck, they are now professional writers who spend all day honing their skills while the other mundane details are taken care by others. Those who didn't have luck have to find other way to make money and on top of that manage everything else in life which leaves very little time for developing their writing skills.

Re:Maybe it's the money?

[plover]

Exactly. What good does it do J. Random Writer to know he writes books with an average review rating of 73.8 while J.K.Rowling's average is rated 62.5? Does he think anyone is going to buy his book because it's reviewed 11.3 better? Is Amazon going to push his book to the top of the advertising pile because it's 73.8? No, Amazon is going to push J.K.Rowling's book to the top of the pile because people are buying it.

I mean people are buying 50 Shades of Taupe, and from what I understand the book is full of typos and hackneyed writing, and might score about a 16 when compared to anything by JKR. But the only review that counts is the one the customer opens his wallet for, and money is the scoring token. It might be too bad that we're not a more enlightened society, but that's pretty much the whole game as it's played today, right there.

Talent only gives you a chance

[jd.schmidt]

But Luck makes you a superstar. J.K.R. is certainly talented, no doubt (no book I wrote would get even 1,500 people to buy it!), but there are lots of talented people. Too many for the average person to keep track of, in fact too many for experts in most cases. There are thousands, maybe millions of great works of art I will never have time to appreciate.

Also, there is a certain amount of luck in creating a classic. The number of artists who can create more than one true classic is extraordinarily rare! Even the best have many mediocre works besides their great ones, if you want to produce more than once classic it requires an insane amount of dedication and time devoted to it along with accepting that most of your work will not really be great.

Meta-Vote

[coinreturn]

And if we used your voting method on this article, it probably wouldn't have hit the front page.