Slashdot Mirror
"Long Tail Effect" Doesn't Work As Advertised, Say Wharton Researchers
Death Metal writes "In a working paper titled, 'Is Tom Cruise Threatened? Using Netflix Prize Data to Examine the Long Tail of Electronic Commerce,' Wharton Operations and Information Management professor Serguei Netessine and doctoral student Tom F. Tan pull information from the movie rental company Netflix to explore consumer demand for smash hits and lesser-known films. Netflix made its data available as part of a $1 million prize competition to encourage the development of new ways that will improve its ability to introduce customers to lesser-known titles they might find appealing." In short, the researchers say that the Long Tail effect described by Chris Anderson is much less important in the real world than popularly held. Says the article: "The key difference between the opinion of [Anderson's] book and the study by Wharton researchers is how they define 'hits' and 'niches.' In the book, Anderson focuses on the definition of hits in absolute terms such as the top 10 or top 1,000 products, while Netessine and Tan argue that, to take growing product variety into account, one has to define popularity in relative terms, such as the top 1% or top 10% of products, to properly assess the presence or absence of the Long Tail."
That's precisely the point. If the shape is such that a top movie gets only 1% of the market, top movies won't make enough profit to justify hiring Tom Cruise and it's a problem for him.
>If you add an insignificant product to the end of the tail,
>it obviously increases the proportion of market share of the first X% of products.
Yeah, but add a product because it is important by itself. Consider adding Dewar's to a store that has no blended scotch whiskey.
Will Dewar's increase niche sales? No. Will it increase single-malt sales? No. However, with a gaping hole in the scotch section, can you let Dewars die on its own?
Of course not. You must have it. Now that you have Dewar's represented, should you also add Dickface Brand for half the price? That's really the trick of it.
OK, I am not a mathematician, but this paper makes me deeply skeptical.
If the input data is indeed heavy tail (non-existing higher moments) or quasi-heavy tail (existing, but extremely large higher moments) how on earth they can use variance, R^2 and other measures? They may not even exist! And if the input is quasi-heavy tail, then of course they exist, but the convergence time could be arbitrarily long!
I had the unpleasure to work with quasi-heavy-tailed data, and it is really enlightening. You watch the evolution of some metric (e.g.: avg) as the function of incoming data, and you see of course convergence. At least for a while. And then in sudden an extreme outlier comes in, and the avg takes a huge jump! Now if your input is heavy tailed enough, you can be never sure that your measure finally came to rest (converged), or the next jump is just over the corner!
I hope a more educated person clarifies this, I am just an engineer.
The thing that's always struck me about the long-tail effect is that you've got to work it, to get value from it. Just having all the books or films by a particular author / actor isn't enough. You have to use that information and have the intelligent algorithms to guide your website visitors (or maybe "entice" would be a better word) to consider those alternate products. Just saying "Uuh, here's all the other stuff that guy's done" isn't enough, it needs enthusiasm and some knowledge of *why* a visitor might like a particular past work. That's where the gold lies: not in the long tail itself, but how you utilitise it.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
Defining a "hit" as one of the top ten or top 1000 or any absolute number is stupid.
While it sounds stupid, using a top xx% is, in a way, validating the idea of long tail.
Why? Because before Amazon, when book stores are still only brick and mortar, there is only so much physical space to hold top 1000 or however many number of books. Note that this number is fixed, it won't grow because more kinds of books are published.
So having an absolute number of top 100 or top 1000 simply corresponds to the physical constraint that most bookstore can only put so many books on the shelf.
The advantage that Amazon has over physical book stores is that it can hold practically unlimited number of books. So only now, without the physical constrain, we can practically use top 10% instead. This, in fact, proved that there are many more profitable books outside top 1000 (or however many), and that physical bookstores are missing out many sales due to it.
Oliver.
From my own experience, I sometimes get an obscure book because I have a particular reason to get that specific title (be it for the subject or the writer). I listen to obscure music because it sits somewhere in my playlist and the player is on when I'm doing something else.
Reading a novel takes time but I find it no problem to put a book down while I prefer to watch a movie from start to finish in one go. Books on a particular subject I read when I have the time and the interest, or else use them as reference material for when I need to know something specific. Watching a movie requires my time and attention, it's something I plan rather than just listen to some music or read a chapter because I haven't got something better to do.
When I do sit down to watch a movie, I tend to stay on the safe side, try to get the highest chance of being entertained. That may well be with an obscure movie but more often it's with a reasonably wellknown title. But then, I can't remember when I last bought or rented a movie in stead of downloading so my consumption won't show up in any shop's statistic at all.
"I'm not much interested in interoperability. I want substitutability. I want to be able to throw your software out."
This, in fact, proved that there are many more profitable books outside top 1000 (or however many), and that physical bookstores are missing out many sales due to it.
YES.
This is exactly why I've stopped using brick and mortar retailers almost entirely. They carry such a limited selection that it's often a wasted trip.
This goes for video rental stores once they consolidated (in my area) to a chain of "new release-only" stores. I switched to Netflix and have never been back, and have converted many friends to Netflix too.
Music stores, which in my area have never carried anything but the most popular overpriced crap. Now I buy from Amazon or direct from musicians' websites.
Groceries are one of the few markets left worth using brick and mortar stores for. Anything else is just a showroom for cheaper online stores, at this point.
In the entertainment industry we often see an effect where the biggest productions often seem to struggle to break even
It's called Hollywood accounting and has very little to do with actual 'profits'. Small productions might also seem 'unprofitable', once they learn to have their Cayman Island subsidiary charge the project $500k for the producers porta-potty rental. Actual profits are simply funnelled to the desired destination by way of semi-internal charges for rents, distribution, marketing, consulting, etc, etc. As long as your costs are to yourself, nothing really has to make a profit...
And, of course, the monopoly rights industries are nothing like a competitive free market.
Either way I think it's premature to analyse the long tail to any extent. Copyright in itself drives the distortion of the market by encouraging excessive marketing and inordinate market control, creating an economic situation where it's better for the major players that people buy fewer products for more, than more products for less, and they do all they can, with a fair level of success, to push that market shape.
But few retail stores stock more that 50 or 100 current titles, so I think the original idea is quite good. Movie theaters show only 12 or so films at a time, even in big cities. Opening a Brick and Mortar store or theater up to even 1% of "hits" say music/movies that actually made a popularity chart in the last 50 years would be an impressive achievement. Even blockbuster movies like Star Wars or Indiana Jones become "unpurchasable" in a relatively short amount of time.... They've already hit "bargin bin" status in most retail outlets.
Having a surefire way to get at that back catalog would be highly important. The real key is getting business to focus on marketing in "long tail" manner. Something like Netflix is interesting because they are a business that really pays no penalty for keeping extra DVDs in the warehouse.... but how do they get people interested in WATCHING them. I find Emusic to be a similar thing in that area, but again, the hardest part is matching up MORE stuff I'm interested in rather than what publishers are currently marketing. I think Disney has the best handle on it because they republish back catalog in a big way every 10 years or so... making it "new" again to a new group of people. How do you do that for general things like "Bing Crosby" movies or "Rogers and Hammerstein" musicals? Heck, even getting recent Anime published in English in a reasonable time is difficult, or finding material from Electronica/J-Pop scenes due to publishers only wanting to publish "top 10" material.. when the majority of people don't BUY that stuff.. but they all would buy a different part of the top 500 or so songs.
In the book, Anderson focuses on the definition of hits in absolute terms such as the top 10 or top 1,000 products, while Netessine and Tan argue that, to take growing product variety into account, one has to define popularity in relative terms, such as the top 1% or top 10% of products, to properly assess the presence or absence of the Long Tail.
So let me see if I get this:
Anderson says, "Assume a bell curve with a sufficiently large ordinal scale X axis. Select a sufficiently narrow segment in the center. The area under the curve not included in that segment will be greater than the area under the curve included in that segment."
Netissine and Tan respond, "However: Select a segment in the center whose width is a fixed percentage of the ordinal scale of the X axis. Select a percentage which is sufficiently large. Now the area under the curve in the included segment will be greater than the area not included."
For this, Anderson gets a best selling book, and two Wharton academics get a paper out of stating the counterpoint?!?
My brother went to Wharton. He is extremely smart. I know many of his friends from school -- they too are very intelligent folks. Wharton is not a kidding around school, it is pretty hard-core.
My brother and his friends from school mostly make stunning amounts of money. Many of them work for banks in the sorts of positions which, particularly given the recent total failure of banks at their primary wealth-creating task (risk management), might lead one to wonder if they are really responsible for enough sustainable GDP growth to justify their extraordinary compensation. To correct the first sentence in this paragraph, my brother used to make stunning amounts of money. A few years ago this very conundrum led my brother to retire, because he could not live with the disproportion between his production and his compensation. Most of his friends from school are not so infected with ethics.
Seeing this article, and the startling inconsequence of these supposed shining stars of business academia, I am inclined to agree with my brother's conclusion. And to reinforce my belief that we have, over the past 40 years, skewed the distribution of wealth toward the supposed best and brightest business thinkers, and away from all other areas of production, too heavily. Whether we, in chanting the mantra of ensuring that the business analysts and risk managers get fully compensated and motivated regardless of how outsized their compensation may appear to we mere mortals, have pushed the system much too far in that direction at the expense of compensation and motivation for those who are not business analysts and risk managers. Whether there may be forces at work which already influence cashflow in their direction, and our supposed levelling of the tax code has instead removed the normalizing force that was preventing an unhealthy portion of GDP from flowing to those with abnormally high influence on the flow of GDP -- but who, recent evidence suggests, are not really so extraordinary in their contribution to it.
Stop-Prism.org: Opt Out of Surveillance