Slashdot Mirror
The Real Problem With Alexa
As the defacto 'Guy in Charge' of a reasonably large web site, I am routinely asked questions by a variety of people that lead inevitably to Alexa. It might be a question from my Boss at SourceForge about traffic. Or it might be a sales guy asked by a possible advertiser why some other random website is bigger or smaller than Slashdot. Most often it's a random reporter doing background for a story that has nothing to do with Slashdot. Why I'm considered an expert is very confusing, but why they always regard Alexa rankings as meaningful is even more so.
Here's the problem: Alexa doesn't work because of who will install it, and perhaps more importantly, who won't. Let's start with a place I'm very familiar with: Slashdot readers. Until recently Alexa didn't work on Firefox... instead only IE users participated. On the internet as a whole that's fine: like 80% of users run IE. But on Slashdot only like a quarter of you do.
What about re-installing the plug-in after you update your browser? When Firefox 2.0 came out, almost a third of Slashdot readers upgraded within a few days. You upgrade Minor Firefox releases overnight. Even IE users of Slashdot update relatively fast, from 6 to 7 or even minor revisions. New versions often break old plug-ins. When you get that alert that a plug-in is out of date do you just forget about it? I know I do. And that's not even counting clean OS installs. But if I went to random non-technical friends and family installations, I frequently see versions of software so dated it makes me cringe.
And that's not even talking about the fact that Alexa's toolbar is pretty much spyware. How many Slashdot readers are giddy to install spyware? You either? Big surprise. Because of who we are, and what it is, our population will self select out of consideration.
Did you know Alexa excludes SSL? How many etrade users do you think there are? Now personally I'm glad that they aren't tracking my browsing at my credit card company, but it's just another factor reducing accuracy.
Equally perplexing is the accounting of iframes. Let's look at someone like double click's alexa rating. Now it's hard to say, but I don't think I've ever visited their website. Have you? But according to Alexa, they have nearly a 1% share of the internet. I'd tend not to believe it... but they have iframes on zillions of web pages and counting those sure would account for this huge ranking. What about all those badges for the popular social networking websites? What influence are those iframes having on Alexa rankings? Alexa's FAQ says they don't count, but I'm skeptical.
In Fact, Alexa KNOWS that it is a flawed metric for measuring. Have you ever tried actually looking up alexa on alexa? Unsurprisingly, it is unavailable. Why? Visitors to Alexa.com would be the most likely of any user population on-line to have installed their plug-in. I don't know what their 'Rank' would be, but I bet it clearly would be an apples to oranges comparison against ANY other site on-line.
Of course who do you think actually will go out of their way to install something like this? I have a good guess... if you are obsessed with acronyms like SEO or terms like PageRank you are very likely to care very much about these things. I spend a real percentage of my week dealing with people flooding my systems with garbage content designed to screw with these ratings. And you know they all have the toolbar installed so their zillions of worthless spam websites are being counted.
This problem has parallels elsewhere of course: The Nielsen ratings struggle to account for PVRs. Since you got a TiVo, when was the last time you watched "Live" TV? This is part of why Science Fiction shows struggle on TV... scifi fans are early adopters. So we stopped getting counted and our favorite genres are butchered by networks and lost to the void. PVR users tend to be wealthy (those boxes are expensive) and educated. Now I'm not saying that the dumbing down of TV is exclusively the fault of Tivo, but it sure didn't help that we weren't being counted as excellent "Smart" TV shows get canceled while we keep getting more seasons of Survivor. Who we are and how we live causes us to not be counted, and this has unintended consequences.
So what do we do? I wish I had a good answer to this. My first suggestion would be that if anyone mentions Alexa to you that you freak out and go on a 5-minute rant about how Alexa is stupid and anyone who is using it to seriously make a business decision should be fired. It doesn't actually help, but i estimate that every time I do this, I burn the same number of calories as I might on an elliptical trainer. I assure you the beer gut ain't getting smaller on its own.
Alternatively you could just install the toolbar on every machine you can find and skew the numbers ridiculously towards people that are likely unrepresented. Of course, the conspiracy theorists amongst you will just bitch that I'm trying to fudge Slashdot's own rankings in a system I'm claiming to hate. But that only helps proves my point... the conspiracy theorist is a demographic strongly represented on Slashdot that is unlikely to trust this software. We all ignore a broken status quo "Gold" standard that would fail a 100 level college science class on the grounds of flawed methodology. And this only leads to us not being counted.
That's all true, but unless someone's got a better alternative, it doesn't matter.
It isn't surprising that people who spend money on advertising want to have some metric by which to predict (estimate, guess, what-have-you) the impact of each dollar spent on web advertising. Assuming the people spending the money are, as a class, either stupid or ignorant is a mistake. Odds are good that many of them know that Alexa is flawed, but also consider any information better than nothing. If nothing else, Alexa rankings demonstrate the relative popularity of a web site among Alexa participants - which is at least a concrete demographic, and the stats are inarguable on that basis.
What's being missed is that there's a fundamental problem, here. Populations which refuse to share information with such aggregators will always self-select against representation. It's no different, really, than stating that populations who do not vote self-select against being represented in government. That doesn't stop us from using elections as a way to select people into government.
In the specific case of slashdot selecting against itself, it's debatable whether we're a demographic many organizations would even want to target (with web advertising) if they could. How many comments on how many stories have included someone claiming that he's either unaffected by or negatively affected by advertising? That he's less likely to buy a product he sees advertised? Broader yet, how do you suppose the median number of lifetime banner ad clicks for the slashdot user compares to that of the web-using population at large?
I posit that we pose a particularly galling challenge to marketers. On the one hand (if you'll allow me a bit of net-cultural hubris), we're a demographic of above-average intelligence, above-average income, with an above-average tendency to spend money on brand new technology, and who have an above-average impact on what other people will buy. On the other, we refuse to share our habits with "big brother," we're easily offended (eg, we hate proprietary formats solely because they're proprietary), comparatively hard to bamboozle, and have a cultural predisposition towards "free" (both beer and speech). That is, on the one hand, we're a fantastic demographic to succeed with, but on the other, we're a tough nut to crack.
The point is that Alexa is flawed, without a doubt. But it seems more flawed from the point of view of a group which deliberately makes itself all but impossible to measure. And frankly, if we're not willing to provide the information necessary for advertisers to make informed choices, we're going to continue to be ignored, both on the web and on television. (Yes, I do realize that Nielsen is specifically flawed with respect to DVRs - but even if they weren't, how many members of this site would voluntarily install habit-tracking software on their TiVo? How many members of this site would call for a boycott of TiVo if it installed it for them?)
Reality has a conservative bias: it conserves mass, energy, momentum...
That's probably why it's filed under 'Editorial'.
Alexa targets a demographic which are more likey to click on banner ads and buy the junk which they advertise. So for the advertisers targeting those demographics I'm sure it works out ok.