Why Good Data Can Be Hard to Find Online

WSJdpatton writes to mention that Carl Bialik has an interesting look at why good data can be hard to find, much less understand, online. He cites a couple of examples, both Google's first-quarter performance numbers and Alexa's revamp of their number-tracking process. "Now Alexa is incorporating other sources of data -- though it says the prior ranking 'wasn't wrong before, but it was different.' Some sites saw big changes in their rankings following Alexa's move: The tech blog TechCrunch said it fell far from its prior position in Drudge Report territory (rarefied air in Web-traffic terms). On Friday afternoon, Drudge Report ranked 545th, compared with TechCrunch's ranking of 1,784th, according to Alexa's new math."

Comments

Alexa? No.

[Slashdot Suxxors]

This isn't exactly on topic, but I think you should give it a read before you make a final opinion on what the article is trying to stay.

Re:Alexa? No.

[jd]

The article and the slashdot story seem to say the same thing - the numbers produced are just numbers out of a hat. They don't represent anything meaningful and indeed can't because the participants are self-selecting and therefore not a random sample of the population. This is obvious and always has been. The popularity of a site (or a TV show or anything else) cannot be measured by any simple means, if it can be measured at all.

Re:Alexa? No.

[TubeSteak]

The article and the slashdot story seem to say the same thing - the numbers produced are just numbers out of a hat. They don't represent anything meaningful and indeed can't because the participants are self-selecting and therefore not a random sample of the population. Even with a random, statistically relevant sample size... the saying "lies, damn lies, and statistics" still applies.

The popularity of a site (or a TV show or anything else) cannot be measured by any simple means, if it can be measured at all. Tivo & other DVRs would suggest otherwise.

Re:Alexa? No.

[Firehed]

Maybe relative tracking can't be done by simple means since it requires participation on everyone's part, but absolute local tracking is trivially easy on any server that supports server-side scripting and has some sort of database access. A couple lines of code at the bottom of your page to insert a new row on a page load and you've got nearly perfect visitor logs that can easily go beyond your standard server logs.

Again, useless for relative popularity unless you have everyone's data. But it still tells you how popular your site is which is great for ego boosting and advertiser stats if nothing else.

(I'd suggest that Google Analytics is going to be a lot more useful in the long run and at least has the potential to provide relative data in addition to the absolute, but anything that relies on client-side scripting is going to give less accurate numbers since clients can disable or screw around with scripting)

70% of good data...?

[JadeAuto]

I read online somewhere that 70% of statistics online are made up. This article seems to prove the point. 4 out of 5 slashdotters agree! ;)

Re:70% of good data...?

[evanbd]

Four out of Five slashdotters? You must be mad. The only things slashdotters can agree on are that they want to marry CmdrTaco and what concert to see on the honeymoon.

Wow, astoundingly obtuse

[zappepcs]

Just observing the Internet and then reading this ... just wow.

Good data is HARD to find ANY FUCKING WHERE, never mind limiting your search to just online. Seriously!

News online? read the same story from 8 sources, form your own opinion. MSM sucks worse.

Scientific data? Well, unless it's peer reviewed, you know it's probably suspect and need to verify it with other data. Damn, even peer reviewed scientific data should be compared to other data these days.

How about Encyclopedic data.. There is wikipedia, but make sure to corroborate the data, right?

Read it in a blog? Check the data before you make up your mind.

Hmmmm this sounds a lot like trying to find good data before the Internets were active. Damn, all that data is proffered up by humans... Humans are not infallible so I'm guessing that data provided by humans is going to be a bit 'not infallible' also.

Where does the assumption that data online should be good data come from? wtf?

No, really.

[v(*_*)vvvv]

The company tracks the Internet habits of users of its browser toolbar ... These rankings have long been criticized ... because Alexa users may not behave like the Internet as a whole. Ya, who in the world uses the Alexa toolbar!?

Re:No, really.

[choongiri]

Nobody here. That's the point.

Good Data can't be found online

[(TK2)Dessimat0r]

Only Bad Lore can.

Slashdot is dying.

[e9th]

Alexa confirms it.

A Good Date

[GalacticLordXenu]

I initially read this as being, "Why a Good Date Can be Hard to Find Online". Hell, I could have told you that! But alas...

Numbers Games Are A Loss

[Jekler]

The reason good data is hard to find online is chiefly a problem with perspective and the models we are using to differentiate good data from bad data. That model primarily relies on the idea that it's all about numbers, or simply that more data is the same as better data. Whenever we come up with bad data, the "quantity model" dictates we just need a larger sample.

This model is directly related to how companies measure TV show quality. The theory is, the more people who watch a show, the better that show must be. This model is so obviously faulty; almost everyone can agree that American Idol isn't even in the same qualitative ballpark as The X-Files, Arrested Development, or Star Trek. The reason the model is faulty is because of the hugely limited scope of the examination. There are a number of variable factors that aren't being considered, such as people own more TVs than when Star Trek was on, and they're mistaking curios interest with enjoyment. Average person will stop and watch a car wreck for roughly the same amount of time they'll play with a yo-yo, that doesn't mean the entertainment value of each is directly comparable, there's a whole different brain process going on in the observers of each, but the model of measuring quantities assumes that two activities which consume the same amount of time are equivalent in all ways.

Back to internet statistics. All this data mining and gathering is designed to ignore the differences in activities, it's only cataloging information for the purpose of what's the same. As the article states, Alexa is always checking for biases. Well the biggest bias in this model is the assumption "in sufficient quantity, all things are interchangeable." It's the assumption that telemarketers and scammers work on, which is why so many people go broke buying into those schemes, because they buy into an assumption which is absolutely wrong.

Many internet business models, specifically data miners, are designed on, assume that 1 million hits is the same regardless of where it comes from. When you consider real factors, having 1 million people see your hand-made chain pouches at a shopping mall is not going to generate the same level of interest as having 1 million people see them at a renaissance fair.

Of course that introduces a whole different problem with assumptions about targeting (I'm not going to get into that, only state that targeted marketing makes the assumption that timing doesn't matter).

In conclusion, you can't play people as a numbers games, People's behaviors (including their online behaviors) are complex and any model which treats people's differences as a child might divide up a bag of skittles by color is going to have a very high error rate.