Slashdot Mirror
Why Good Data Can Be Hard to Find Online
WSJdpatton writes to mention that Carl Bialik has an interesting look at why good data can be hard to find, much less understand, online. He cites a couple of examples, both Google's first-quarter performance numbers and Alexa's revamp of their number-tracking process. "Now Alexa is incorporating other sources of data -- though it says the prior ranking 'wasn't wrong before, but it was different.' Some sites saw big changes in their rankings following Alexa's move: The tech blog TechCrunch said it fell far from its prior position in Drudge Report territory (rarefied air in Web-traffic terms). On Friday afternoon, Drudge Report ranked 545th, compared with TechCrunch's ranking of 1,784th, according to Alexa's new math."
The article and the slashdot story seem to say the same thing - the numbers produced are just numbers out of a hat. They don't represent anything meaningful and indeed can't because the participants are self-selecting and therefore not a random sample of the population. This is obvious and always has been. The popularity of a site (or a TV show or anything else) cannot be measured by any simple means, if it can be measured at all.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
Just observing the Internet and then reading this ... just wow.
Good data is HARD to find ANY FUCKING WHERE, never mind limiting your search to just online. Seriously!
News online? read the same story from 8 sources, form your own opinion. MSM sucks worse.
Scientific data? Well, unless it's peer reviewed, you know it's probably suspect and need to verify it with other data. Damn, even peer reviewed scientific data should be compared to other data these days.
How about Encyclopedic data.. There is wikipedia, but make sure to corroborate the data, right?
Read it in a blog? Check the data before you make up your mind.
Hmmmm this sounds a lot like trying to find good data before the Internets were active. Damn, all that data is proffered up by humans... Humans are not infallible so I'm guessing that data provided by humans is going to be a bit 'not infallible' also.
Where does the assumption that data online should be good data come from? wtf?
Support NYCountryLawyer RIAA vs People
Nobody here. That's the point.
The reason good data is hard to find online is chiefly a problem with perspective and the models we are using to differentiate good data from bad data. That model primarily relies on the idea that it's all about numbers, or simply that more data is the same as better data. Whenever we come up with bad data, the "quantity model" dictates we just need a larger sample.
This model is directly related to how companies measure TV show quality. The theory is, the more people who watch a show, the better that show must be. This model is so obviously faulty; almost everyone can agree that American Idol isn't even in the same qualitative ballpark as The X-Files, Arrested Development, or Star Trek. The reason the model is faulty is because of the hugely limited scope of the examination. There are a number of variable factors that aren't being considered, such as people own more TVs than when Star Trek was on, and they're mistaking curios interest with enjoyment. Average person will stop and watch a car wreck for roughly the same amount of time they'll play with a yo-yo, that doesn't mean the entertainment value of each is directly comparable, there's a whole different brain process going on in the observers of each, but the model of measuring quantities assumes that two activities which consume the same amount of time are equivalent in all ways.
Back to internet statistics. All this data mining and gathering is designed to ignore the differences in activities, it's only cataloging information for the purpose of what's the same. As the article states, Alexa is always checking for biases. Well the biggest bias in this model is the assumption "in sufficient quantity, all things are interchangeable." It's the assumption that telemarketers and scammers work on, which is why so many people go broke buying into those schemes, because they buy into an assumption which is absolutely wrong.
Many internet business models, specifically data miners, are designed on, assume that 1 million hits is the same regardless of where it comes from. When you consider real factors, having 1 million people see your hand-made chain pouches at a shopping mall is not going to generate the same level of interest as having 1 million people see them at a renaissance fair.
Of course that introduces a whole different problem with assumptions about targeting (I'm not going to get into that, only state that targeted marketing makes the assumption that timing doesn't matter).
In conclusion, you can't play people as a numbers games, People's behaviors (including their online behaviors) are complex and any model which treats people's differences as a child might divide up a bag of skittles by color is going to have a very high error rate.