Slashdot Mirror
Using Twitter Data To Approximate a Telephone Survey
cremeglace writes "A team led by a computer scientist at Carnegie Mellon University has used text-analysis software to detect tweets pertaining to various issues — such as whether President Barack Obama is doing a good job — and measure the frequency of positive or negative words ranging from 'awesome' to 'sucks.' The results were surprisingly similar to traditional surveys. For example, the ratio of Twitter posts expressing either positive or negative sentiments about President Obama produced a 'job approval rating' that closely tracked the big Gallup daily poll across 2009. The analysis also produced classic economic indicators like consumer confidence." By averaging several days' worth of tweets on presidential job approval, the researchers got results that correlated 79% with daily Gallup polling. Lead researcher Noah Smith said, "The results are noisy, as are the results of polls. Opinion pollsters have learned to compensate for these distortions, while we're still trying to identify and understand the noise in our data. Given that, I'm excited that we get any signal at all from social media that correlates with the polls." Here is CMU's press release.
Has anyone read Neal Stephenson's novel "Interface"? We're getting oh so close to it.
No thank-you, please take me off your list... click
EA David Gardner -"... but the consumers have proven that actually what they want is fun."
It seems that the age demographics of twitter users wouldn't be very representative of the population as a whole.
I'm guessing it will take no more than a month for a combination of "conservative" and "progressive" blogs to rev up their teams of dittoheads to start flooding Twitter with politically themed messages, thus totally skewing the results. Same principle as Google-bombing, I guess. As someone who already views Twitter as almost entirely content-free, I can't say I'm particularly dismayed by this possibility. . . but anything that encourages the self-absorbed political zealots of this country can't possibly be good.
Just like traditional pollsters, social media researchers will have to address how representative Twitter users are of the general population. And unlike telephone surveys, small groups of people can wildly skew the results of Internet data,
Yes I did STFA (Skim the fucking article).
It mentioned the two main problems I see with this, cheating the system and whether twitter really is a large enough sample and a random enough sample to be considered a viable alternative.
Twitter has a whole range of people who don't actually use the damned thing. As with any poll though, people are going to say that the minority polled is what everyone says.
"The American people want to do x! Our poll says 80% of the American people want it!" No. No it doesn't. It just means 80% of the people you polled want it.
I despise how easy it is to use statistics and polls to manipulate people.
I've collaborated on research using Twitter traffic as a predictor so I applaud their efforts, but a 79% correlation with telephone responses is not as high as it sounds. For example, the minimum acceptable correlation for interrater reliability is typically 80%.
Put simply, the Twitter data can only account for about two thirds of the variation in phone responses. That's useful but there's still a lot of unexplained variance -- we have a long way to go.
I bet you could use this same system to assemble data about all kinds of interesting subjects. TV show viewership, web application downtime, top news articles by reader interest, etc. Really cool.
That is NOT nearly correlated? That is BARELY correlated. And will not get you meaningful results. This will also stop being meaningful as soon as it is publicized people are paying attention to the content of the tweets.
Cripes.
The noise coming from one group of twits is the same as the noise coming from another group of twits.
Film at 11.
Science advances one funeral at a time- Max Planck
As soon as something like this comes to light it's only a short matter of time until turfers screw it up. Turfers are like spammers, as soon as there's a new medium they abuse it into uselessness.
The preceding post was not a Slashvertisement.
...were they doing a poll on windows 7?
As someone who spends a lot of their time working to uncover endogeneities in statistical analysis, I feel that analyzing Tweets will never be a viable measure of general American opinions.
Remember when The Literary Digest predicted Alf Landon would crush FDR in the 1936 presidential election based on a poll of its subscribers? Okay, you don't *remember* that, but you've probably heard of it. Same problem here.
The readers of Literary Digest were not representative of the average American in 1936.
The users of Twitter are not representative of the average American in 2010.
Twitter polling is no better than straw polling, which is usually worse than nothing.
Umm... from TFA:
Likewise, both the Twitter-derived sentiments and the traditional polls reflected declining approval of President Obama's job performance during 2009, with a 72 percent correlation between them.
Okay... not a great correlation, but let's continue....
But the researchers found that their sentiment analysis did not correlate as well with election polling during 2008. For instance, increased mentions of "Obama" tended to correlate with rises in Barack Obama's polling numbers, but increased mentions of "McCain" also correlated with rises in Obama's popularity.
WTF? Is all of this built on how many times "Obama" or "McCain" is uttered on Twitter? And, given the obvious skewed demographics on Twitter (i.e., younger people, which tended to poll way toward Obama), increased conversations about McCain probably were bad in general.
Well, how do they explain this? Ah, the next sentence....
Improved computational methods for understanding natural language, particularly the unusual lexicon of microblogs, will be necessary before Twitter feeds can be reliably mined to predict elections, the researchers concluded.
Ah yes, the "unusual lexicon of microblogs," which probably consisted of sentiments like "I luv Obama!" and "McCain too old - WTF?"
Perhaps if they bothered to measure more than "mentions" of a candidates name, the data might have some (albeit still vague) meaning...
If this is the best stuff from the study which they actually mention in a press release, how much crap results are they not reporting?
hopefully these polling algorithms will take into account when i'm saying "microsoft doesen't suck" that i'm actually meaning that they do suck, and not attempting to derive my meaning from my literal translation. If they profile my use of irony they should be fine.
We're in the midst of attempting to figure out how the computer went from being a mathematical tool used for calculations, to a communications device used for elemental interaction. These days, the machines are used more often to interact than to calculate--that is, unless the calculation is need for an interaction.
This transition from a practical business tool to a communications device has caused a dislocation in society that may take years--or decades--to settle. It's possible that all the rules of the past regarding social change are not applicable when a mechanism designed for one particular part of the social framework, developed and evolved over millennia, jumps the fence and invades an environment that has no ability to properly deal with the new mechanism. It's like when a disease vector is introduced into a population with zero immunity.
Pazzuzu rulez your underware drawer!
This is what we may be dealing with. The computer functioning as a disease.
Initially, computers were used for calculations. The first intended purpose was for artillery trajectory calculations--hardly a noble purpose, but certainly a practical one. In the early days, computers were described as electronic bins. They were also demonized for their potential to become a giant, electronic dictator, but those ideas tended to be marginalized by the argument that all computers were nothing more than a big, dumb calculator.
As the desktop computer revolution developed, the devices' uses were inevitably based on some aspect of calculation. Spreadsheets were the perfect example. At the time, the only communication aspect of computers was the fact that they could double as powerful aids to word processing software.
By 1979, however, modems and networks were making in roads. They made it possible for computers to talk to each other in some crude way. That was the beginning of the end. The computation aspect of computers continued to grow, but it was the networking aspect that was the disease vector, so far as social upheaval is concerned. You can figure out the rest of the networking timeline. It began 30 or more years ago--40 years, if you want to count the invention of Arpanet in 1969.
Once the computer jumped the barrier into the communications side of the social structure in a big way, it quickly overwhelmed the immune system. When all was said and done, the culprits were Ethernet and TCP/IP.
When real analysis is done on the computer revolution, the conclusions may not all be positive. These machines haven't made our lives easier--they've just changed them. Computers have made things like calculating change simpler. The cash register tells the cashier how much change to give. The cashier is obedient. Most people cannot make change at all anymore.
In the same way that the computer calculations have made the public dependent on machines, the communications-oriented computer is making us dependent on technology for basic human interactions. As this progresses, most social interactions will be performed with the aid of computer networks. Most people will meet online. It will become more trusted than meeting in-person. Nothing is stopping this trend. There is no immunity from the disease. The best society can do is issue the lame warnings that were trumpeted years back, about the dangers of meeting strangers online. That didn't fly.
An early victim of this process was newspapers and magazines--that is if you don't also count the decimation of the moral fabric of society caused by the torrent of free and easily accessible porn. There was no immunity for that. There was no practical way to stop it. It continues, unabated.
Some have argued that the decimation of newspapers began with the introduction of radio and TV news. The declination graph supports this somewhat, but the fact is that TV could not "put away" newspapers. Newspapers were fighting boldly in the 1980s and 1990s. There was no knockout punch delivered by TV news.
But with the Internet came a new opponent that had not
"I'm excited that we get any signal at all from social media"
s/excited/amazed/
OK - Maybe I don't get it - WHY are we using a Twitter survey to be representative of a telephone survey ? WTF - are we stupid ? A Twitter survery is one thing; a telephone survey is another... Never the twain shall meet... MS
Wow! You mean modern technology has progressed to the point where it can approximate the results of totally inaccurate guesswork derived from people so sad that they'll actually stay on the line when phoned up by a total stranger?
Truly, technology amazes me. (Or humanity does. I can never quite keep that straight.)
HP already did something very similar. You can google predicted box office success twitter, or simply view HP's report. http://arxiv.org/PS_cache/arxiv/pdf/1003/1003.5699v1.pdf So congrats on being behind CMU? I guess the concept is slightly different making it new research, but not new enough to really merit a whole lot of discussion.
measure the frequency of positive or negative words ranging from 'awesome' to 'sucks.'
So what about "Obama doesn't suck!" or "McCain is as awesome as my grandmother who isn't awesome at all!"
Damn.
'Political power grows out of the barrel of a gun.' - Mao Tse-tung
This is why you can't have self-selecting samples.
Also, this applies a lot more broadly. They call it Goodhart's Law. For example, there's an example about a fictitious Communist factory making nails. When they were told that they had to make X nails, they made small and useless ones to meet the goal. When they were told that they had to make them by weight, they made a few ridiculously large ones.
There weren't enough mentions of Apple either in the summary or the comments posted.
To have a right to do a thing is not at all the same as to be right in doing it
i mean computers says, president obama is doing bad, he shall do more of "the other thing"... computer says.
I bet this is already implemented in people survellance software.