Will Cellular Phones Skew Survey Results?

FriedEgg writes "Recently, many people have started to forgo traditional landline telephones in favor of cellular phones only. This presents a problem for telephone pollsters who are prohibited by the FCC from calling cell phones with automated equipment, and from calling people for whom receiving the call costs money. While they recognize the exclusion of cellular only users can skew their results, they're not yet sure how much... because they're unable to survey cellular only users to find out their demographic information. Some evidence does indicate the frequency of cellular-only is highest among 18-24 year olds, traditionally the hardest to survey anyway. If the problem grows worse, it's possible we could end up with a "Dewey Defeats Truman"-like situation where the telephone poll results were skewed because Truman supporters were less likely to own a telephone."

Comments

For Those...

[PakProtector]

...Of you who slept through History class, 'Dewey Defeats Truman' refers to the 1948 presidential election in which it was projected a man named Dewey would defeat encumbant president Truman, due to skewed mail-in poll results.

Link is to Historybuff for more info. Dewey Defeats Truman

Re:For Those...

[Tim C]

I'd suspect that the fact that you're an American has something to do with why you were taught about it in your stats class.

I'm from the UK, and have had a fairly extensive mathematical training, including dedicated theoretical statistics classes. This was never even mentioned.

It's hardly surprising, though - there are plenty of ways of addressing the topic, why choose one poll from (then) 45 years ago that took place in another country, when there are other, more recent examples to use?

Re:For Those...

[PakProtector]

I concede your point.

I guess I was being rather America-centric. I apologize.

The results are bogus already ...

[Dark$ide]

Aren't the results of any phone survey skewed already. If someone calls me unsolicited I'll tell them to go away impolitely.

Haven't the Americans just got a 50 million list of folks who don't want to be called.

Do the folks who fit those two categories have a common demographic?

Re:The results are bogus already ...

[bangular]

The do not call registry doesn't apply to pollsters. So they can still call. You are right though, what kind of freak actually takes phone polls (unless they directly apply to something important to you)? I don't even hang up on them anymore. I take the time out of my day to give them fake information. "As a Florida resident, what theme park do you most frequent" "Gatorland!"

Re:The results are bogus already ...

[redtape]

Unfortunately, (at least from my point of view), political pollsters are exempt from the "do not call" list, as are charity organizations.
And for those who made earlier comments about liberal vs. conservatives with cell only access, I know of several political conservatives (30s and 40s) with only cell phones, and numerous luddite liberals with only landlines. I really doubt that political affiliation or leaning has much to do with this, it is more of a comfort level with the technology. Yes, that seems to skew things toward the younger more often, but many of the younger generation are more conservative than their parents.

Caller ID and call screening already do that

[WuphonsReach]

I would have to say that there is already an effect from the number of people with Caller ID or who screen their calls with answering machines. I know I don't bother picking up if there's no caller info or it says "GALLUP POLL".

The question for the market research is:

Does the universe of people who have cell phones correlate to the universe of respondents that I'm trying to reach or does it introduce bias? (I know just enough to get myself in trouble here...)

e.g. if 55% of your population is A, and 45% is B, and the cellphone population is also 55% A / 45% B, then it will have no impact on the results. OTOH, if the cellphone population is 75% A / 25% B, then there is going to be bias that will have to be corrected for in order to extrapolate back to the main population. (Guaranteed there is some wrong terminology there...)

Re:Caller ID and call screening already do that

[HarveyTheWonderBug]

There is already a bias. In a typical poll, about 1000 people are contacted. Here is an example. Note that an accuracy of 3% is quoted, for such a small sample. What are the odds that calling randomly people, you will find at home young or active persons ? Pretty small. And now, from the article, we learn that:

"A recent Sprint Wireless survey of more than 500 college students found that half are cell-only customers"

Oops, the odds of the pollsters to have 18-24 year old in their sample have just been halved...

I'd like to see the raw data of such polls, but I imagine (correct me if I'm wrong), that in their sample of ~1000 people, they have maybe 20 18-24 year old. Assuming this population represent 20% of the potential voters (IANAD), they would have to correct their answer by a factor 10.

Makes you wonder where does the 3% accuracy comes from...

Won't matter much for elections

[Anonymous Coward]

The 18-24 demographic is the one least likely to vote anyway

is it *so* hard to take a hint?

[tuxette]

People don't want to be called and pestered by pollsters. The only people that don't seem to understand are the pollsters.

Some excerpts from the first article:

In an unscientific poll last month by CNN and Money magazine, 41 percent of more than 11,000 voluntary participants said they would be inclined to pull the plug on their land lines and go entirely wireless.

Unfortunately no reason is stated, but I have a feeling that "so these stupid pollsters don't call and pester me" is one of the leading reasons.

A recent Sprint Wireless survey of more than 500 college students found that half are cell-only customers.

No kidding, Sherlock. Think about it. Students are always on the move, changing addresses frequently. It is MUCH easier to have a cell phone only than to deal with changing land lines every 6 months.Students are also the demographic group with too many other things to do and think about than answer some stupid survey.

"That makes it an opt-in situation where the cellphone user somehow signs up to allow researchers to call," she said. "Who's going to do that?"

Like....DUUUUUUH! Get a clue already! YOU'RE UNWANTED!

'Cept for one thing....

[telstar]

"If the problem grows worse, it's possible we could end up with a "Dewey Defeats Truman"-like situation where the telephone poll results were skewed because Truman supporters were less likely to own a telephone."

• Telephone polls are meaningless vehicles for polling agencies to generate meaningless statistics to sell to whoever's paying their bill. Aside from potentially swaying people that are on the fence and whose minds are as maleable as chewed bubble gum, in my experience telephone polls aren't used as conclusive findings ... but as rough indicators and estimates of the current state of whatever issue is being polled.

karma, karma, karma, karma, karma chameleon....

Re:'Cept for one thing....

[loyalsonofrutgers]

Telephone polls are meaningless... Alright, let's try this one on for size Let's say there is a close race going on, and one of the candidates is "actively and happily" married. The other candidate is single. If a polling firm was to call a lot of voters and ask this question "If you were told that one of the candidates in the upcoming [office] race was a homosexual, would that impact your decision?" Now let's say that this race is taking place in Alabama. This is called push polling. It can have a very real impact on races, depending on the severity of the suggestion ("if you found out that [candidate x] used crack/crack cocaine.." etc).

Re:'Cept for one thing....

[rifter]

" in my experience telephone polls aren't used as conclusive findings ... but as rough indicators and estimates of the current state of whatever issue is being polled."

Yes, but politicians routinely rely on polls for forming policy. Maybe they shouldn't, but they do.

When the issue at hand is something like, "Should cell phone usage be permitted while driving?", only polling land-line customers may give a tremendously skewed view of the public's opinion on this issue.

Personally, I think that phone polls are stupid, intrusive and almost completely spurious. If you really want to get polling information about political stuff, use the internet.

Now, of course the results will be skewed because only people who are informed and interested in the issue at hand will vote. BUt that is exactly the point. IN this case those are the only people who matter, becasue they may vote. If you can't even click on a web poll, you probably won't vote so who cares what you think? The telephone poll would have called someone and woke them from their slumber or interrupted their dinner, in which case an angered respondant might give either whatever answer they think will get rid of the pollster quicker or the exact opposite of what they think the pollster wants (in oorder to get revenge on the pollster).

Now some "liberal" will probably whine that internet polls are unfair to the poor and especially harmful to the elderly, women and minorities (to say nothing of the damage to the children!) like everything else under the sun. Personally, I don't believe a word of it. There are homeless people in this country who were on the internet sooner and more often than I and I am a pretty wired guy. There are various access points for getting online. IF you care, you will go there. If you don't care enough to go there, you will likely not care enough to show up at the ballot box and vote me out of office for voting differently than you might have liked on that issue.

Does it matter ?

[Krapangor]

From a mathematical point of view most surveys are fucked anyway. Biased data collection, misunderstood mathematical rules and "correction formulas" if the results don't seem to fit.
The problem is that many of the people in this business aren't mathematician and if they are then they are at best only second rates ones.
This poblem doesn't appear only in public survey - in fact most "statistics" in natural and political sciences are usually fucked. Most mathematicians don't care - we know that significance tests don't really give any viable results etc. So if people aren't interested in maths or aren't willing to invest time to understand it, then let them believe their rubbish.

Nevertheless all this whining about "skewed" surveys is meaningless, it's like complaing that the moon is made from bad tasting cheese.

Re:Unfortunately Polls Sometimes Dictate Results

[ingenuus]

Perhaps you are referring to instant runoff voting?

"Instant runoff voting (IRV) is a voting reform that asks the voter to rank the candidates in order of preference."

The customers think it matters...

[tuxette]

...otherwise this business wouldn't exist.

And if you think those in the business are bad at maths, think of who the customers are. Politicians and the such. Notorious maths flunkies. Show them some fancy pie charts and percent signs, especially those in their favor, and they're your best friend forever.

Boo f-ing hoo!

[glassesmonkey]

Seriously, cry me a river. So, politicians won't know what to think. TV execs won't know what good shows to cancel next. Phone polling is the equivalent of jury duty and we know who is sitting on juries in this country. So next time a telemarketer calls, just tell them "I love chinks."

(before people get all politically upset do a little research, I can't help it that you don't watch Conan O'Brien or even have a sense of humour)

The obvious answer

[anon*127.0.0.1]

Why can't the wireless companies set up a way to allow the caller to pay the charges? Politicians would get more accurate surveys, wireless companies would get another revenue stream, polling companies would finally be able to reach those elusive cell-only customers... everyone wins!

Well okay, the consumer gets screwed with a bunch of survey calls that they don't want... but really, how important are the consumers anyway?

Re:The obvious answer

[shani]

In Europe the caller pays. It's a Good Thing(tm).

The only trick in America would be that there would ahve to be some way to alert the caller to the type of charges they are about to incur.

Re:Yes.

[October_30th]

To my mind, one should always have a land-line for emergency calls.

I don't want a dead battery or a faulty link station to block my call when someone is having a heart-attack, for instance.

Would you like to take a survey?

[shadowcabbit]

I'm somewhat reminded of the old Animaniacs skit from years ago.

Seriously, I get a lot of calls from telemarketers and poll people. They annoy me more than other people because I sleep during the day, but for the most part I accept it as a consequence of being in the phone book. Anyone can look up my name, phone number, and address in the local white pages, run to a terminal, and google one or all of those to bug me.

Every once in a while if the poll topic sounds interesting I'll do the poll-- well, OK, so I did it twice in the last six months, and really only because the interviewer sounded cute-- but the point is that I sometimes do the telephone polls in an attempt to influence the bias one way or another. If it's a poll I feel strongly about (like, say, the grocery stores in my area-- all of which are crap, thank you very much), I'll do it with the attempt to get something better in the area. If it's about gaming or electronics or somesuch, hell yeah I'll do it-- I like talking about games, and if you say that doing the survey gets me free stuff I'll ask to do it twice.

The point is that not everyone unconditionally says "fuck off" to anyone who doesn't get recognized on their Caller ID. So telephone polls are skewed, simply because the people who do them anyway are either too dumb to say no or think they can change the results with it. Which one of these two categories I fall into is left as an exercise for the reader.

Poster gets it semi-right, Film @ 11

[Fortunato_NC]

The actual Dewey Defeats Truman headline was based more on early election returns than skewed polling. The famous error made concerning polling in the 1948 election was that Gallup simply stopped doing polls two weeks before the election and proclaimed that Dewey would win, not taking into account the massive sea change in public opinion that can occur over a two week span.

The famous example of a poll gone bad because of telephone ownership statistics was the poll used by the magazine The Literary Digest to predict the 1936 presidential election between FDR and Al Landon. The magazine mailed ballots to 10 million individuals with listed phone numbers and/or car registrations, and tallied up roughly 2 million returned ballots. Based on the results, the magazine predicted a Landon victory, while in fact the result was a 46-state landslide for FDR (and remember, there were only 48 states in 1936)!

The sample error in this survey was that telephone ownership and car ownership did not correlate to likeliness to vote in 1936. In 1936, at the height of the Great Depression, telephones and cars were luxury items that few people besides the wealthiest Americans could afford. The poor and lower class who were more likely to vote for FDR were not even sent ballots, so there was no way for their voices to be heard in this survey.

The impact of this flawed survey was such that within a short time after the publication of the survey, The Literary Digest would go out of business.

Re:Poster gets it semi-right, Film @ 11

[sidecut]

Thank god you corrected this! I just heard about this on Nova. That's a fascinating story.

As for the "Dewey Defeats Truman" episode, my understanding is that this was exit polling conducted by the Chicago Tribune in a massive, first of its kind, nationwide effort. Exit polling, as I'm sure you know, is generally highly accurate, since it reflects actual voters, not people who claim they're going to vote some months or weeks hence (or not).

My father worked at the Tribune in 1948 and described to me the massive bank of telephone operators they had there, with stringers throughout the nation phoning in exit poll results, which were then tabulated by electromechanical adding machines. It was a very labor intensive effort.

Anyway, despite its apparent "failure" -- i.e. in predicting the winner -- I believe that the Tribune's use of exit polls was actually looked at as quite a success, because it did produce good numbers. Their problem wasn't the data they received from exit polling, it was hubris: they looked at the data, which showed a very close election, and decided to predict the winner despite the closeness. They gambled, and "lost," but didn't lose much.

Tell that to the people in Florida...

[Raul654]

...whose election was decided by just over 500 votes. That's what, a small high school?

Re:Yes.

[Gordonjcp]

It's never been a problem for me. If I'm *really* stuck, there's always telephone boxes. But I've never had my battery go flat at a critical time (and I keep a charging lead in the car anyway), and the only time I've ever had network problems was when the land-line network was tango uniform too (busied out at New Year).

As I say, I can't really see a need for a landline.

And some pictures...

[VValdo]

Surprisingly, the parent didn't post any links to the famous picture of the debacle. Well, I guess it turns out there are a few of them.

Here are three, including a wide shot I hadn't seen before..

W

The problem is big and real

[dybdahl]

The article doesn't provide many numbers, so I better provide some. In Denmark, there are more cellular subscribers now than fixed line subscribers. Of the 11 houses on our street, 2 only use cellular phones, and one of these has ADSL but no fixed line subscription.

In my nephiew's class, 2 families out of 24 don't have a phone number that belongs to a fixed line.

Currently, a fixed line costs 255 euros to create in Denmark, and 16 euros a month. It's cheaper per minute than a mobile phone, unless you call a mobile phone - in which case the mobile phone is cheaper per minute. Add to that the costs of buying a phone.

A mobile phone can be bought very cheap - used phones are normally 15-30 euros, there is no subscriptions fees and it's now down to 9 cents a minute outside workhours and down to 11 cents a minute in workhours. All prices include the Danish 25% Value Added Tax.

So in general, mobile phones are cheaper for most people, and more and more people are abandoning their fixed lines. The skew is already here and it's growing.

Who Cares ?

[CmdrGravy]

Who cares if pollsters can't get accurate results, although newspapers, TV seem to love polls I'm sure that most people couldn't care less what the polls say and are happy to wait until the results of ( whatever ) are announced offically.

Re:Who Cares ?

[sidecut]

Much polling is done not for public consumption, but for private (read: corporate or political). I'd be willing to bet that the majority of polls conducted are never published. So who cares about the accuracy of polling? The organizations or political candidates who paid bigtime to have these done. Decisions that can be worth millions of dollars are based sometimes on polling data. Poll results are a form of business or political intelligence. Certainly in war time we all appreciate the value of intelligence. The stakes aren't as high in the polling biz, but that doesn't mean those stakes are zero.

I agree -- participating in a poll is annoying and time-consuming. And the news media perhaps do put too much emphasis on the horserace nature of political races, based on frequent polling results. But I believe that's market-driven. People want to see that polling data, so they publish it as often as they get it.

And People who read news & blogs online not po

[Cryofan]

Because often they are using their phone line to go online via their dialup ISP connection. These people are those who spend MOST of their leisure online reading news, blogs, and doing other leisure reading and reading, ALL online. And at night or at other times, they download books and music from Kazaa or Usenet. Thus, they cannot be reached by phone by either parent, friend, or pollster. Thus, the Always-Online political persuasion is not reflected in political polls, such as Presidential race polls (Go Kucinich!) and other polls.

I propose that we Always-Online types are often of radical political persuasions. And because our opinions are not reflected in the polls, the polls come out looking LESS radical, and more mainstream than America really is in reality.

And since one big factor in politics is The BandWagon Factor, our absence from political polls means that America is being deradicalized. Look for example at online polls. DO they have somewhat different results from telephone polls? You bet!

What can we Always-Onliners do about it? VOTE!!

See you at the polling place.

Land-line number portability

[Anonymous Coward]

In Denmark we got number portability for land-lines a couple of years ago, so area codes do not necessarily tell you where people live. Since telephone pollsters use area codes to make sure they have geographical coverage, this presents a similar problem.

I don't understand this position

[GregWebb]

I _want_ the world to reflect my tastes. I want companies to introduce products that match what I want, I want my favourite TV programs to get more airtime, I want my political positions to be listened to and regarded as significant.

Opinion polls are an excellent way to do this. I've not been invited to participate in any number by phone but have several by e-mail or banner ads. Unless it's simply not possible for me to participate for some reason, I _will_ complete the poll. Yes, it's a small contribution but it's another point in their dataset and it corresponds to _me_ and helps drag data towards me just a little.

I won't participate in loyalty card schemes because I don't like the data density they're building up and don't think many shoppers appreciate quite how much data is being stored on exactly what they do and what can be done with it. Polls are rather different though - it's upgront about what's being gathered and due to the different nature of the data, has rather less nefarious possibilities for data mining. Net result I'm absolutely fine with giving them data to help swing towards me. After all, if I've got the chance to help steer the world towards what I want and I _can_ take it, why shouldn't I?

For whom the call costs money?

[p3d0]

This presents a problem for telephone pollsters who are prohibited by the FCC from calling cell phones with automated equipment, and from calling people for whom receiving the call costs money.

I'd like to sign up immediately for a land phone plan whereby I pay 1 cent per call I receive.

20% of those surveyed are liars.

[G4from128k]

A survey held at my university on the day after student government elections showed that 30% of those surveyed had voted. The actual turnout was only 10%. Without the slightest embarassment, the poll takers reported a "margin of error" of 2% on their numbers.

There are three very different types of people in the world: 1) those that lie on surveys, 2) those that avoid surveys, and 3) those that actually participate and tell the truth on surveys. I would suggest that the errors and biases introduced by groups 1 & 2 make the data from group 3 all but meaningless.

Re:Yes.

[squiggleslash]

Well, you can look at it two ways.

Politicians can use polls to determine if their policies are viable, if they have popular support, and if they're too "out there", the politicians can either communicate those policies better, if they believe the problem is simple misunderstanding, or can ditch policies that have little or no popular support. This would be democracy in action.

Alternatively, politicians can use polls to determine if their policies are popular, if they have can get elected with them, and if they're too "out there", the politicians can either hide those policies better, or can ditch policies that have little or no popular support. This would be the same as what I just said, only worded from a more cynical standpoint.

In all fairness, opinion polls help move political parties towards general consensus. Without this kind of feedback, you might have to choose between extreme-left and extreme-right parties at election time. Rather than two extreme-not-what-you-actually-want parties.

So switch to a sensible phone system!

[beeblebrox87]

This wouldn't be a problem if US mobile phone companies weren't so braindead in their charging plans, forcing people to pay for incoming calls. The US is essentially the only country in the world that does this, and it really doesn't make sense. Everywhere else, only outgoing calls cost money, and mobile phone companies are still very profitable.

*sigh* How is it that, with all their money, the US is stuck with a mobile phone system inferior to that of most third world countries? I live in Tanzania, and our mobile phone system is superior to the US' in reception, coverage, audio quality, interoperability, price, and convenience.

Surveys Circumvent FCC Regulations

[omnirealm]

I have a brother-in-law who sells vacuum cleaners. He told me that his company sets up sales appointments for him. I asked him how his company finds these people, and he responded that the company starts out with a "survey" call. If "they qualify," then they will get a second phone call asking if they would like a visit from a company representative to tell them more about these vacuum cleaners.

This company argues that those who participated in the initial survey have a "prior business relationship" with the company, and so even if they are on the National Do-not-call Registry, the company can still make a soliciting call to them. The survey is just a front to get around FCC regulations. Hence, I personally will never respond to a survey call, because I cannot trust the intent of the survey-takers.

On that note, I got a call on behalf of my local state trooper organization a couple of weeks ago. After some small talk, the guy on the phone asked, "We offer a $45, a $50, and a $60 contribution amount to the state trooper fund. Which of these amount would work best for you?" Well, first of all, I *hate* coercive questions. I told him, "I don't know; I'll have to think about it." He responded along the lines of, "Well, we do have a minimum contribution option of $15. Why don't I send you a letter about this, and you don't have to commit to anything at the moment." Okay, whatever. So I got a letter in the mail thanking me for committing to giving them money, and I noticed (in small print) a mention of thanks for this telemarketing company for helping the state troopers raise these funds.

That did it for me. Under no uncertain circumstances will I ever deliberately justify a telemarketing firm's existence. Part of my contribution would go to help support the telemarketers. I shredded that letter on the spot, and when that marketing firm calls again, I'm going to tell them in a very unpleasant way exactly why they never heard back from me (hint: it will probably emphasize exactly what I think about telemarketing types).

How accurate can polls be anyway?

[silverbax]

If you look over most Gallup polls, the average poll sample is usually 600-1200 people. That's less than 1/100th of 1% of the United States population.

How could it possibly be accurate? Short answer: It isn't remotely accurate. Gallup had Bush winning the 2000 election by as much as 7-10 percentage points. He lost the popular vote by 1-2%. That's a 8-11% inaccuracy, roughly. Not to spark debate by using this particular election for an example, but just referring to one most people would remember. Polls come out on a wide variety of subjects daily...some of which are commissioned by specific groups. You want a poll that says more people use Linux? Just buy it. You want one that says more people prefer death to ice cream? Buy it. If you can find 160,000 people who would buy Justin Gaurini's CD, you can be damn sure you can find a couple hundred who will say pretty much anything.

Then, release the poll to try and start some viral marketing. "More people love Linux? I gotta get in on this!"

Of course, if you read Gallup's disclaimers, they pretty musch say two things:

1. These polls are not really accurate.
2. Any poll can be skewed.

But, Gallup never explains what steps they take to prevent skewing, only that they are aware skewing can occur. Well, duh!

Polls don't reflect me anyway

[koreth]

I usually participate in polls when someone calls, because I want to see my point of view reflected in the results -- to the extent that the poll is going to influence someone's policy somewhere, I'd rather have them do something closer to what I want.

The trouble is, it seems like many of the polls I'm hit with don't really account for someone as weird as me.

For example, a couple months ago I got a call from someone doing a poll on consumer electronics and home automation. Great, I thought, that's right up my alley. Unfortunately, a lot of the questions were along the lines of, "Do you plan to purchase a computer capable of recording TV shows in the next six months?" The honest answer is, "Not unless one of the two I already own breaks." So I have to decide on the spot how I want to misrepresent myself: do I answer honestly and underrepresent my interest in such things, or lie and overrepresent my interest in purchasing such a thing? (I answered "no" to that one.)

Political polls aren't quite as bad in that regard, but they also usually fail to include reasonable options for someone with, say, a vehemently pro-personal-freedom point of view, so you're stuck with five or ten seconds to figure out which of the unsuitable multiple choices will skew the poll closest to what you'd like to see.

Now, obviously in most cases this is due to poor poll design, but in my experience most polls aren't all that well designed. So if I decide to switch off my landline at some point, I don't think I'll lose much sleep over pollsters' inability to get highly inaccurate poll results out of me.

Doesn't Caller ID have more of an effect?

[bgfay]

I would think that the increased use of caller ID would already be making phone polls almost worthless. Who answers their phone anymore if they don't know the caller? There is a certain population that just doesn't want to participate in surveys any more and there is another group that won't be bothered with phone calls. So who is answering these survey phone calls? That's a piece of data I would like to see.

Of course, the news media outlets (radio, television, newspaper, and web) like the polls because they make for quick headlines and short articles. Thus, the polls results become a force driving opinion disproportionately.

Of course, I'm not a statistician, so I'm probably just blowing smoke out of my ass.