Inside the Internet's Hidden Science Factory

tcd004 writes Sarah Marshall has completed roughly 20,000 academic surveys. Clay Hamilton has finished about 40,000. Marshall and Hamilton are part of a small but highly-active community of paid online study participants on Amazon's Mechanical Turk who generate data at break-neck speed to fuel modern scientific research. But can a person who's completed thousands of surveys still provide good data? Here's a look at the humans feeding science from inside the machine.

Comments

I can see where this is going.

[Anonymous Coward]

Name: Sarah Marshall

Age: 18-34

Education: Overqualified.

Employment: FIlling in surveys like this one

Hobbies: Writing about filling in surveys like this one

Can a person still provide good data?

[Anonymous Coward]

Well, I don't know. Let me get out my trusty coin. Heads, yes.... tails, no....

No

[tehlinux]

I do these surveys when I need money and I assure you, my answers are usually not truthful.

Re:No

[Anonymous Coward]

Then you're an asshole.

Re:No

[NatasRevol]

At least he's truthful that he's not truthful.

Re: No

[Anonymous Coward]

Who cares about honesty in some bullshit online survey?

Re: No

[Anonymous Coward]

Because those bullshit online surveys, especially ones that pay money, are used to make real decisions. Sometimes very important decisions and not always directly (meaning those making the decisions don't realize their charts and graphs are polluted).

Re: No

[Anonymous Coward]

This. If your profession in any way involves taking shit people write online at face value then spinning it as science you are as anti-science as a person can get.

Re: No

[Anonymous Coward]

Then they are the idiots

Re:No

[Em Adespoton]

At least he's truthful that he's not truthful.

Are you sure about that?

Also, I can see great benefit in turning some code like OpenCyc loose on the mechanical turk to generate answers to the survey questions, then reaping the results as OpenCyc answers thousands of surveys a day for you. I'd guess you should probably create multiple identities for that purpose though, as Amazon might get suspicious if someone spends 5,000 hours/day filling out surveys.

Re:No

[Anonymous Coward]

I'm a tenured researcher who has used MTurk from time to time for different things. It's a small part of what I've utilized, and I think maybe it's over utilized somewhat, but here's my thoughts in no particular order:

1. We know about people like you. We can't filter you out all the time, but we have ways of reducing your input (e.g., whether or not your response times are psychophysically possible, consistency of response, consistency of responses with known information about you, like general location).

2. There's been tons of research about MTurk as a participant pool. Generally speaking, data from MTurk is comparable to data from other sources, although MTurkers probably are dealing with a teeny bit more more problems in their life than the average person (which can be good or bad from a researcher's perspective). So you might be lying, but there's always people lying, and the question is whether they're lying more on MTurk than anywhere else.

3. Studies have shown that most MTurkers do it for fun rather than money.

4. Having said that, there have been some issues brewing about the original intent of MTurk and how it's sometimes used. I've seen problems related to the fact some MTurkers see the tasks as jobs, whereas researchers see it as volunteer research participation with a extra gift thrown in. It's something that I used to see as being a non-issue, but I've changed in that opinion over time. I do think there's some subtle ethical issues that arise in terms of differences of expectations between researchers and participants going into things (e.g., as a participant am I a volunteer or an employee? What does it mean if I'm participating in research for a job?).

5. MTurk is great for some tasks--those that are short and quick. Things that take more time, not so much.

6. You can reach a very broad audience with MTurk, which can be useful if you're trying to reach people out of your country.

Personally, I'd like to see something like MTurk set up where it's explicitly about the research. You're seeing people do this with FB and whatnot sometimes, but it would be nice to see something more organized. I've seen attempts, but nothing quite the same.

Re: No

[Anonymous Coward]

And you are committing fraud. Sure it is not enough money to seek legal recourse, but how does that weigh on your conscious? Obviously not at all and that tells us a lot about you (e.g. you are a sociopath).

Re:No

[tcd004]

Thanks for sharing this. really fascinating.

Re:No

[tcd004]

This is excellent additional perspective -would you be willing to add this to the comments in the article?

Re: No

[Anonymous Coward]

And you are committing fraud. Sure it is not enough money to seek legal recourse, but how does that weigh on your conscious? Obviously not at all and that tells us a lot about you (e.g. you are a sociopath).

...Which makes you a good capitalist businessperson, so carry on!

Re:No

[rtb61]

Those people probably know far more about you than you think they do and that is in your most wild imaginings. Never assume based upon the title of those kinds of surveys and questions being asked, what the real nature of the survey might actually be about and how they choose to interpret and analyse you answers. Be careful how long the answers to your questions last and what future potential employer might gain access to interpretations of your behaviour.

Re: No

[Anonymous Coward]

I fully agree with this.

Online survey results are meaningless 100% of the time. They only serve the purposes of entertainment and are not even close to being science. Anyone who uses this data for any real purpose which assumes even remote accuracy IS an idiot.

Re: No

[Anonymous Coward]

It's not fraud to enter false information on a survey, even if you are paid to take it. The money is the same. It may be doing a poor job, but it is not illegal and is most certainly not fraud.

If it was me, I'd have no ethical dilemma whatsoever. The people paying others to complete surveys are suckers and don't deserve to profit off of such an idiotic business. The info is all useless anyway for anything beyond a bit of fun, and that's why I see absolutely nothing wrong with making things up.

Re:No

[Anonymous Coward]

Hey guys, I found the humanities major!

Re:No

[Jack Griffin]

I get to do these market research surveys from time to time. They pay about $80 for 60-90 minutes worth of questions, so I go along, eat their food, say whatever I feel like and take their cash.

Re:No

[tehlinux]

>3. Studies have shown that most MTurkers do it for fun rather than money.

That one made me laugh.

Re:No

[tehcyder]

At least he's truthful that he's not truthful.

Telling the truth is not the highest moral quality: acting morally is.

Someone who does good but sometimes lies (eg. telling a white lie about Santa Claus to avoid upsetting a child) is morally superior to someone who does evil but is entirely honest about it.

Re: No

[Anonymous Coward]

uh huh...
i don't do those, but seem to have gotten on a list for being willing to participate in telephone surveys... for the most part, i don't mind taking 5-10 mins to have my POV registered in some fashion somewhere, EXCEPT -almost without exception- they are ALL utter BULLSHIT...
(AGAIN, realizing i am in a different realm of polling, not 'real' scientific research... whatevs...)
*besides* the 'normal' push-polling crap that goes on (Question: People are generally idiots, should we A. Kill them all, B. Just shoot them in the leg ? Answer: uh.... neither ? Q: That is not an allowable response, A or B...), they also do all kinds of REALLY unethical tricks:
set it up so there is a sequence of "A. Always, B. Most of the time, C. Sometimes, D. Rarely, E. Never", and go through a series of questions on that until you anticipate what your answer is going to be and don't bother listening to all the choices, *THEN* they pull a switcheroo on the question they are REALLY trying to make come out how they want, and make A= Never and E = Always;
similarly -had this happen about 3-4 times in nearly identical form, so i KNOW it was a slimy trick- would poll you on some general stuff at the end of the survey, age, gender, etc, then throw in some about religion/attending church, they had choices for attending services -say- A. Once a year, B. Infrequently, C. Once a week, D. Once a day...
THE LEAST you could 'choose', was 'once a year', or whatever, you could NOT CHOOSE "NEVER"...
along the same lines, they would ask about religious affiliation, but there was NO CHOICE for atheist/agnostic, the 'LEAST RELIGIOUS' you could be was 'All others'....
fuckers...

Re: No

[Anonymous Coward]

The purpose of the science? To R00t people? I'd do the same.

Re: No

[Anonymous Coward]

The point is:
To get the survey results they want.
To find the magical wording that will bypass your word filter (usefull for politicians corporations)
To find your perceptual blind spots.
To R00T you.

Wow, what a world!

[Anonymous Coward]

a 20 something "director" of something called "behavioral economics"... I wonder how deep down the deception goes when no one does anything, but your entire job is convincing everyone else how important you are.

Been there

[g0bshiTe]

I did Turking for a whopping 20 minutes, after 20 minutes I realized that it was costing me more in electricity, time, and cpu heat exhaust than the $4.00 I made or the less than $10 and hour I would have potentially made Turking, never went back.

After skimming this glad I left it.

Re:Been there

[Anonymous Coward]

I don't see how people could even subsist with the money these kinds of things pay. There was another article about freelance sites and they have TEAMS of developers working for less than you would need to buy computer equipment. I don't see why these things don't collapse.

Sarah Marshall?

[Anonymous Coward]

Pffft. Forget her.