Slashdot Mirror
Inside the Internet's Hidden Science Factory
tcd004 writes Sarah Marshall has completed roughly 20,000 academic surveys. Clay Hamilton has finished about 40,000. Marshall and Hamilton are part of a small but highly-active community of paid online study participants on Amazon's Mechanical Turk who generate data at break-neck speed to fuel modern scientific research. But can a person who's completed thousands of surveys still provide good data? Here's a look at the humans feeding science from inside the machine.
Name: Sarah Marshall
Age: 18-34
Education: Overqualified.
Employment: FIlling in surveys like this one
Hobbies: Writing about filling in surveys like this one
At least he's truthful that he's not truthful.
There are two types of people in the world: Those who crave closure
At least he's truthful that he's not truthful.
Are you sure about that?
Also, I can see great benefit in turning some code like OpenCyc loose on the mechanical turk to generate answers to the survey questions, then reaping the results as OpenCyc answers thousands of surveys a day for you. I'd guess you should probably create multiple identities for that purpose though, as Amazon might get suspicious if someone spends 5,000 hours/day filling out surveys.
I did Turking for a whopping 20 minutes, after 20 minutes I realized that it was costing me more in electricity, time, and cpu heat exhaust than the $4.00 I made or the less than $10 and hour I would have potentially made Turking, never went back.
After skimming this glad I left it.
I am Bennett Haselton! I am Bennett Haselton!
I'm a tenured researcher who has used MTurk from time to time for different things. It's a small part of what I've utilized, and I think maybe it's over utilized somewhat, but here's my thoughts in no particular order:
1. We know about people like you. We can't filter you out all the time, but we have ways of reducing your input (e.g., whether or not your response times are psychophysically possible, consistency of response, consistency of responses with known information about you, like general location).
2. There's been tons of research about MTurk as a participant pool. Generally speaking, data from MTurk is comparable to data from other sources, although MTurkers probably are dealing with a teeny bit more more problems in their life than the average person (which can be good or bad from a researcher's perspective). So you might be lying, but there's always people lying, and the question is whether they're lying more on MTurk than anywhere else.
3. Studies have shown that most MTurkers do it for fun rather than money.
4. Having said that, there have been some issues brewing about the original intent of MTurk and how it's sometimes used. I've seen problems related to the fact some MTurkers see the tasks as jobs, whereas researchers see it as volunteer research participation with a extra gift thrown in. It's something that I used to see as being a non-issue, but I've changed in that opinion over time. I do think there's some subtle ethical issues that arise in terms of differences of expectations between researchers and participants going into things (e.g., as a participant am I a volunteer or an employee? What does it mean if I'm participating in research for a job?).
5. MTurk is great for some tasks--those that are short and quick. Things that take more time, not so much.
6. You can reach a very broad audience with MTurk, which can be useful if you're trying to reach people out of your country.
Personally, I'd like to see something like MTurk set up where it's explicitly about the research. You're seeing people do this with FB and whatnot sometimes, but it would be nice to see something more organized. I've seen attempts, but nothing quite the same.