Experimenting On Mechanical Turk

itwbennett writes "In a recent article, Dr. Markus Jakobsson, a Principal Scientist at PARC, offers some tips on effectively running human-subject research studies on Amazon's Mechanical Turk. '...[B]enefits [include] very low experiment costs, quick turn-around rates, and relatively simple approvals from human subjects boards. But you have to be careful to avoid bias and error.' says Dr. Jakobsson. For example, in many situations subjects may be biased just from knowing that they are participating in a study, or by knowing the goals of a study. To avoid this bias, you need to 'convey a different task to your subject than what you are observing — essentially deceive them — to see how they react when faced with the situation of interest. Consider a study of user reactions to phishing sites. You may, for example, say that you are studying the common reaction to online e-commerce sites, and ask them to rate how helpful various sites are, with a free-text input field where they can add other observations. You first show them three or four legitimate websites, asking them to rate and describe them; then you show them a phishing site and do the same. Will they tell you that this is a site run by fraudsters? If they do, they noticed signs of fraud without you prompting them.'" The author also gives tips on avoiding cheaters, and determining how much to pay and when.

Comments

Re:What the hell is mechanical turk?

[pamar]

...

Although I'm still not sure what is mechanical or turkish about it. The Amazon part apparently refers to the fact that payment is made in way of credits to Amazon.com.

Amazon got the name from this: http://en.wikipedia.org/wiki/The_Turk

Re:Better idea: take a research methods class

[mikael]

I've done 'turking' a few times - mainly transcription work and some paragraph writing. For an individual, whether a task is worth doing really depends on how quickly they can do it. The going rate would be around $20/hour.

Simple quick repetitive tasks pay the least, while long creative tasks pay the most.
Quick tasks might just pay 10 cents and require the 'turker' to look at an image of a road from a car and click yes/no buttons to say whether there are road markings or traffic lights. Some people are good at doing this kind of repetitive work, so they can keep going for hours and hours.

Creative work usually consists of paragraph writing, and would involve writing 500 words or so for $10, with a particular set of words repeated a specified number of times. A person with English as a first language could do that within 30 minutes, so that's a quick little earner.

There are legends about how the first Amazon Tasks were recognizing music artists and songs from a 10 second snippet of audio. For each task, turkers were being paid 50 cents. People were quick to realize that a single four minute track was being automatically chopped up into 20+ such snippets, so if they got one snippet right, the odds are they could get the next 20 or so tasks done as well. Another nice little earner for $10 every 30 minutes.

Transcription work involves listening to a multi-minute segment of audio and converting it into written text. Tricky work as you have to write down the time, who is speaking and what they said, taking into account regional accents. Alternatively, you could work as an editor and edit together 20 or so transcriptions to form a single consistent transcript that covered an hour or so. Just 3 minutes work would take half an hour, but with bonuses that would make $10 for 30 minutes work.

A search for Steve Fossett was performed using Amazon turk. They took satellite photos of the desert, diced them up and had people look at each square. Many other wrecks were found, but not Steve, as the crash site was actually up at 10,000 feet in the mountains.