PLoS ONE· 2013Q1
Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research
- 1,682citations
- Q1SCImago
- 2013year
Short summary
Amazon Mechanical Turk (AMT) can successfully collect data for many complex cognitive behavioral experiments, replicating laboratory findings for tasks like Stroop and attentional blink.
AI-generated from the title and abstract; the full text is not read.
Key points
- AMT can be used to replicate diverse experimental psychology tasks, including Stroop, Switching, and Posner Cuing.
- Most replications using AMT were qualitatively successful, validating online data collection for complex tasks.
- Some tasks showed disparities between laboratory and online results, indicating potential limitations of the AMT platform.
- Researchers encountered important lessons when conducting these replications using AMT.
AI-generated from the title and abstract; the full text is not read.
Abstract
Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenter's control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers.
The authors' abstract, as published at the source. PLoS ONE, 2013 · DOI ↗
Continue with a free account
Ask the paper: 3 free questions a day about this paper; save it, get its citation, new summaries every day for your field. Takeaways are Premium.
Continue free on the webSign in with Google or Apple; no card needed. You come back to this paper.
On your phone:
Field: Computer Science Applications
Computer Science ApplicationsComputer Science