Please use this identifier to cite or link to this item: https://hdl.handle.net/10419/263216 
Year of Publication: 
2022
Series/Report no.: 
GLO Discussion Paper No. 1157
Publisher: 
Global Labor Organization (GLO), Essen
Abstract: 
Amazon's Mechanical Turk is a very widely-used tool in business and economics research, but how trustworthy are results from well-published studies that use it? Analyzing the universe of hypotheses tested on the platform and published in leading journals between 2010 and 2020 we find evidence of widespread p-hacking, publication bias and over-reliance on results from plausibly under-powered studies. Even ignoring questions arising from the characteristics and behaviors of study recruits, the conduct of the research community itself erodes substantially the credibility of these studies' conclusions. The extent of the problems vary across the business, economics, management and marketing research fields (with marketing especially afflicted). The problems are not getting better over time and are much more prevalent than in a comparison set of non-online experiments. We explore correlates of increased credibility.
Subjects: 
online crowd-sourcing platforms
Amazon Mechanical Turk
p-hacking
publication bias
statistical power
research credibility
JEL: 
B41
C13
C40
C90
Document Type: 
Working Paper

Files in This Item:
File
Size





Items in EconStor are protected by copyright, with all rights reserved, unless otherwise indicated.