Please use this identifier to cite or link to this item: https://hdl.handle.net/10419/283498 
Year of Publication: 
2023
Series/Report no.: 
Working Papers in Economics and Statistics No. 2023-17
Publisher: 
University of Innsbruck, Research Platform Empirical and Experimental Economics (eeecon), Innsbruck
Abstract: 
A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty not accounted for in reported standard errors and confidence intervals. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population heterogeneity, design heterogeneity, and analytical heterogeneity. We estimate each type's heterogeneity from multi-lab replication studies, prospective meta-analyses of studies varying experimental designs, and multi-analyst studies. Our results suggest that population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. A conservative interpretation of the estimates suggests that incorporating the uncertainty due to heterogeneity would approximately double sample standard errors and confidence intervals. We illustrate that heterogeneity of this magnitude-unless properly accounted for-has severe implications for statistical inference with strongly increased rates of false scientific claims.
Subjects: 
Conflict
contest
conflict resolution
group decision-making
group identity
alliance
experiment
Document Type: 
Working Paper

Files in This Item:
File
Size





Items in EconStor are protected by copyright, with all rights reserved, unless otherwise indicated.