Please use this identifier to cite or link to this item: https://hdl.handle.net/10419/266342 
Authors: 
Breznau, Nate
Rinke, Eike Mark
Wuttke, Alexander
Nguyen, Hung H. V.
Adem, Muna
Adriaans, Jule
Alvarez-Benjumea, Amalia
Andersen, Henrik K.
Auer, Daniel
Azevedo, Flavio
Bahnsen, Oke
Schlueter, Elmar
Schmidt, Regine
Schmidt, Katja M.
Schmidt-Catran, Alexander
Schmiedeberg, Claudia
Schneider, Jürgen
Schoonvelde, Martijn
Schulte-Cloos, Julia
Schumann, Sandy
Bauer, Paul C.
Christmann, Pablo
Schunck, Reinhard
Schupp, Jürgen
Seuring, Julian
Silber, Henning
Sleegers, Willem
Sonntag, Nico
Staudt, Alexander
Steiber, Nadia
Steiner, Nils
Sternberg, Sebastian
Connelly, Roxanne
Baumann, Markus
Stiers, Dieter
Stojmenovska, Dragana
Storz, Nora
Striessnig, Erich
Stroppe, Anne-Kathrin
Teltemann, Janna
Tibajev, Andrey
Tung, Brian
Vagni, Giacomo
Czymara, Christian S.
Van Assche, Jasper
Baute, Sharon
van der Linden, Meta
van der Noll, Jolanda
Van Hootegem, Arno
Vogtenhuber, Stefan
Voicu, Bogdan
Wagemans, Fieke
Wehl, Nadja
Werner, Hannah
Damian, Elena
Wiernik, Brenton M.
Winter, Fabian
Benoit, Verena
Wolf, Christof
Yamada, Yuki
Zhang, Nan
Ziller, Conrad
Zins, Stefan
Żółtak, Tomasz
Bernauer, Julian
Ecker, Alejandro
Berning, Carl
Berthold, Anna
Bethke, Felix S.
Biegert, Thomas
Blinzler, Katharina
Blumenberg, Johannes N.
Bobzien, Licia
Bohman, Andrea
Bol, Thijs
Bostic, Amie
Edelmann, Achim
Brzozowska, Zuzanna
Burgdorf, Katharina
Burger, Kaspar
Busch, Kathrin B.
Carlos-Castillo, Juan
Chan, Nathan
Eger, Maureen A.
Ellerbrock, Simon
Forke, Anna
Forster, Andrea
Micheli, Leticia
Gaasendam, Chris
Gavras, Konstantin
Gayle, Vernon
Gessler, Theresa
Gnambs, Timo
Godefroidt, Amélie
Grömping, Max
Groß, Martin
Gruber, Stefan
Gummer, Tobias
Mijs, Jonathan
Hadjar, Andreas
Heisig, Jan Paul
Hellmeier, Sebastian
Heyne, Stefanie
Hirsch, Magdalena
Hjerm, Mikael
Hochman, Oshrat
Hövermann, Andreas
Hunger, Sophia
Hunkler, Christian
Moya, Cristóbal
Huth, Nora
Ignácz, Zsófia S.
Jacobs, Laura
Jacobsen, Jannes
Jaeger, Bastian
Jungkunz, Sebastian
Jungmann, Nils
Kauff, Mathias
Kleinert, Manuel
Klinger, Julia
Neunhoeffer, Marcel
Kolb, Jan-Philipp
Kołczyńska, Marta
Kuk, John
Kunißen, Katharina
Kurti Sinatra, Dafina
Langenkamp, Alexander
Lersch, Philipp M.
Löbel, Lea-Maria
Lutscher, Philipp
Mader, Matthias
Nüst, Daniel
Madia, Joan E.
Malancu, Natalia
Maldonado, Luis
Marahrens, Helge
Martin, Nicole
Martinez, Paul
Mayerl, Jochen
Mayorga, Oscar J.
McManus, Patricia
McWagner, Kyle
Nygård, Olav
Meeusen, Cecil
Meierrieks, Daniel
Mellon, Jonathan
Merhout, Friedolin
Merk, Samuel
Meyer, Daniel
Ochsenfeld, Fabian
Otte, Gunnar
Pechenkina, Anna O.
Prosser, Christopher
Balzer, Dave
Raes, Louis
Ralston, Kevin
Ramos, Miguel R.
Roets, Arne
Rogers, Jonathan
Ropers, Guido
Samuel, Robin
Sand, Gregor
Schachter, Ariela
Schaeffer, Merlin
Bauer, Gerrit
Schieferdecker, David
Editors: 
Year of Publication: 
2022
Citation: 
[Journal:] PNAS - Proceedings of the National Academy of Sciences [ISSN:] 1091-6490 [Volume:] 119 [Issue:] 44 [Article No.:] e2203150119 [Publisher:] National Academy of Sciences [Place:] Washington, DC [Year:] 2022 [Pages:] 1-8
Publisher: 
National Academy of Sciences, Washington, DC
Abstract: 
This study explores how researchers’ analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.
Subjects: 
metascience
many analysts
researcher degrees of freedom
analytical flexibility
immigration and policy preferences
Persistent Identifier of the first edition: 
Creative Commons License: 
cc-by Logo
Document Type: 
Article
Document Version: 
Published Version

Files in This Item:





Items in EconStor are protected by copyright, with all rights reserved, unless otherwise indicated.