Please use this identifier to cite or link to this item:
https://hdl.handle.net/10419/312260
Authors:
Huntington-Klein, Nick
Pörtner, Claus C.
Acharya, Yubraj
Adamkovic, Matus
Adema, Joop
Agasa, Lameck Ondieki
Ahmad, Imtiaz
Akbulut-Yuksel, Mevlude
Andresen, Martin Eckhoff
Angenendt, David
Antón, José-Ignacio
Arenas, Andreu
Aslim, Erkmen Giray
Avdeev, Stanislav
Bacher-Hicks, Andrew
Baker, Bradley J.
Bandara, Imesh Nuwan
Bansal, Avijit
Bartram, David
Bech-Wysocka, Katarzyna
Bennett, Christopher Troy
Berha, Andu
Berniell, Inés
Bhai, Moiz
Bhattacharya, Shreya
Bjoerkheim, Markus
Bloem, Jeffrey R.
Brehm, Margaret E.
Brun, Martín
Buisson, Florent
Burli, Pralhad
Camp, Andrew M.
Cerutti, Nicola
Chen, Weiwei
Clement, Jeffrey
Collins, Matthew
Crawfurd, Lee
Cullinan, John
Deer, Lachlan
Dorsey-Palmateer, Reid
Duquette, Nicolas J.
Fages, Diego Marino
Falken, Grace
Farquharson, Christine
Feld, Jan
Feyman, Yevgeniy
Fiala, Nathan
Fitzpatrick, Anne
Fradkin, Andrey
French, Evaewero
Fu, Wei
Fumarco, Luca
Gallegos, Sebastian
Galárraga, Julio
Gamino, Aaron M.
Gauriot, Romain
Gay, Victor
Gayaker, Savas
Gazeaud, Jules
de Gendre, Alexandra
Gilpin, Gregory
Girardi, Daniele
Goldhaber, Dan
Harris, Mark N.
Heller, Blake H.
Henderson, Daniel J.
Henningsen, Arne
Henry, Junita
Herman, Clément
Hernæs, Øystein
Hill, Andrew J.
Holzmeister, Felix
Huysmans, Martijn
Imtiaz, M. Saad
Jain, Anil K.
Jakobsson, Niklas
Kaire, José
Kameshwara, Kalyan Kumar
Karney, Daniel H.
Kim, Sie Won
Klotzbücher, Valentin
Kronenberg, Christoph
LaFave, Daniel
Lang, David
Lee, Ryan
Liégey, Maxime
Long, Dede
Marcus, Jan
Mari, Gabriele
McCarthy, Ian
Meinzen-Dick, Laura
Merkus, Erik
Miller, Klaus M.
Mogge, Lukas
Murad, S. M. Woahid
Najam, Rafiuddin
Naumann, Elias
Nmadu, Job Nda
Ozer, Gorkem Turgut
Paudel, Jayash
Petroulakis, Filippos
Peukert, Christian
Pitkänen, Visa
Porcher, Simon
Prakash, Manab
Pua, Andrew Adrian Yu
Pugatch, Todd
Putman, Daniel S.
Rayamajhee, Veeshan
Ur Rehman, Obeid
Reimão, Maira Emy
Reuter, Anna
Ricks, Michael David
Rios-Avila, Fernando
Rodriguez, Abel
Roeckert, Julian
Ropovik, Ivan
Roy, Jayjit
Salamanca, Nicolas
Samahita, Margaret
Samudra, Aparna
Sanogo, Vassiki
Sariyev, Orkhan
Schaak, Henning
Segel, Joel E.
Sievertsen, Hans Henrik
Smet, Mike
Smith, Brock
Sorensen, Lucy C.
Spantig, Lisa
Szczygielski, Krzysztof
Tagat, Anirudh
Tastan, Huseyin
Trombetta, Martin
Venkatesan, Madhavi
Vernet, Antoine
Volkov, Eden
Wagner, Gary A.
Wang, Yue
Ward, Zachary
Waters, Tom
Weber, Ellerie
Weinberg, Stephen E.
Weißmüller, Kristina S.
Westheide, Christian
Williams, Kevin M.
Ye, Xiaoyang
Yu, Jisang
Zahid, Muhammad Umer
Zanoli, Raffaele
Pörtner, Claus C.
Acharya, Yubraj
Adamkovic, Matus
Adema, Joop
Agasa, Lameck Ondieki
Ahmad, Imtiaz
Akbulut-Yuksel, Mevlude
Andresen, Martin Eckhoff
Angenendt, David
Antón, José-Ignacio
Arenas, Andreu
Aslim, Erkmen Giray
Avdeev, Stanislav
Bacher-Hicks, Andrew
Baker, Bradley J.
Bandara, Imesh Nuwan
Bansal, Avijit
Bartram, David
Bech-Wysocka, Katarzyna
Bennett, Christopher Troy
Berha, Andu
Berniell, Inés
Bhai, Moiz
Bhattacharya, Shreya
Bjoerkheim, Markus
Bloem, Jeffrey R.
Brehm, Margaret E.
Brun, Martín
Buisson, Florent
Burli, Pralhad
Camp, Andrew M.
Cerutti, Nicola
Chen, Weiwei
Clement, Jeffrey
Collins, Matthew
Crawfurd, Lee
Cullinan, John
Deer, Lachlan
Dorsey-Palmateer, Reid
Duquette, Nicolas J.
Fages, Diego Marino
Falken, Grace
Farquharson, Christine
Feld, Jan
Feyman, Yevgeniy
Fiala, Nathan
Fitzpatrick, Anne
Fradkin, Andrey
French, Evaewero
Fu, Wei
Fumarco, Luca
Gallegos, Sebastian
Galárraga, Julio
Gamino, Aaron M.
Gauriot, Romain
Gay, Victor
Gayaker, Savas
Gazeaud, Jules
de Gendre, Alexandra
Gilpin, Gregory
Girardi, Daniele
Goldhaber, Dan
Harris, Mark N.
Heller, Blake H.
Henderson, Daniel J.
Henningsen, Arne
Henry, Junita
Herman, Clément
Hernæs, Øystein
Hill, Andrew J.
Holzmeister, Felix
Huysmans, Martijn
Imtiaz, M. Saad
Jain, Anil K.
Jakobsson, Niklas
Kaire, José
Kameshwara, Kalyan Kumar
Karney, Daniel H.
Kim, Sie Won
Klotzbücher, Valentin
Kronenberg, Christoph
LaFave, Daniel
Lang, David
Lee, Ryan
Liégey, Maxime
Long, Dede
Marcus, Jan
Mari, Gabriele
McCarthy, Ian
Meinzen-Dick, Laura
Merkus, Erik
Miller, Klaus M.
Mogge, Lukas
Murad, S. M. Woahid
Najam, Rafiuddin
Naumann, Elias
Nmadu, Job Nda
Ozer, Gorkem Turgut
Paudel, Jayash
Petroulakis, Filippos
Peukert, Christian
Pitkänen, Visa
Porcher, Simon
Prakash, Manab
Pua, Andrew Adrian Yu
Pugatch, Todd
Putman, Daniel S.
Rayamajhee, Veeshan
Ur Rehman, Obeid
Reimão, Maira Emy
Reuter, Anna
Ricks, Michael David
Rios-Avila, Fernando
Rodriguez, Abel
Roeckert, Julian
Ropovik, Ivan
Roy, Jayjit
Salamanca, Nicolas
Samahita, Margaret
Samudra, Aparna
Sanogo, Vassiki
Sariyev, Orkhan
Schaak, Henning
Segel, Joel E.
Sievertsen, Hans Henrik
Smet, Mike
Smith, Brock
Sorensen, Lucy C.
Spantig, Lisa
Szczygielski, Krzysztof
Tagat, Anirudh
Tastan, Huseyin
Trombetta, Martin
Venkatesan, Madhavi
Vernet, Antoine
Volkov, Eden
Wagner, Gary A.
Wang, Yue
Ward, Zachary
Waters, Tom
Weber, Ellerie
Weinberg, Stephen E.
Weißmüller, Kristina S.
Westheide, Christian
Williams, Kevin M.
Ye, Xiaoyang
Yu, Jisang
Zahid, Muhammad Umer
Zanoli, Raffaele
Year of Publication:
2025
Series/Report no.:
I4R Discussion Paper Series No. 209
Publisher:
Institute for Replication (I4R), s.l.
Abstract:
We use a rigorous three-stage many-analysts design to assess how different researcher decisions-specifically data cleaning, research design, and the interpretation of a policy question-affect the variation in estimated treatment effects. A total of 146 research teams each completed the same causal inference task three times each: first with few constraints, then using a shared research design, and finally with pre-cleaned data in addition to a specified design. We find that even when analyzing the same data, teams reach different conclusions. In the first stage, the interquartile range (IQR) of the reported policy effect was 3.1 percentage points, with substantial outliers. Surprisingly, the second stage, which restricted research design choices, exhibited slightly higher IQR (4.0 percentage points), largely attributable to imperfect adherence to the prescribed protocol. By contrast, the final stage, featuring standardized data cleaning, narrowed variation in estimated effects, achieving an IQR of 2.4 percentage points. Reported sample sizes also displayed significant convergence under more restrictive conditions, with the IQR dropping from 295,187 in the first stage to 29,144 in the second, and effectively zero by the third. Our findings underscore the critical importance of data cleaning in shaping applied microeconomic results and highlight avenues for future replication efforts.
Document Type:
Working Paper
Appears in Collections:
Files in This Item:
File
Description
Size
Format
Items in EconStor are protected by copyright, with all rights reserved, unless otherwise indicated.