Abstract:
Zhang (2023) used an online, pre-registered, large-scale controlled experiment to test the effect of an endorsement of Joe Biden by the scientific journal Nature on several perceptual and behavioural outcomes. The main results of the paper were the following: the endorsement of Biden caused a large reduction in Trump supporters' trust in Nature and a considerably smaller reduction in their 'trust in US scientists'. The estimated effects are larger for individuals who, prior to the treatment, believed that Nature was unlikely to have endorsed a presidential candidate. The endorsement also made Trump supporters less likely to request COVID and vaccine related information from the endorsing journal. For Biden supporters, the respective estimated effects were generally positive, but small and insignificant. In his abstract, the author summarizes his key causal claim as follows: "political endorsement by scientific journals can undermine and polarize public confidence in the endorsing journals and the scientific community" (p.696). In this replication study, we computationally reproduced all results, with few and trivial exceptions. We then tested the robustness of those results that gave rise to Zhang's (2023) main causal claim. These tests include an alternative estimation method, an alternative way to capture support for the candidates, and a series of heterogeneity analyses by demographics. All test results support the author's findings but add interesting nuance. Some of our tests exploit variables from the raw data that were not included in the clean, published dataset, but the author willingly provided: a post-treatment 'manipulation check' that asked respondents to indicate the candidate that Nature actually endorsed, and data on requests for COVID related articles from other outlets besides Nature. We used these variables to conduct an Instrumental Variables (IV) procedure and test a 'causal mediation' model. Overall, and for Trump supporters in particular, our report corroborates the author's main finding of a strong negative effect of the endorsement on the overall perception of the endorser (Nature). However, the additional analysis provides weaker evidence for a reduction in trust in the scientific community more generally.
