Putting the Self in Self-Correction: Findings from the Loss-of-Confidence Project

Abstract

Science is often perceived to be a self-correcting enterprise. In principle, the assessment of scientific claims is supposed to proceed in a cumulative fashion, with the reigning theories of the day progressively approximating truth more accurately over time. In practice, however, cumulative self-correction tends to proceed less efficiently than one might naively suppose. Far from evaluating new evidence dispassionately and infallibly, individual scientists often cling stubbornly to prior findings. Here we explore the dynamics of scientific self-correction at an individual rather than collective level. In thirteen written statements, researchers from diverse branches of psychology share why and how they have lost confidence in a published finding. We qualitatively characterize these disclosures and explore their implications. A cross-disciplinary survey suggests that such loss-of-confidence sentiments are surprisingly common among members of the broader scientific population, yet rarely become part of the public record. We argue that removing barriers to self-correction at the individual level is imperative if the scientific community as a whole is to achieve the ideal of efficient self-correction.

Correction notices in psychology: impactful or inconsequential?

“…However, that was not the case in any of the corrections we examined. Indeed, retrospective disclosure of this nature is rare in general (see also Rohrer et al [ 19 ]).…”

Reanalysis Suggests Evidence for Motor Simulation in Naming Tools Is Limited: A Commentary on Witt, Kemmerer, Linkenauger, and Culham (2010)

“…With a growing appreciation of multiverse and Bayesian approaches to data analyses in psychology, we no longer believe that the original article accurately reflects the results. We were inspired by the Loss-of-Confidence Project by Julia Rohrer and colleagues (in press) and their call for science to have an outlet to be self-correcting. One option was to retract our original article.…”

Error Tight: Exercises for Lab Groups to Prevent Research Mistakes

“…Programming in an in luential di ference in the timing of two conditions, 5 writing a program that is intended to randomly assign people to conditions but only assigns to one condition Set up a lab style guide with clear and consistent naming standards, 13 include codebooks or metadata So tware errors Excel converting things to dates 14 Using so tware without the known issues, 14 in-house independent checking…”

“…Creating composite scores without reverse coding the necessary items, failing to exclude participants you should have, variable treated as an integer rather than a factor, scripting/coding error [15][16][17] Use a scripting language in which every step is documented, 18 in-house independent checking, co-piloting, 19 "Red Team" 20 , unit testing 21,22 Statistical errors Failing to include random slopes in an analysis that warranted them 5 In-house independent checking, code co-piloting 19 "Red Team" 20…”