Variability in the analysis of a single neuroimaging dataset by many teams

Abstract

SummaryData analysis workflows in many scientific domains have become increasingly complex and flexible. To assess the impact of this flexibility on functional magnetic resonance imaging (fMRI) results, the same dataset was independently analyzed by 70 teams, testing nine ex-ante hypotheses. The flexibility of analytic approaches is exemplified by the fact that no two teams chose identical workflows to analyze the data. This flexibility resulted in sizeable variation in hypothesis test results, even for teams whose statistical maps were highly correlated at intermediate stages of their analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Importantly, meta-analytic approaches that aggregated information across teams yielded significant consensus in activated regions across teams. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that analytic flexibility can have substantial effects on scientific conclusions, and demonstrate factors related to variability in fMRI. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for multiple analyses of the same data. Potential approaches to mitigate issues related to analytical variability are discussed.

How computational modeling can force theory building in psychological science

“…In the case of b), mass cooperation to work on "larger experimental wholes" (Newell, 1973, p. 24), is perhaps realistic given projects that involve many labs are commonplace (e.g., Botvinik-Nezer et al, 2020;Silberzahn et al, 2018). We advise cautious optimism since these collaborations are operating only at the data and hypothesis levels, which are insufficient to force theory building.…”

“…Not only are there often no substantive nor formalised theories for some datasets in practice (causing chaos, Forscher, 1963), but also the principle of multiple realisability (Putnam, 1967) implies that for every theory there are infinitely many possible implementations consistent with it and datasets that can be collected to test it (Blokpoel, 2018). This helps contextualise studies that show divergence in data modeling decisions given the same hypotheses (e.g., Botvinik-Nezer et al, 2020;Silberzahn et al, 2018).…”

Addressing selective reporting of experiments – the case for predefined exclusion criteria

“…Although we have focused on inclusion and exclusion criteria for deciding on whether or not including experiments in an article, the idea of prespecified criteria can meaningfully apply to any field in which some degree of data censoring is important. Fields in which data must be filtered or selected before analysis, such as electrophysiology, functional neuroimaging or epidemiology, are subject to many researcher degrees of freedom in preprocessing pipelines (Phillips, 2004;Carp, 2012;Botvinik-Nezer et al, 2019); thus, prespecification of criteria for these steps is likely warranted to prevent bias. Genomics and other high-throughput fields have also evolved towards standard evaluation criteria to avoid ad-hoc analysis or simple thresholding in selecting experimental hits for further analysis (Kang et al, 2012).…”

Consensus-seeking and conflict-resolving ---an fMRI study on college couples’ shopping interaction

“…In a similar spirit, our study, although narrow 80 , helps address this ubiquitous social phenomenon (interacting college couples), with two-people fMRI experiments, and illuminate inner interactions in IPL-TPJ, on top of their alluded functions, including (but not limited to): attention, memory, language, with social behaviors. Future studies, with the similar vein, should also follow the path toward more transparency in data, stimuli, code, and even scripts for advanced analyses 81 . Task In this event-related fMRI study, two types of shopping contexts were designed to examine how significant others' preferences affect one's own shopping decisions.…”