Type I and Type II error concerns in fMRI research: re-balancing the scale

Abstract

Statistical thresholding (i.e. P-values) in fMRI research has become increasingly conservative over the past decade in an attempt to diminish Type I errors (i.e. false alarms) to a level traditionally allowed in behavioral science research. In this article, we examine the unintended negative consequences of this single-minded devotion to Type I errors: increased Type II errors (i.e. missing true effects), a bias toward studying large rather than small effects, a bias toward observing sensory and motor processes rather than complex cognitive and affective processes and deficient meta-analyses. Power analyses indicate that the reductions in acceptable P-values over time are producing dramatic increases in the Type II error rate. Moreover, the push for a mapwide false discovery rate (FDR) of 0.05 is based on the assumption that this is the FDR in most behavioral research; however, this is an inaccurate assessment of the conventions in actual behavioral research. We report simulations demonstrating that combined intensity and cluster size thresholds such as P < 0.005 with a 10 voxel extent produce a desirable balance between Types I and II error rates. This joint threshold produces high but acceptable Type II error rates and produces a FDR that is comparable to the effective FDR in typical behavioral science articles (while a 20 voxel extent threshold produces an actual FDR of 0.05 with relatively common imaging parameters). We recommend a greater focus on replication and meta-analysis rather than emphasizing single studies as the unit of analysis for establishing scientific truth. From this perspective, Type I errors are self-erasing because they will not replicate, thus allowing for more lenient thresholding to avoid Type II errors.

The mentalizing network orchestrates the impact of parochial altruism on social norm enforcement

“…We primarily found increased activity (at P < 0.005, voxel extent threshold: 10 voxels, Lieberman and Cunningham, 2009) in three brain regions that are well-known to form a neural network involved in mentalizing processes [Gallagher and Frith, 2003;Rilling et al, 2004 Table S1). All these a priori regions of interests survive small volume family-wise-error (FWE) corrections at P < 0.05 (see Method section for details).…”

“…We report results in a priori regions of interests (previously defined in neuroimaging studies on punishment [Buckholtz et al, 2008;de Quervain et al, 2004;Knoch et al, 2006;Sanfey et al, 2003;Spitzer et al, 2007;Strobel et al, 2011] and mentalizing [Van Overwalle 2009]): OFG, right lateral PFC, caudatus, DMPFC, TPJ where activations are significant at P < 0.005 uncorrected for multiple comparisons with an extent threshold of 10 voxels [Lieberman and Cunningham, 2009], and survive small volume corrections (SVC) for multiple comparisons (or family-wise error [FWE] corrections across the whole brain). The SVC procedure, as implemented in SPM5 using the FWE correction procedure (P < 0.05), allows results to be corrected for multiple nonindependent comparisons with a defined region of interest.…”

“…The resulting statistical parametric map (SPM, at P < 0.005, voxel extent threshold: 10 voxels, Lieberman and Cunningham, 2009) mainly revealed increased activation in a hypothesized network of brain regions which have been shown to play key and functionally separated roles in punishment-related decision processes [de Quervain et al, 2004;Knoch et al, 2006Knoch et al, , 2008Kringelbach 2005;Sanfey et al, 2003;Spitzer et al, 2007], including the right orbitofrontal gyrus (rOFG, BA 11/47, x ¼ 33, y ¼ 39, z ¼ À9), the right lateral prefrontal cortex (rLPFC, BA 44/45, x ¼ 57, y ¼ 12, z ¼ 15), and the right dorsal caudatus (x ¼ 15, y ¼ 24, z ¼ 9; Fig. 3, Supporting Information Table S1).…”

Dissociable relations between amygdala subregional networks and psychopathy trait dimensions in conduct‐disordered juvenile offenders

“…That is, trait‐specific variance in amygdalar iFC over and above what can be explained by the other traits. Following earlier work on amygdala subregional iFC patterns [Aghajani et al, 2016; Brown et al, 2014; Roy et al, 2009; Singh et al, 2015], the resulting statistical maps were all corrected for multiple comparisons using cluster‐based correction with initial cluster forming threshold of Z  > 2.3 and cluster extent threshold of P < 0.05, which tends to adequately balance the propagation of false positives and false negatives [Bennett et al, 2009; Jenkinson et al, 2012; Lieberman and Cunningham, 2009]. …”

Rule and similarity in grammar: Their interplay and individual differences in the brain

“…This more liberal threshold was chosen in order to reduce the possibility of a type II error (Lieberman & Cunningham, 2009) and due to the more explorative nature of the analysis of covariates. Correlations between differential activation of grammatical and ungrammatical items averaged across activation-clusters, and respective rule or similarity estimates derived from the ROC-analysis were tested using the statistical analysis-software SPSS in a repeated-measures GLM.…”