The (un)reliability of item-level semantic priming effects

Abstract

Many researchers have tried to predict semantic priming effects using a myriad of variables (e.g., prime-target associative strength or co-occurrence frequency). The idea is that relatedness varies across prime-target pairs, which should be reflected in the size of the priming effect (e.g., cat should prime dog more than animal does). However, it is only insightful to predict item-level priming effects if they can be measured reliably. Thus, in the present study we examined the split-half and test-retest reliabilities of item-level priming effects under conditions that should discourage the use of strategies. The resulting priming effects proved extremely unreliable, and reanalyses of three published priming datasets revealed similar cases of low reliability. These results imply that previous attempts to predict semantic priming were unlikely to be successful. However, one study with an unusually large sample size yielded more favorable reliability estimates, suggesting that big data, in terms of items and participants, should be the future for semantic priming research.

Determining the Relativity of Word Meanings Through the Construction of Individualized Models of Semantic Memory

“…A good example of the impact of individual differences on lexical semantic processing is given by the Semantic Priming Project (Hutchison et al, 2013), which was a mega study designed to examine semantic priming across a large number of participants collected at a variety of laboratories across the United States. This study found a variety of individual differences and reliability issues across stimuli (Yap, Hutchison, & Tan, 2016), replicating previous findings about variability in semantic priming (Stolz, Besner, & Carr, 2005), and established across a reanalysis of a number of different semantic priming experiments (Heyman, Bruninx, Hutchison, & Storms, 2018). Given that semantic priming effects rely upon the association strength of words (Hutchison, 2003), and association is likely to be at least partly determined through environmental occurrence patterns of words, word relativity is likely related to issues of reliability and replication in this literature.…”

Sensitivity to misinformation retractions in the continued influence paradigm: Evidence for stability

“…demonstrated intra-class correlation coefficients (ICC) in the range of .00-.82, with an average reliability of ICC = .45; these results were replicated and expanded upon with further cognitive measures by Enkavi et al (2019). Furthermore, testretest reliability of related cognitive phenomena (e.g., false memory effects) has varied significantly, ranging from acceptable (e.g., Deese-Roediger-McDermott paradigm, r = .51-.76; Blair et al, 2002), to poor (e.g., semantic-priming paradigm, r = .00-.29; Heyman et al, 2018). Considering this, and the fact that we were only investigating whether CIE susceptibility had any demonstrable stability, a cut-off criterion of ICC = .45 (based on the average reliability found by Hedge et al, 2018) was used in the current study to signify relative stability.…”

The Primacy of Experience in Language Processing: Semantic Priming Is Driven Primarily by Experiential Similarity

“…Importantly, the assignment of specific primes to the first or second session was counterbalanced across participants, so that the effect of session order was cancelled out in the computation of priming. Split-half reliability of the magnitude of the priming effect was .52, which is high for semantic priming in lexical decision (Heyman et al, 2018). The difference in prime- target orthographic Levenshtein distance between the two sessions did not correlate with priming (π = .00), neither did differences in the other 23 lexical variables listed in Supplemental Table 2 , except for concreteness (π = .18, uncorrected p = .022), ruling out the possibility that priming magnitude was driven by ortho-phonological factors.…”