scite: a smart citation index that displays the context of citations and classifies their intent using deep learning

Abstract

Citation indices are tools used by the academic community for research and research evaluation which aggregate scientific literature output and measure scientific impact by collating citation counts. Citation indices help measure the interconnections between scientific papers but fall short because they only display paper titles, authors, and the date of publications, and fail to communicate contextual information about why a citation was made. The usage of citations in research evaluation without due consideration to context can be problematic, if only because a citation that disputes a paper is treated the same as a citation that supports it. To solve this problem, we have used machine learning and other techniques to develop a "smart citation index" called scite, which categorizes citations based on context. Scite shows how a citation was used by displaying the surrounding textual context from the citing paper, and a classification from our deep learning model that indicates whether the statement provides supporting or disputing evidence for a referenced work, or simply mentions it. Scite has been developed by analyzing over 23 million full-text scientific articles and currently has a database of more than 800 million classified citation statements. Here we describe how scite works and how it can be used to further research and research evaluation.

Investigating disagreement in the scientific literature

“…Roughly 0.31% of all citances in our dataset are instances of disagreement, a share that has remained relatively stable over time. However, this number is much smaller than in past studies—such as the 2.4% for so-called “negative” references ( Catalini et al, 2015 ), and the estimated 0.8% for “disputing” citations ( Nicholson et al, 2021 ). This is explained by our operationalization of disagreement, which although conceptually broader than negative or disputing citations, is narrowed to only 23 queries to prioritize precision.…”

“…Quantitative measures can be valuable tools to better understand the role and extent of disagreement across fields of science. Previous research has focused on consensus as evidenced by citation networks ( Bruggeman et al, 2012 ; Shwed and Bearman, 2010 ; Shwed and Bearman, 2012 ); on concepts related to disagreement in scientific texts such as negative citations, disputing citations, and uncertainty ( Catalini et al, 2015 ; Chen et al, 2018 ; Nicholson et al, 2021 ); and on approaches based on word counts ( Bertin et al, 2016 ). Studying disagreement is challenging, given the lack of a widely accepted theoretical framework for conceptualizing disagreement combined with major challenges in its operationalization, for instance, the limited availability of large-scale collections of scientific texts.…”

Replication value as a function of citation impact and sample size

“…In some cases this will be challenging. For example, while it might be possible to classify whether citations occured due to arbitrary citation conventions, citations would need to be manually classified for each replication target (though see Nicholson et al, 2021 for an example of innovations in citation classification). However, three common and substantial sources of noise could likely be corrected for or held constant to improve the reliability of citation counts for measuring impact; the age of the article (Bornmann & Daniel, 2008;Wang, Song, Figure 1: Proposed causal relationship between value and citation count (citation count is outlined by a square to signal it is observable).…”

A decade of in-text citation analysis based on natural language processing and machine learning techniques: an overview of empirical studies

“…The F1 measure is a weighted harmonic mean of the precision (percentage of total results which are relevant) and recall (percentage of relevant results correctly classified by the algorithm). More recent advancement in this direction is offered by Nicholson et al (2021), who proposed a tool entitled 'scite'. scite is a citation index for the classification of citations based on over 23 million full-text publications using machine learning and deep learning techniques.…”

“…In recent times, open access to full-text research publications and technical advancements have brought about extensive changes in the methods and techniques to analyse the in-text citations (Nicholson et al, 2021;Tahamtan & Bornmann, 2019). Therefore, identifying and describing these changes are an important motivation to write this review.…”