Detecting Stance in Scientific Papers: Did we get more Negative Recently?

Abstract

In this paper, we classify scientific articles in the domain of natural language processing (NLP) and machine learning (ML) into whether (i) they extend the current state-ofthe-art by introduction of novel techniques which beat existing models or whether (ii) they mainly criticize the existing state-ofthe-art, i.e., that it is deficient with respect to some property (e.g., wrong evaluation, wrong datasets, misleading task specification). We refer to contributions under (i) as having a "positive stance" and contributions under (ii) as having a "negative stance" to related work. We annotate over 2k papers from NLP and ML to train a SciBERT based model to automatically predict the stance of a paper based on its title and abstract. We then analyze large-scale trends on over 41k papers from the last ∼35 years in NLP and ML, finding that papers have gotten substantially more positive over time, but negative papers also got more negative and we observe considerably more negative papers in recent years. Negative papers are also more influential in terms of citations they receive.1 This concept is related to positive/negative citations within a paper, which has been annotated in a few works, e.g., Teufel et al. (2006). Our work goes beyond individual citations and assesses the stance of the authors' main message.