“I’ve discovered a key paper on a topic I know relatively little about. How do I see where it sits in the broader literature?”

For many researchers, this is a common task. When finding a key paper - one that has received hundreds or thousands of citations, been the subject of debate, or made a controversial claim - understanding how it has influenced a scientific field or been received by others can be a daunting task.

In this blog post, we will use scite to examine an important paper and answer a number of key questions about it, such as:

• How often is the paper cited? Is it cited less in recent years, or is its influence growing?
• Are there any major challenges to the theory proposed by the key paper? Have these challenges been widely supported with empirical evidence?
• How widely applicable are the claims in the paper? To what topics has it been applied?
• Have the authors of the paper made any significant modifications to their original theory? If so, how?

Overview

Here we use as an example Graham, Haidt, and Nosek’s seminal work, Liberals and conservatives rely on different sets of moral foundations - a paper that charted a new direction for moral psychology. According to the scite database, it has been cited by roughly 2,400 publications in over 3,000 citation statements since its publication in 2009. In short, the authors claim that there are five basic subdomains of moral cognition (how people reason about right and wrong), and that political liberals and conservatives differ in how much importance they place on each subdomain.

The scite report reveals a few interesting facts at a glance in the “Filters” panel on the left: first, it has received nearly 200 supporting citations and a handful of contrasting citations (for more information on what these categories mean and how citations are classified, see our help article).

Moreover, its influence appears to be at least consistent - if not increasing - over the past decade, given the sheer number of citations it has received across time (the y-axis represents the number of citations).

Let’s start by filtering by paper section to see whether the paper was cited as a theoretical work (and thus cited in the Introduction) or in an attempt to replicate and/or extend its results. We can also limit our results to peer-reviewed papers (excluding, for example, preprints and other non- peer-reviewed manuscripts).

Finally, we can limit our results to those appearing in a select set of journals - ones that directly address moral psychology.

This combination of filters reduces the citation count from over 3,000 statements to a mere eight. In these filtered results, we can see various instances in which the key theoretical contribution of the paper in question – Moral Foundations Theory – has been applied to a variety of topics, such as political ideology, political activism, and specific issues, such as Scottish independence from the United Kingdom.

Finding major challenges to Graham, Haidt and Nosek

We can also use scite to find major challenges to Graham, Haidt and Nosek. To do so, let’s limit our findings to citation statements that seem to be in contrast with their central empirical claims.

A number of results are definitely relevant. For example, Igarashi and Laurence (2021) find that the original findings do not necessarily hold with respect to attitudes about immigration.

Similarly, Hubert Lyall, Järvikivi (2021) were unable to replicate the original paper’s central claims about disgust and political ideology.

However, perhaps the most interesting results are Schein, Gray’s (2015) paper entitled The myth of harmless wrongs in moral cognition: Automatic dyadic completion from sin to suffering, in which the authors challenge Moral Foundations Theory and propose an alternative (dyadic morality), as well as provide evidence for it.

The scite report page for this paper also contains a link to another paper by the same authors which outlines dyadic morality in greater detail. As it turns out, this is the most well-developed challenge to Moral Foundations Theory. By examining this paper’s report, we can further see how this alternate theory of moral reasoning has been developed and tested. This process is sometimes called “forward chaining” (for more information on how to use this technique with scite, see our help page).

How well supported are the claims made in Graham, Haidt and Nosek?

As mentioned earlier, this paper has a fair number of supporting citations (198). Filtering by citation type gives us a report of only those statements. By further filtering by section - limiting to Results and Discussion sections, where empirical evidence is presented and discussed - we can further narrow our report to 88 articles.

If we sort by engagement, we can see that the theory presented in the paper has been applied by highly-cited articles to many different domains (e.g., immigration, climate change, COVID-19) to produce results that are roughly consistent with Moral Foundations Theory. At this point, it might be helpful to browse through the first few pages of results, looking for topics that are most relevant to our own research.

We also might be interested in how Graham, Haidt, and Nosek has been supported in the most recent literature. This can be accomplished by moving the “Year Published” slider to limit results to the past two years. This further limits our results to citations from 15 papers, all from 2020 on.

How are Graham, Haidt and Nosek citing their own paper?

A final question we can ask about the paper in question is how its authors continue to cite it. That is, have Graham, Haidt and Nosek modified their theory? Found conflicting results? Abandoned it altogether? This can be accomplished by resetting all the other fields and limiting results to self-citations, which shows 64 papers that have one or more of the paper’s original authors citing themselves.

Sorting by engagement, we see Iyer, Koleva, Graham, Ditto, and Haidt (2012) listed on the first page of results. In this paper, two of our original authors, along with three additional researchers, explore the possibility that the original theory should be modified to account for moral reasoning in a group heretofore unexamined (political libertarians).

Using the same approach we used for other citing papers, we can examine the scite report for Iyer et al. and see how its findings have replicated (or not) and extended. For example, by limiting results to those citations which appear in Method sections, we can see a number of subsequent papers that used the survey instrument Iyer et al. developed for their 2012 paper. Given that these citations continue through 2022, it seems that this modification of Graham, Haidt and Nosek’s (2009) original theory continues to be an active area of research.

To wrap up, let’s review what we have learned about Graham, Haidt and Nosek’s (2009) seminal paper thanks to a quick review of citing research:

• The original paper has been cited a great deal and continues to be cited through 2022.
• A number of challenges to its central claims have been offered, including one major line of research with multiple papers presenting empirical support for an alternative theory.
• The overall findings of the paper have been frequently supported in subsequent research, although some contrasting findings exist.
• The theory presented in the original paper has been used to address a wide variety of topics, including emotion, communicable disease, and the environment.
• There is at least one proposed modification to the theory presented in the original paper, and it continues to be used in ongoing research.