Journal Peer Review and Editorial Evaluation: Cautious Innovator or Sleepy Giant?

Horbach, Serge P. J. M.; Halffman, Willem

doi:10.1007/s11024-019-09388-z

Cited by 32 publications

(53 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In this study, we were primarily interested in the (qualitative description of) the process of changing editorial practices. We hence did not focus on the frequency or number of such innovations, for which we can refer to previous work providing a quantitative assessment of editorial innovations at scholarly journals [8].…”

Section: Discussionmentioning

confidence: 99%

Innovating editorial practices: academic publishers at work

Horbach

Halffman

2020

Res Integr Peer Rev

Self Cite

View full text Add to dashboard Cite

Background Triggered by a series of controversies and diversifying expectations of editorial practices, several innovative peer review procedures and supporting technologies have been proposed. However, adoption of these new initiatives seems slow. This raises questions about the wider conditions for peer review change and about the considerations that inform decisions to innovate. We set out to study the structure of commercial publishers’ editorial process, to reveal how the benefits of peer review innovations are understood, and to describe the considerations that inform the implementation of innovations. Methods We carried out field visits to the editorial office of two large academic publishers housing the editorial staff of several hundreds of journals, to study their editorial process, and interviewed editors not affiliated with large publishers. Field notes were transcribed and analysed using coding software. Results At the publishers we analysed, the decision-making structure seems to show both clear patterns of hierarchy and layering of the different editorial practices. While information about new initiatives circulates widely, their implementation depends on assessment of stakeholder’s wishes, impact on reputation, efficiency and implementation costs, with final decisions left to managers at the top of the internal hierarchy. Main tensions arise between commercial and substantial arguments. The editorial process is closely connected to commercial practices of creating business value, and the very specific terms in which business value is understood, such as reputation considerations and the urge to increase efficiency. Journals independent of large commercial publishers tend to have less hierarchically structured processes, report more flexibility to implement innovations, and to a greater extent decouple commercial and editorial perspectives. Conclusion Our study demonstrates that peer review innovations are partly to be understood in light of commercial considerations related to reputation, efficiency and implementations costs. These arguments extend beyond previously studied topics in publishing economics, including publishers’ choice for business or publication models and reach into the very heart of the editorial and peer review process.

show abstract

Section: Discussionmentioning

confidence: 99%

Innovating editorial practices: academic publishers at work

Horbach

Halffman

2020

Res Integr Peer Rev

Self Cite

View full text Add to dashboard Cite

show abstract

“…Despite the many suggestions and innovations that promise to improve the peer review system, adoption of these new initiatives seems slow (7,8). On a global scale, editorial procedures are rather stable and traditional ways of organising the editorial process still prevail, despite evidence of aws in old practices and proposed advantages of new ones (9)(10)(11).…”

Section: Introductionmentioning

confidence: 99%

“…On a global scale, editorial procedures are rather stable and traditional ways of organising the editorial process still prevail, despite evidence of aws in old practices and proposed advantages of new ones (9)(10)(11). Implementation of novel review procedures seems to be restricted to speci c niches (specialties, publishing platforms), with the exception of the implementation of text similarity software or 'plagiarism scanners' (8). Slow adoption may be partly explained by a lack of systematic evidence of their effectiveness (12).…”

Section: Introductionmentioning

confidence: 99%

Innovating Editorial Practices: Academic Publishers at Work

Horbach

Halffman

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Abstract Background: Triggered by a series of controversies and diversifying expectations of editorial practices, several innovative peer review procedures and supporting technologies have been proposed. However, adoption of these new initiatives seems slow. This raises questions about the wider conditions for peer review change and about the considerations that inform decisions to innovate. We set out to study the structure of commercial publishers’ editorial process, to reveal how the benefits of peer review innovations are understood, and to describe the considerations that inform the implementation of innovations Methods: We carried out field visits to the editorial office of two large academic publishers housing the editorial staff of several hundreds of journals, to study their editorial process, and interviewed editors not affiliated with large publishers. Field notes were transcribed and analysed using coding software..Results: at the publishers we analysed, the decision-making structure seems to show both clear patterns of hierarchy and layering of the different editorial practices. While information about new initiatives circulates widely, their implementation depends on assessment of stakeholder’s wishes, impact on reputation, efficiency and implementation costs, with final decisions left to managers at the top of the internal hierarchy. Main tensions arise between commercial and substantial arguments. The editorial process is closely connected to commercial practices of creating business value, and the very specific terms in which business value is understood, such as reputation considerations and the urge to increase efficiency. Journals independent of large commercial publishers tend to have less hierarchically structured processes, report more flexibility to implement innovations, and to a greater extent decouple commercial and editorial perspectives.Conclusion: Our study demonstrates that peer review innovations are partly to be understood in light of commercial considerations related to reputation, efficiency and implementations costs. These arguments extend beyond previously studied topics in publishing economics, including publishers’ choice for business or publication models and reach into the very heart of the editorial and peer review process.

show abstract

“…For one, bias in peer review has been discussed with respect to gender, race, language, career stage and interdisciplinarity (see, for example, Helmer, 2017;Lee, et al, 2013). Peer reviewers also tend to be conservative and risk-averse in their evaluation of innovation methods and approaches (Luukkonen, 2012), not to mention the inconsistent reliability and validity of peer review notwithstanding the availability of innovative procedures and platforms (Bornmann, 2011;Horbach & Halffman, 2019).…”

Section: Introductionmentioning

confidence: 99%

Ethical Theories in Research Evaluation: An Exploratory Approach

Biagetti

Gedutis

2020

Scholarly Assessment Reports

View full text Add to dashboard Cite

Research evaluation encompasses the practices of assessing research quality and impact at various stages of research. The processes and criteria of research evaluation vary depending on the nature and objectives of the assessment. Different research evaluation systems influence the research strategies of universities and institutes. There are, however, some known issues of research evaluation with regards to the peer review and, most prominently, the use of citation-based metrics, which lead to recent calls for responsible use of metrics. In this paper, we argue that there is a need for ethical theories for considering research evaluation and that research evaluation ethics, as an overlapping area between research ethics and evaluation ethics, deserve its own treatment. The core of the article consists of a discussion of the most influential ethical theories in the context of the research evaluation, including the deontological ethics, the consequentialist ethics and the virtue ethics. The aim is to highlight the need to assume an ethical view that combines the deontological and the consequentialist concepts, adopting 'common good' as the most likely pillar for the research evaluation procedures. We propose that the mixed approach would be useful for developing a framework for research evaluation ethics and for analysing ethical approaches and ethical dilemmas in research evaluation. Policy Highlights The misuses and abuse of evaluative metrics have been discussed and debated in many high-profile publications including the San Francisco Declaration on Research Assessment (DORA), The Metrics Tide, the Leiden Manifesto, and the Hong Kong Principles. There are also many studies stating the limitations of and bias in peer review. The debates and discussions, however, have not been explored in light of ethical theories. The article considers also good practices in evaluation, including the American Evaluation Association Guiding Principles for Evaluators (AEA 2018), the Australasian Evaluation Society Guidelines for the ethical conduct of evaluations (AES 2000, 2010, 2013) the UK Evaluation Society Guidelines for Good Practices in Evaluation (UK 2019) and the United Nations Ethical Guidelines for Evaluation (UNEG 2008). The paper argues that ethical theories are useful in understanding ethical assumptions and ethical dilemmas in research evaluation and are pertinent in future design and development of research evaluation processes and criteria. Ethical theories that can construct ethical principles for research evaluation, including deontological and consequentialist ethics, taking into account the Mertonian normative theory, have been examined. In order to address the issues of research evaluation, we propose a mixed approach that combines the deontological and the consequentialist concepts that is able to infringe the boundaries of the rivaling theories and provide basis needed for research evaluation ethics.

show abstract

Journal Peer Review and Editorial Evaluation: Cautious Innovator or Sleepy Giant?

Cited by 32 publications

References 47 publications

Innovating editorial practices: academic publishers at work

Innovating editorial practices: academic publishers at work

Innovating Editorial Practices: Academic Publishers at Work

Ethical Theories in Research Evaluation: An Exploratory Approach

Contact Info

Product

Resources

About