Reproducible Research Practices and Transparency across the Biomedical Literature

Iqbal, Shareen; Wallach, John R.; Khoury, Muin J.; Schully, Sheri D.; Ioannidis, John P. A.

doi:10.1371/journal.pbio.1002333

Cited by 274 publications

(299 citation statements)

References 39 publications

Supporting

Mentioning

276

Contrasting

Unclassified

Order By: Relevance

“…For making their data available to us and their research transparent, the authors should be applauded. Indeed, recent empirical evaluations have shown that the published biomedical literature generally lacks transparency, including public access to raw data and code (Goodman, Fanelli, & Ioannidis, 2016;Iqbal, Wallach, Khoury, Schully, & Ioannidis, 2016;Leek & Jager, 2016). Furthermore, a recent series of studies has indicated that half of the published psychology papers include at least one statistical inconsistency, and one in eight even a gross inconsistency (Nuijten, Hartgerink, van Assen, Epskamp, & Wicherts, 2015).…”

Section: Discussionmentioning

confidence: 99%

A critical reexamination of doing arithmetic nonconsciously

Moors

Hesselmann

2017

Psychon Bull Rev

View full text Add to dashboard Cite

A recent study claimed to have obtained evidence that participants can solve invisible multistep arithmetic equations (Sklar et al., 2012). The authors used a priming paradigm in which reaction times to targets congruent with the equation's solution were responded to faster compared with incongruent ones. We critically reanalyzed the data set of Sklar et al. and show that the claims being made in the article are not fully supported by the alternative analyses that we applied. A Bayesian reanalysis of the data accounting for the random variability of the target stimuli in addition to the subjects shows that the evidence for priming effects is less strong than initially claimed. That is, although Bayes factors revealed evidence for the presence of a priming effect, it was generally weak. Second, the claim that unconscious arithmetic occurs for subtraction but not for addition is not supported when the critical interaction is tested. Third, the data do not show well-established features of numerosity priming as derived from V-shaped response time curves for prime-target distances. Fourth, we show that it is impossible to classify reaction times as resulting from congruent or incongruent primetarget relationships, which should be expected if their results imply that participants genuinely solve the equations on each trial. We conclude that the claims being made in the original article are not fully supported by the analyses that we apply. Together with a recent failure to replicate the original results and a critique of the analysis based on regression to the mean, we argue that the current evidence for unconscious arithmetic is inconclusive. We argue that strong claims require strong evidence and stress that cumulative research strategies are needed to provide such evidence.

show abstract

Section: Discussionmentioning

confidence: 99%

A critical reexamination of doing arithmetic nonconsciously

Moors

Hesselmann

2017

Psychon Bull Rev

View full text Add to dashboard Cite

show abstract

“…There has been an increasing movement towards "open science", an umbrella term that covers study registration, data sharing, public protocols and more detailed, transparent reporting. [19][20][21][22][23][24][25][26][27][28] To address these issues in the field of healthcare data- did you actually do?). This paper led by ISPE focuses on the latter topic, reporting of the specific steps taken during study implementation to improve reproducibility and assessment of validity.…”

Section: Introductionmentioning

confidence: 99%

Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0

et al. 2017

View full text Add to dashboard Cite

Purpose: Defining a study population and creating an analytic dataset from longitudinal healthcare databases involves many decisions. Our objective was to catalogue scientific decisions underpinning study execution that should be reported to facilitate replication and enable assessment of validity of studies conducted in large healthcare databases. This article is a joint publication by Pharmacoepidemiology and Drug Safety and Value in Health. Methods:This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

show abstract

“…The lack of availability of original, primary scientific data represents a major factor contributing to reproducibility problems (Iqbal et al, 2016). The structural biology community (led by protein crystallographers) has already taken significant steps towards making experimental data available.…”

Section: Introductionmentioning

confidence: 99%

A public database of macromolecular diffraction experiments

Grabowski

Langner

Cymborowski

et al. 2016

Acta Cryst Sect D Struct Biol

108

View full text Add to dashboard Cite

The low reproducibility of published experimental results in many scientific disciplines has recently garnered negative attention in scientific journals and the general media. Public transparency, including the availability of `raw' experimental data, will help to address growing concerns regarding scientific integrity. Macromolecular X-ray crystallography has led the way in requiring the public dissemination of atomic coordinates and a wealth of experimental data, making the field one of the most reproducible in the biological sciences. However, there remains no mandate for public disclosure of the original diffraction data. The Integrated Resource for Reproducibility in Macromolecular Crystallography (IRRMC) has been developed to archive raw data from diffraction experiments and, equally importantly, to provide related metadata. Currently, the database of our resource contains data from 2920 macromolecular diffraction experiments (5767 data sets), accounting for around 3% of all depositions in the Protein Data Bank (PDB), with their corresponding partially curated metadata. IRRMC utilizes distributed storage implemented using a federated architecture of many independent storage servers, which provides both scalability and sustainability. The resource, which is accessibleviathe web portal at http://www.proteindiffraction.org, can be searched using various criteria. All data are available for unrestricted access and download. The resource serves as a proof of concept and demonstrates the feasibility of archiving raw diffraction data and associated metadata from X-ray crystallographic studies of biological macromolecules. The goal is to expand this resource and include data sets that failed to yield X-ray structures in order to facilitate collaborative efforts that will improve protein structure-determination methods and to ensure the availability of `orphan' data left behind for various reasons by individual investigators and/or extinct structural genomics projects.

show abstract

Reproducible Research Practices and Transparency across the Biomedical Literature

Cited by 274 publications

References 39 publications

A critical reexamination of doing arithmetic nonconsciously

A critical reexamination of doing arithmetic nonconsciously

Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0

A public database of macromolecular diffraction experiments

Contact Info

Product

Resources

About