52% Yes, a signiicant crisis 3% No, there is no crisis 7% Don't know 38% Yes, a slight crisis 38% Yes, a slight crisis 1,576 RESEARCHERS SURVEYED M ore than 70% of researchers have tried and failed to reproduce another scientist's experiments, and more than half have failed to reproduce their own experiments. Those are some of the telling figures that emerged from Nature's survey of 1,576 researchers who took a brief online questionnaire on reproducibility in research. The data reveal sometimes-contradictory attitudes towards reproduc-ibility. Although 52% of those surveyed agree that there is a significant 'crisis' of reproducibility, less than 31% think that failure to reproduce published results means that the result is probably wrong, and most say that they still trust the published literature. Data on how much of the scientific literature is reproducible are rare and generally bleak. The best-known analyses, from psychology 1 and cancer biology 2 , found rates of around 40% and 10%, respectively. Our survey respondents were more optimistic: 73% said that they think that at least half of the papers in their field can be trusted, with physicists and chemists generally showing the most confidence. The results capture a confusing snapshot of attitudes around these issues, says Arturo Casadevall, a microbiologist at the Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland. "At the current time there is no consensus on what reproducibility is or should be. " But just recognizing that is a step forward, he says. "The next step may be identifying what is the problem and to get a consensus. "

Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.

Software contributions to academic research are relatively invisible, especially to the formalized scholarly reputation system based on bibliometrics. In this article, we introduce a gold‐standard dataset of software mentions from the manual annotation of 4,971 academic PDFs in biomedicine and economics. The dataset is intended to be used for automatic extraction of software mentions from PDF format research publications by supervised learning at scale. We provide a description of the dataset and an extended discussion of its creation process, including improved text conversion of academic PDFs. Finally, we reflect on our challenges and lessons learned during the dataset creation, in hope of encouraging more discussion about creating datasets for machine learning use.

better practices. Ultimately we hope that CiteAs will increase the visibility of research software, improving incentives for the software work needed to advance research. CCS CONCEPTS• Human-centered computing → Collaborative and social computing systems and tools.

No abstract

The open science movement promotes use of digital technology to increase the efficiency, inclusivity, and quality of scientific research. Developers of these platforms often advocate for open science on the grounds that it is in keeping with scientific values, specifically referencing Mertonian norms. However, many scientists are agnostic toward open science; as policies and technology enforce the movement's aims of sharing openly, they seek to protect the research they view as their own. My dissertation work studies the enactment of open science by a variety of stakeholders in the Open Science Framework (OSF)-its developers, its users, and also its nonusers. Through interviews, trace data collection, and observation of these various populations, I will explore how these stakeholders construct different technologies-in-practice. By taking the OSF not as a given, but as a technology whose purpose and effects are structured by the constraints and resources of its stakeholders, I will unpack the tensions among open science advocates, agnostics, and their use of technology. With these insights, science policy and open systems can be designed to better accommodate the diverse concerns of stakeholders. CCS CONCEPTS• Software and its engineering → Collaboration in software development; • Information systems → Collaborative and social computing systems and tools; • Social and professional topics → User characteristics.

No abstract

No abstract