Victoria Stodden, Marcia McNutt (President of the American Association for the Advancement of Science), David H. Bailey, Ewa Deelman, Yolanda Gil, Brooks Hanson, Michael Heroux, John Ioannidis and Michela Taufer have published an article in Science (the principal journal of the AAAS) entitled Enhancing reproducibility in computational methods.
In this article we argue that the field of mathematical and scientific computing lags behind other fields in establishing a culture and tools to ensure reproducibility. All too often, the authors of computations, even those that are published in peer-reviewed conferences and journals, have not fully documented their algorithms, code, input data and output, nor have they made this code and data available on a public repository. Further, in all too many cases even the authors themselves no longer have the source code and other data that was used for their runs. Thus it is increasingly difficult for other researchers (or even the same researchers) to reproduce published work.
We list a set of seven recommendations to address this:
- Share data, software, workflows, and details of the computational environment that generate published findings in open trusted repositories.
- Persistent links should appear in the published article and include a permanent identifier for data, code, and digital artifacts upon which the results depend.
- To enable credit for shared digital scholarly objects, citation should be standard practice.
- To facilitate reuse, adequately document digital scholarly artifacts.
- Use Open Licensing when publishing digital scholarly objects.
- Journals should conduct a reproducibility check as part of the publication process and should enact the TOP standards at level 2 or 3.
- To better enable reproducibility across the scientific enterprise, funding agencies should instigate new research programs and pilot studies.
Full details are available in the Science article.