What Does It Mean If Data Are Reproducible But Not Accurate?

Abstruse

Computational science has led to exciting new developments, only the nature of the work has exposed limitations in our power to evaluate published findings. Reproducibility has the potential to serve equally a minimum standard for judging scientific claims when full contained replication of a study is not possible.

The ascent of computational scientific discipline has led to exciting and fast-moving developments in many scientific areas. New technologies, increased calculating power, and methodological advances have dramatically improved our ability to collect circuitous hig-dimensional data (1, ii). Large data sets have led to scientists doing more than ciphering, too as researchers in computationally oriented fields straight engaging in more science. The availability of big public databases has allowed for researchers to make meaningful scientific contributions without using the traditional tools of a given field. As an instance of this overall trend, the Sloan Digital Heaven Survey, a big publicly available astronomical survey of the Northern Hemisphere, was ranked the most cited observatory (three), assuasive astronomers without telescopes to make discoveries using data collected by others. Similar developments tin can be plant in fields such as biology and epidemiology.

Replication is the ultimate standard by which scientific claims are judged. With replication, independent investigators address a scientific hypothesis and build up evidence for or against information technology. The scientific community's "civilisation of replication" has served to quickly weed out spurious claims and enforce on the community a disciplined approach to scientific discovery. Withal, with computational science and the respective collection of large and complex data sets the notion of replication can be murkier. Information technology would require tremendous resource to independently replicate the Sloan Digital Sky Survey. Many studies—for example, in climate—science require computing power that may not exist available to all researchers. Even if computing and data size are not limiting factors, replication tin be difficult for other reasons. In environmental epidemiology, large cohort studies designed to examine subtle wellness effects of environmental pollutants tin can exist very expensive and require long follow-up times. Such studies are difficult to replicate because of time and expense, especially in the time frame of policy decisions that need to be made regarding regulation (ii).

Researchers across a range of computational science disciplines have been calling for reproducibility, or reproducible research, as an attainable minimum standard for assessing the value of scientific claims, peculiarly when full independent replication of a report is not viable (4–8). The standard of reproducibility calls for the information and the computer code used to analyze the information exist made available to others. This standard falls short of full replication because the aforementioned data are analyzed over again, rather than analyzing independently collected data. Withal, under this standard, express exploration of the data and the analysis code is possible and may be sufficient to verify the quality of the scientific claims. One aim of the reproducibility standard is to fill the gap in the scientific prove-generating procedure between total replication of a study and no replication. Between these two extreme end points, there is a spectrum of possibilities, and a study may be more or less reproducible than another depending on what data and code are made available (Fig. one). A recent review of microarray gene expression analyses found that studies were either not reproducible, partially reproducible with some discrepancies, or reproducible. This range was largely explained by the availability of data and metadata (9).

An external file that holds a picture, illustration, etc. Object name is nihms382015f1.jpg

The spectrum of reproducibility.

The reproducibility standard is based on the fact that every computational experiment has, in theory, a detailed log of every action taken by the figurer. Making these computer codes available to others provides a level of detail regarding the analysis that is greater than the analagous non-computational experimental descriptions printed in journals using a tongue.

A critical bulwark to reproducibility in many cases is that the computer code is no longer bachelor. Interactive software systems often used for exploratory data analysis typically do not keep track of users' deportment in any concrete form. Fifty-fifty if researchers apply software that is run by written code, often multiple packages are used, and the code that combines the different results together is not saved (10). Addressing this trouble volition require either changing the beliefs of the software systems themselves or getting researchers to use other software systems that are more acquiescent to reproducibility. Neither is likely to happen quickly; one-time habits dice hard, and many will be unwilling to discard the hours spent learning existing systems. Non–open source software can only be inverse past their owners, who may not perceive reproducibility as a high priority.

In gild to advance reproducibility in computational science, contributions will need to come from multiple directions. Journals tin play a role here as role of a joint effort by the scientific community. The journal Biostatistics, for which I am an associate editor, has implemented a policy for encouraging authors of accepted papers to make their work reproducible by others (eleven). Authors can submit their lawmaking or data to the periodical for posting as supporting online material and can additionally request a "reproducibility review," in which the associate editor for reproducibility runs the submitted code on the data and verifies that the code produces the results published in the article. Articles with data or code receive a "D" or "C" kite-marker, respectively, printed on the first page of the article itself. Articles that take passed the reproducibility review receive an "R." The policy was implemented in July 2009, and as of July 2011, 21 of 125 articles have been published with a kite-marking, including five manufactures with an "R." The manufactures have reflected a range of topics from biostatistical methods, epidemiology, and genomics. In this absolutely minor sample, we have non yet encountered cases in which data and code were submitted for reproducibility review but results were non reproducible as claimed. It is encouraging that authors are taking advantage of the policy to make their piece of work reproducible by others, but more piece of work could be done to promote a broader adoption of the policy.

The fact that an analysis is reproducible does non guarantee the quality, correctness, or validity of the published results. The "R" kite-mark is meant to convey the thought that a knowledgeable individual has reviewed the code and data and was capable of producing the results claimed by the author. In cases in which questionable results are obtained, reproducibility is disquisitional to tracking downwardly the "bugs" of computational science. In cases with interesting findings, reproducibility can profoundly facilitate building on those findings (12).

Mayhap the biggest barrier to reproducible research is the lack of a deeply ingrained culture that simply requires reproducibility for all scientific claims. Not unlike the culture of replication that persists across all scientific disciplines, the scientific community needs to develop a "culture of reproducibility" for computational science and require it of published claims. Some other important barrier is the lack of an integrated infrastructure for distributing reproducible research to others. The current system is ad hoc with researchers in some fields having access to sophisticated central data repositories and researchers in other fields having few useful resources for sharing lawmaking and data. In many cases, a researcher does non take an obvious place to turn to make sure their work is reproducible and attainable past others. Journals' supporting online materials have some severe limitations, such every bit the inability to search and alphabetize bachelor information.

Given the barriers to reproducible inquiry, it is tempting to look for a comprehensive solution to arrive. However, fifty-fifty incremental steps would be a vast improvement over the current situation. To this end, I advise the following steps (in lodge of increasing affect and cost) that individuals and the scientific community can accept. First, anyone doing any computing in their research should publish their code. Information technology does not have to be clean or cute (13), it just needs to be available. Even without the corresponding information, code can be very informative and tin can be used to check for problems as well as quickly translate ideas. Periodical editors and reviewers should demand this and then that it becomes routine. Publishing code is something we can do at present for almost no boosted cost. Free code repositories already exist [for case, GitHub (http://github.com) and SourceForge (http://sourceforge.net)], and at a minimum, code can be published in supporting online material. The next step would be to publish a cleaned-upward version of the code along with the information sets in a durable non-proprietary format. This will involve some additional cost because non anybody will have the resource to publish data. Some fields such as genomics have already created data repositories, only there is not yet a full general solution.

Concluding, the scientific community tin can pool its commonage resources to create a DataMed Central and CodeMed Central, analogous to PubMed Cardinal for all information, metadata, and lawmaking to be stored and linked with each other and with corresponding publications. Such an attempt would probably need regime coordination and support, merely each would serve as a single gateway that would guide researchers to field-specific data and code repositories. Existing repositories could continue to exist used and would interface with the gateway, whereas fields without existing infrastructure would be given access to these resources. The ultimate goal would exist to provide a unmarried identify to which people in all fields could plough to make their work reproducible.

The field of scientific discipline volition not alter overnight, but but bringing the notion of reproducibility to the forefront and making it routine will make a deviation. Ultimately, developing a civilisation of reproducibility in which information technology currently does not exist will require time and sustained effort from the scientific customs.

References

1. Hanson B, Sugden A, Alberts B. Science. 2011;331:649. [PubMed] [Google Scholar]

2. Peng RD, Dominici F, Zeger SL. Am J Epidemiol. 2006;163:783. [PubMed] [Google Scholar]

3. Madrid JP, Macchetto D. Bull AAS. 2009 arXiv:0901.4552. [Google Scholar]

4. Schwab M, Karrenbach N, Claerbout J. Comput Sci Eng. 2000;ii:61. [Google Scholar]

5. Laine C, Goodman SN, Griswold ME, Sox HC. Ann Intern Med. 2007;146:450. [PubMed] [Google Scholar]

6. King Thousand. PS: Polit Sci Polit. 1995;28:444. [Google Scholar]

7. Gentleman R. Stat Appl Genet Mol Biol. 2005;four:article 2. [PubMed] [Google Scholar]

8. Yale Law School Roundtable on Data and Code Sharing. Comput Sci Eng. 2010;12:viii. [Google Scholar]

12. McCullough BD, McGeary KA, Harrison TD. Can J Econ. 2008;41:1406. [Google Scholar]