Software Freedom and Scholarship: Reproducible Research

An article about computational result is advertising, not scholarship. Actual scholarship is the full software environment, code and data, that produced the result. - Claerbout and Karrenbach, Proceedings of the 62nd Annual International Meeting of the Society of Exploration Geophysics. 1992

The issue of computational result being present as scholarship is common today. When a study reviewed 613 papers in eight computer science conferences and five ACM journals, they found they could only reproduce 24.9% ². As an anecdote: I have seen plenty of articles where the code is not available or the code is proprietary. The lack of reproducibility in science is related to a number of other issues but in this post, I would like to detail one: studies often present results that were generated using proprietary software. As a result of scholars using software that does not guarantee the four essential freedoms, distribution and investigation is limited. Consequently, reproducibility is hindered. Furthermore, scholars often neglect publishing the in-house code used to generate the results. As a result, readers cannot run and inspect the software on their own computers.

Reproducible research is academic research where the final result contains the publication of everything that was used in the project: including notebooks and full computational environment. This computational environment includes: the code, data, etc used to produce the academic work ^{3 , 4}. In other words, reproducible research implores scholars to use and write software that guarantees its users the following essential freedoms ^{4 , 5}:

1. The freedom to run the program as you wish, for any purpose (freedom 0).
2. The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.
3. The freedom to redistribute copies so you can help your neighbour (freedom 2).
4. The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.

The prevalence of proprietary software in academia is a threat to the legitimacy of academia. The programs used and their source code should be expected in the Materials and Methods section of an academic article alongside the four essential freedoms.

In the 2010 issue of Economic Review, economists Carmen Reinhart and Kenneth Rogoff (RR) published a paper whose conclusions favoured austerity. Specifically, it was concluded that a given country’s GDP declines by two percent when “gross external debt reaches 60 percent of GDP” and when external debt is greater than 90%, GDP growth is halved ⁶. The paper was published in the aftermath of the 2008 economic crash and was widely cited by Paul Ryan’s Republican Party Budget and British MP George Osborne ^{7 , 8}.

Whatever your political leanings may be, the paper has multiple methodological flaws ⁹. Hermon, Ash and Pollin (HAP) requested the code and data. HAP published a rebuttal which showed a coding error that resulted in the results ignoring data from several countries ⁷. RR conceded that this coding error had been made ¹⁰.

Beyond quality, this case study portrays the threat that irreproducible research has towards society. If HAP did not have the freedom to check RR’s research, we might still be operating under the conclusions of the paper.

Free software is about having control over the technology we use in our homes, schools and businesses, where computers work for our individual and communal benefit.¹¹

Free software is necessary to create a free society in which we are able to criticize the decisions which are generated by computers.

On a larger scale, the issue of proprietary software restrictions appears in many ordinary computer labs. Students are at the mercy of their institutions to use whatever software is installed. It may be the industry standard, but the benefits associated with being the industry standard end when students want to learn and critique the decisions the software makes in their lives. Restrictions are rampant with proprietary software, and non free software often resists reverse engineering to prevent discovering how calculations are done. Indeed, the license restrictions threatens the user with monetary and/or legal repercussions for investigating the system. Common examples are Matlab, Solidworks, Eagle, ANSYS. In this way, proprietary software is a threat to the social mission of school itself: “to teach students to be citizens of a strong, capable, independent, cooperating and free society” ¹².

Free software has an exclusive feature that it guarantees to students: the freedom to learn ¹³. Scholars are not to be blamed for prevalence of proprietary software in schools. Proprietary software is entrenched in society today and we cannot control the circumstances we grow up in. Going forward, it is up to educators to recognize the threat proprietary software poses to society, and promote software that does social good. Indeed, educators can lead in making software freedom the industry standard.

In an ideal world, all the software used in a study (including its dependencies ie. the operating system and programming language) would be released as free software. Freedom in software at this point will come gradually. For example, if you must use Matlab, please consider publishing your scripts under a free license ^{13 , 14}.

If you are making a study, please commit yourself to using as much free software that is available and you have time to transition to. I hope this article serves as an introduction to why software freedom matters in academia. Precisely, the future of academia, its legitimacy, and its ability to criticize the system we live under is at risk.

This post was inspired by this video. Special thanks to Evan Misshula for describing the issue of reproducible science and its relation to software freedom. Please note: I love emacs but reproducible research is not as difficult as learning emacs and using it for your research. In the first half Evan Misshula’s talk, he recommends other free software projects with lower barriers to entry that are also committed to helping scholars create reproducible research. For notebooks that encourage radical reproducibility: jupyter Rmarkdown

Other notable projects for scholars include: http://www.runmycode.org/ and ².

Of course, I am available via email: anjan -at- momi.ca for free software recommendations =).

Further readings with recommendations are in the footnotes. Notably: ^{2 , 4 , 3}.

I use proprietary software for my academic work. I use ansys, matlab, solidworks, etc. Proprietary software is entrenched in society and the free software movement has a lot of work to do. This is an article to make other scholars aware of software freedom and what it can do for academia.

guest editor: Colin Leitner http://www3.telus.net/colinl333