Behind the Program for Reproducibility at NeurIPS 2019

The Neural Information Processing Systems (NeurIPS) conference has long been the leading venue for new and exciting machine learning research. This year, the spirit of innovation and scientific leadership goes beyond the conference content, with the creation of the new role of Reproducibility Chair within the program committee. This guest blog post is written by the Reproducibility Chairs for NeurIPS 2019, Joelle Pineau and Koustuv Sinha, to explain the reproducibility program being rolled out to support high-quality scientific contributions at the conference and beyond.

One of the challenges in machine learning research is to ensure that presented and published results are sound and reliable. Reproducibility, that is obtaining similar results as presented in a paper or talk, using the same code and data (when available), is a necessary step to verify research findings. Reproducibility is also an important step to promote open and accessible research, thereby allowing the scientific community to quickly integrate new findings and convert ideas to practice. Reproducibility also promotes the use of robust experimental workflows, which potentially reduce unintentional errors.

The NeurIPS 2019 program committee has selected 1430 accepted papers, from a set of 6743 submissions. While the conference committee, year after year, implements a rigorous review process, it remains difficult to assess the quality of the work on the basis of a paper alone, as shone a few years ago by the NeurIPS consistency experiment. The goal of the reproducibility program at NeurIPS 2019 is not to interfere with the review process. But rather to complement this by the use of new tools and activities designed to encourage high quality scientific output.

As reproducibility chairs and in collaboration with the program chairs, our program for 2019 contains three major components:

1. Reproducibility checklist: Reproducibility is hard — even in highly deterministic and open field such as computer science. The reproducibility checklist was designed to verify several components of a solid paper. It places particular emphasis on good empirical methods. It can be used by researchers at any stage of their work. For NeurIPS 2019, authors of all submitted papers had to file the checklist along with their submission. Authors of accepted papers will have to file again when submitting their final camera-ready version. We will analyze responses to discover trends in research practices.
2. Code submission policy: Traditionally, reviewers were tasked with determining the scientific quality and impact of a work from the manuscript along. In recent years, there has been a strong movement towards encouraging authors to provide other artifacts to support their work, including code. At NeurIPS 2019, authors of accepted papers are encouraged to provide code with the final paper version. Of course this will be too late to be considered during the review process, but the code will facilitate reproducibility of the work, and further discoveries, well beyond the review period by the entire community. Furthermore, we will be able to analyze characteristics of the code submitted (who, what, how), and link this with responses to the reproducibility checklist to get a richer picture of our community, its research practices, and support of open science and reproducibility. We will report on the results of this analysis at the NeurIPS conference in Vancouver.
3. Reproducibility Challenge @ NeurIPS 2019 : Following previous editions (ICLR 2018, ICLR 2019), this most recent edition of the reproducibility challenge provides a great opportunity for members of the machine learning community to dive deep into cutting-edge research by aiming to re-implement parts of a paper. It offers a unique opportunity for authors to strengthen their own contributions by working with challenge participants, addressing their queries and incorporating their feedback if necessary using the OpenReview platform. Participants at all expertise levels are welcome. It is not too late to sign up!

Reproducibility in machine learning has recently garnered a considerable amount of attention and momentum thanks to key efforts by top researchers. Conferences such as ICLR, AAAI, ICML have organized dedicated workshops on the topic. The premier conference in the field, NeurIPS, has recently adopted the pledge of reproducibility as part of their submission process. We hope our endeavor will similarly spur more efforts in reproducing existing ideas and papers, and in turn promote open, accessible and sound machine learning research.

Joelle Pineau and Koustuv Sinha

Reproducibility Chairs for NeurIPS 2019