Call for Papers
Time for another update on preparations for NeurIPS 2019!
We’ve made a lot of progress in our recruitment of reviewers. We now have over 4200 confirmed reviewers. We can’t emphasize enough how much their engagement and dedication are greatly appreciated. NeurIPS would not be possible without them. We’re now confident we will reach our target of 4500 reviewers.
The Call for Papers
In the past month, we’ve also been working on finalizing the Call for Papers (CfP), i.e. the document meant to advertise and provide more details on the submission and review process for NeurIPS 2019. It is now on the NeurIPS website here. If you are hoping to submit your work, we strongly encourage you to read it in full.
Among many things, the CfP points to the 2019 style files, the FAQ (which will be updated moving forward based on questions we receive) and the updated list of subject areas (the least used subjects from 2018 were removed, some new ones were added and some subjects were moved under different parent subject areas).
Most importantly, the CfP highlights the biggest changes introduced this year. Only submissions with a proper title and abstract entered a week before (May 16) the full paper submission deadline (May 23) will be reviewed. This gives us as Program Chairs more time to adjust the Program Committee to better match the 2019 paper distribution. We are also adopting the same policy as ICML 2019 on author lists, whereby no changes will be permitted to the list of authors provided with the full paper submission. Having the complete list of authors at the beginning of the review process ensures that conflicts of interests with reviewers and area chairs are avoided. Additionally, to make NeurIPS 2019 content more accessible to people who cannot attend, one or more accompanying materials (video spotlight, slides, or poster) will have to be submitted for all accepted papers prior to the conference.
But the two most significant changes for NeurIPS 2019 are the introduction of 1) a mandatory Reproducibility Checklist for all submissions and 2) a formal statement of expectations regarding the submission of code through a new Code Submission Policy.
Reproducibility Checklist
The reproducibility of research published at NeurIPS and other conferences has been a subject of concern and debate by many in the community. The Posner Lecture at NeurIPS 2018 by Joelle Pineau (which you may view here) presented an overview of these concerns and challenges. For this reason, along with Hanna Wallach (NeurIPS 2019 General Chair), we decided to experiment with an initiative to bring more awareness towards this issue.
Joelle Pineau has been appointed Reproducibility Chair to coordinate this effort. Given her involvement and thought leadership on the subject, Joelle was a natural choice for this new role.
As part of the submission process, we are requiring responses to the questions from the Reproducibility Checklist that Joelle introduced at NeurIPS 2018 and has since been perfecting. The answers will be available to reviewers and area chairs, who may use this information to help them assess the clarity and potential impact of submissions.
Reproducibility Checklist responses will be analyzed, hopefully shedding more light on the state of reproducibility of research at NeurIPS.
Code Submission Policy
Another topic that has generated much debate recently is whether code should accompany a scientific machine learning paper and, if so, what mechanisms to put in place to make this more common.
We do believe that code is often a key artefact of the scientific process, and thus should play an important role in the dissemination of scientific findings in our community. For this reason, we’ve written a Code Submission Policy which clarifies the expectations of NeurIPS 2019 regarding the submission of code. We encourage everyone to read it in full.
In short, code is expected to accompany the camera-ready revisions of accepted papers covered by the policy (due on October 27). The policy describes the types of papers that should be accompanied by code and the sufficient conditions for code to reach the standards of the policy. Note that, as in previous years, code may also be provided at submission time, as supplementary material (though it should not reveal the identity of the authors).
This initiative is an experiment, which we hope will inform the policy for future years. As such, submission of code (for papers covered by the policy) is expected, although it will not be enforced. The Reproducibility Chair will guide the analysis on the impact of this measure.
We’ve considered introducing more formal incentives for code submission. However, through conversations with various people in the community (which started as early as last Fall), it has become clear that this topic requires we move at a careful pace, as we learn where our “comfort zone” is as a community.
We take this opportunity to thank Kamalika Chaudhuri, Ruslan Salakhutdinov, Yoshua Bengio, Rich Zemel, Samy Bengio, Hanna Wallach, Max Welling, Joelle Pineau and the members of NeurIPS Board in general for their feedback on our proposed Code Submission Policy.
That’s it for this time. We hope to have another update some time before the submission deadline. Stay tuned!
Alina Beygelzimer, Emily Fox, Florence d’Alché-Buc, Hugo Larochelle
NeurIPS 2019 Program Chairs
