Getting Started with NeurIPS 2020
Hello and welcome! This is the first in a series of notes that we will write to shed light on the organization, review process, and program for NeurIPS 2020.
We are Hsuan-Tien Lin, Maria-Florina Balcan, Raia Hadsell and Marc’Aurelio Ranzato, a diverse team composed of researchers with different backgrounds and expertise, who hail from 3 continents and speak four different native languages… who are enthusiastically pitching in together to organize this year’s conference! As this year’s Program Chairs, we will work together with Hugo Larochelle, who is the 2020 General Chair. So far, we’ve been having twice-weekly video conferences, daily messaging, and have put in lots of hard work (and the conference is still 10 months away!). We are committed to making NeurIPS 2020 the most successful year yet, by inspiring a new generation of scientists, by giving a place for the community to meet and spark the discussions that will lead to tomorrow’s breakthroughs, and by recognizing the remarkable achievements in machine learning that are happening all around the world.
To keep NeurIPS thriving we are instituting some significant changes in the submission and review process.
TL;DR:
0) Watch this video summarizing the major changes! or keep reading…
1) This year’s abstract submission deadline is moved up to May 5th, 2020, while the paper submission deadline is on May 12th, 2020. This shift allows us to handle the increased volume of submissions and accommodates the early rejection phase of the review process… which brings us to the next point.
2) Early rejection: Area Chairs will have two weeks to recommend papers for early rejection. We expect up to 20% of the papers to be selected for this category. Senior Area Chairs then have one week to approve the decision, at which point authors of rejected papers will be notified that their paper won’t undergo any further review.
3) Authors are reviewers: We require that every (co-)author of every paper agrees to review papers, if asked. This requirement is useful for increasing our reviewer pool size and to fairly distribute the reviewing load more evenly among community members who submit papers.
4) Broader impact: Authors are asked to include a section in their submissions discussing the broader impact of their work, including possible societal consequences — -both positive and negative.
5) Video spotlight: We request that all authors upload a video recording of the spotlight presentation of their work at camera-ready submission time and we are also working on ways of enabling remote presentations as well as remote attendance.
LONG VERSION
When we started planning for NeurIPS 2020 last year, well before the opening of the 2019 conference, we focused on three issues that we felt were critical: First, to accommodate the no-end-in-sight growth of submissions and attendees; second, to grapple with the ethical considerations and implications of our research; and third, to implement sustainable practices such as supporting remote attendance and participation. Each of these concerns has yielded some concrete changes to the review process or to the conference itself.
First, the most pressing concern is the growth of NeurIPS, year after year, with respect to both submitted papers and attendees. While some changes will be made to the conference program in December (avoiding poster stampedes and blockades is first on the list), we also need to find ways to scale the reviewing process. Right now, NeurIPS is trying to do journal-quality peer review at a previously unimaginable scale and pace. No other field attempts to source double-blind, high-quality reviews for over 6000 full length papers in matter of a few weeks, as was done in NeurIPS 2019. For 2020, we expect the bar to rise further, with a projected 10,000 submissions, which would necessitate a mind-boggling 30,000 reviews under the current review process. Recognizing that producing such a high volume of quality reviews is not tenable, we are implementing changes to the structure to allow for a more manageable process.
First, we have decided to proceed with an early rejection phase, an idea that has been discussed for years but not implemented (until now: this year IJCAI has also adopted early rejection). This modification is the most direct approach for managing growth, and one that is commonly employed by journals: submissions that are poorer quality or less relevant are declined by editors of the journal (who are presumed to have good judgement and broad expertise), and not sent for review. We are going to implement this process for NeurIPS by asking Area Chairs to assess their assigned papers and identify up to 20% that are likely to be rejected. These papers are further checked by Senior Area Chairs, who confirm the rejection. Authors are notified promptly, and the papers are removed from the review process, thus allowing reviewers to spend their valuable time on higher quality papers. We know that the spectre of receiving a summary rejection is stressful, but we hope that it will encourage authors to submit only high-quality work.
Another change that we hope will improve the review process is to require authors to disclose whether their paper is a resubmission (that is, it has been previously rejected from a peer-reviewed venue). Additionally, they are asked to summarize the reason for the rejection and enumerate the changes that have been made to the current version to address the concerns. This step will give helpful context to the reviewers, especially since they may have reviewed the paper previously and will want to know the key changes, and will encourage authors to improve their papers prior to resubmission.
To better handle the growth in terms of number of attendees and to address the carbon-footprint concerns of a growing international conference, we are working to enable, and even encourage, remote attendance. In the past few years, the conference has sold out very quickly, to the point that registration tickets are allocated through a lottery system. Following last year’s success, we are continuing to promote meetups all over the world as a way for community members to remotely attend the conference and to network. To further enable remote access of content, this year, presenters will be required to upload a video presentation of their work together with the camera-ready version of their paper. Moreover, registered attendees will be asked whether they want to receive personalized recommendations for published papers to read based on our paper-matching system, as it is often difficult to discover papers of interest in the ever-growing proceedings. And finally, we are exploring ways to enable remote presentations of both orals and posters. We will share more details about our plans in the near future.
To promote a healthier and more sustainable growth path of the peer-reviewing system, we require that each co-author serves as a program-committee member, if asked. While in an ideal “peer-reviewing economy” authors would review proportionally to the number of papers they submit and submissions would be contingent on the quality of the reviewing service provided, we decided to make a less drastic yet important step in this direction by requiring all co-authors to register in CMT and enlist as reviewers. Failure to comply may result in their paper being desk-rejected. We hope that this requirement will increase our reviewer pool, spread the reviewing load more evenly in the community, and encourage authors to be more sympathetic towards the efforts of the program-committee members.
Finally, the impact of the NeurIPS community has been more and more pervasive in our society thanks to the myriad of applications derived from the work presented at this conference. It is therefore incumbent on all of us to think more broadly about what it means to develop new methods and systems, and to consider not only the beneficial applications and products enabled by our research, but also potential nefarious uses and the consequences of failure. To encourage a broader and more balanced perspective, we ask authors to dedicate a section of their paper to discussing broader impacts. This additional section can be placed after the 8 page limit together with the acknowledgments and references.
Lastly, we are asking authors to disclose both direct funding sources as well as indirect financial relationships that could possibly result in a conflict of interest or competing interest situation. This practice is standard for the natural and physical sciences as well as medicine, and we believe that it is appropriate for the machine-learning community as well. This disclosure, like the acknowledgements, could compromise double-blind reviewing, so it is required to be added only to the camera-ready version.
We would like to conclude with a personal note. We are truly honored to serve this community by organizing NeurIPS. The task is objectively daunting for the scale, the number of individuals involved and the complexity of the whole process. Participating in this organization is taking a significant toll on our research work and time we can spend with our families. However, it is also a lot of fun to work together on something bigger than ourselves, and more importantly, it is perhaps the best way we can give back to the NeurIPS community that has shaped both our careers and society as well.
We will post additional updates in a few weeks; stay tuned!
Meantime, please checkout the call for papers and the video.
Hsuan-Tien Lin, Maria-Florina Balcan, Raia Hadsell and Marc’Aurelio Ranzato
NeurIPS 2020 Program Chairs
