Updates on program committee & desk rejections
Planning and preparation for NeurIPS 2020 has continued apace since our last post, despite the extraordinary global emergency which has unfolded around us over the last several weeks. Truly, it can feel like a jarring incongruity to deal with cancelling, postponing, and virtualizing conferences at the same time as recruiting area chairs and configuring CMT for NeurIPS submissions, and some might wonder why we are not throwing in the towel already and accepting that even NeurIPS may be disrupted by Covid-19. The general chair and program chairs as well as the board are optimists, however, and we are taking the hopeful view that NeurIPS will be able to go on as planned, in snowy Vancouver in December. We are therefore planning accordingly, although we are also monitoring the situation to adjust if needed.
In the past month, preparations have taken the form of recruiting the program committee, designing an ethical review process (a first for NeurIPS), pinning down the details of the early reject workflow, and thinking about how to ensure high-quality paper-reviewer assignments. We have also worried about coronavirus and how it will affect our process, your submissions, and the conference in December. Starting with this last topic, since it is certainly on your minds as well as ours, we would like to confirm that the paper submission deadline will be May 12, with no extension currently planned. Although we sympathize greatly with the challenges and constraints that the ML community is facing right now, we simply do not have enough flexibility in our timeline to significantly extend the deadline. The Canadian immigration office requires 12 weeks to process visa requests, which means that we need to notify authors in early September. This provides a backstop to the review process, which includes many phases: bidding, matching, reviewing, author feedback, discussion, meta-reviewing, and final decision making. Since we don’t think that it would be prudent to shorten any of those phases given the continued disruption of the pandemic and the very large number of submissions we still expect to receive, this leads us to conclude that we must maintain the May 12 paper submission deadline so that we can deliver a sufficiently high-quality reviewing process. We understand that this is not a popular stance, and we have thought about it very carefully. Another question which has been raised relates to the dual submission policy, which states that papers which are under review at any other archival venue cannot be submitted to NeurIPS unless they are first withdrawn from the other venue (see the Call for Papers and the FAQ). We have been asked to change this policy to allow papers under submission at ICML, UAI, and other conferences with notification dates later than May 12 to be submitted to NeurIPS on the understanding that they will be withdrawn from NeurIPS review if they are accepted. We will not allow this, as it tacitly encourages thousands of papers to be submitted and then possibly withdrawn, which makes it very hard to proceed with the initial stages of the review process (bidding, matching, and desk rejection) without compromising on quality or overburdening our program committee members. While we are sympathetic to ICML authors that were planning to submit to NeurIPS as a backup plan and are now faced with a difficult decision, we note that there are many ML conferences every year — ICLR, AISTATS, AAAI, to name a few — whose deadlines are only a few months after the ICML notifications.
Many of you have accepted our invitation to be a part of the 2020 review process, and we thank you in advance for donating your time and effort! The program committee includes 62 Senior Area Chairs, which can be broken down by subject area into Algorithms (20), Deep Learning (14), Applications (11), Reinforcement Learning and Planning (5), Probabilistic Methods (4), Theory (3), Optimization (3), and Neuroscience (2). This closely matches the distribution of paper topics received in 2019, so hopefully we have an appropriate representation of expertise. We have also recruited 486 Area Chairs, with a similar distribution over topic areas. Interestingly, 74% of the invited Senior Area Chairs and 64% of the invited Area Chairs accepted our invitation, showing a great deal of support and enthusiasm for this conference as well as for the changes we have made this year. Having discussed all candidates and taken into account their research and program committee experience and level of seniority, we are confident that this is an exceptionally strong team and we look forward to working with them closely in the months to come. Recruiting reviewers is currently ongoing, with over 9000 invitations issued and almost 5000 accepted so far. We have sent invitations to NeurIPS 2019 reviewers who received average-or-better ratings, SAC and AC-nominations, authors of 2 or more top-tier ML papers, and self-nominated individuals who were manually screened for appropriate qualifications (experience equivalent to senior PhD students or above and significant publishing experience at ML conferences: at least two accepted papers at NeurIPS or similar top-tier conference or journal). If you would like to review for NeurIPS, you can nominate yourself by filling out this form.
Next we describe the new ethical review process which will be introduced for 2020, motivated by the increased relevance of our field in the ‘real world’. Directly following from a more explicit handling of broader impact and societal consequences, we are going to recruit a pool of emergency reviewers who have expertise in the fields of AI ethics, fairness, accountability, and transparency with a research emphasis on ethics. Notwithstanding that papers will be evaluated and rated by reviewers, ACs, and SACs based on technical merits, papers can also be flagged by program committee members for potential ethical concerns. Such papers will be reviewed by this separate pool of experts on ethics who can better assess whether a submission should be rejected on the basis of ethical concerns — in particular research that has the primary purpose of harming people or approaches that create or reinforce unfair social biases. We believe that this will be a very rare outcome, but one which should be prepared for and handled fairly.
Another area of activity in recent weeks concerns the early rejection process which was announced in our first blog post. To recap, we have decided to implement a policy similar to that employed by journals and some conferences: senior members of the program committee (Area Chairs) will be asked to identify lower-quality or less-relevant papers which will then be rejected without receiving further review. Recognizing that this will be a challenging task for Area Chairs, we have ensured that we have a sufficient number of ACs so that each will be assigned no more than 20 papers, and out of those identify the 20% which are most likely to be rejected by reviewers. We believe this is a manageable assignment for ACs. Senior ACs are then asked to cross-check all papers flagged for desk rejection. Only papers where both ACs and Senior ACs agree to desk-reject, will be rejected. Lastly, we also recognize that receiving an early reject decision could be frustrating for some authors, so ACs will be asked to provide high-level justification for the rejection and this limited feedback will be shared with authors.
We will post additional updates in the next few weeks; until then we hope that everyone in the global NeurIPS community and beyond stays safe and healthy.
Hsuan-Tien Lin, Maria Florina Balcan, Raia Hadsell and Marc’Aurelio Razato
NeurIPS 2020 Program Chairs
