Submissions and Start of Reviewing
We are back with an update on what has been going on since our last post!
Yet again, NeurIPS has received a record number of submissions this year.
We originally received 9185 abstract submissions. Of these, 276 were placeholder abstracts: either nonsensical (“xyz”, “[placeholder]”), only containing an incomplete or uninformative sentence, or a copy of the title (or a trivial extension, such as “We study TITLE.”). These were desk rejected (i.e. rejected without further review).
Only 6947 of the remaining 8909 abstract submissions had a paper uploaded by the full paper submission deadline. Area chairs later inspected these submissions, and identified some cases violating the NeurIPS 2019 submission guidelines (e.g., non-anonymized submissions, content going beyond the 8-page limit or not following the official NeurIPS style guidelines). We also rejected submissions that no longer had any of the authors originally listed on the abstract submission (suggesting the abstract had been recycled from a different paper).
The above process reduced the total submission count going through the full review process to 6809, an all-time high and a 40% increase compared to NeurIPS 2018. Last year, 4854 submissions were reviewed.
Unfortunately, the submission process didn’t go without difficulties. Minutes before the full paper submission deadline, the online platform used to support the review process, Conference Management Toolkit (CMT), became overwhelmed by the large number of submission requests. This made it impossible for many authors to submit the last version of their manuscript.
To address this, first we provided a 2 hour extension period (during which CMT resumed functioning normally), meant to accommodate these last minute submissions. However, since we weren’t able to broadly communicate this extension via CMT (because it was overwhelmed, instead we used the NeurIPS website, the FAQ page for submitting authors and Twitter), the next day it became clear that a significant number of authors were never informed of the extension and for that reason missed it.
So, we determined the best option was to reopen CMT for a complete 24-hour cycle, to provide an opportunity for an author anywhere in the world to submit the final version of their PDF manuscript (authors received information about this opportunity by email, sent through CMT).
These difficulties came as a surprise, since the CMT team was working closely with us on preparing for the NeurIPS submission deadline. All along, we have greatly appreciated the time and effort they’ve been putting (for free!) into tailoring CMT for the needs of NeurIPS. Also, thanks to having an earlier abstract submission deadline, the CMT team had the opportunity to leverage information on the number of full paper submissions to expect. But the CMT team later determined a performance issue with responses to the reproducibility checklist questions (a new feature for CMT) when users were editing their submissions. This issue is being addressed as we speak.
But, a lesson learned for future years is that a contingency plan for disruptions in the last minutes of the submission deadline should both be prepared and, perhaps most importantly, advertised to authors ahead of time, so everyone knows what to expect if there are any exceptional disturbances to CMT. We hope that all future organizing committees will learn from our experience!
Start of the Reviewing Process
The weeks following the full paper submission deadline is a particularly important period for program chairs. It’s the crucial period during which each submission is assigned an area chair (AC) and three reviewers, who evaluate the submission.
First, the AC assignment is performed. We start by asking ACs to bid on at least 40 submissions they would like to handle, by labeling their interest as either “eager”, “willing”, “in a pinch” or “not willing”. These preferences are combined with other information about the AC (expertise subject areas and TPMS profile) to generate an initial assignment of submissions to ACs. This assignment is generated automatically by formalizing the problem as a linear program with constraints (see here for the code used for NeurIPS 2017). Then, the Senior ACs (SACs, responsible for overseeing the work of the ACs) are asked to make adjustments to this initial assignment and fix any inappropriate matches.
Second, the reviewer assignment is performed. Here again, we ask reviewers to bid on papers, providing their preferences for at least 20 submissions. We also ask ACs to suggest reviewers they think would be appropriate for each submission. Then, again combining that information with reviewer subject areas and TPMS profiles, a linear program similar to the one used for AC assignments is run to produce an initial assignment to reviewers. Each paper is assigned exactly 3 reviewers, and reviewers are assigned at most 5 papers at this point.
Finally, ACs are given a chance to make adjustments to these reviewer assignments. The maximum quota for NeurIPS reviewers is 6 papers. By initially assigning at most 5 papers to reviewers, it made it easier for ACs to make these adjustments, since many reviewers could then still handle 1 additional paper. This year, we’ve also worked with CMT to facilitate the recruitment of external reviewers, i.e. reviewers who are not already in the reviewer pool but that an AC thinks would be particularly appropriate for a specific submission in their stack. CMT has implemented a new feature that allows ACs to send such reviewer invitations, which are similar to those that program chairs send, but sent in the context of a specific paper.
And with that, the work of reviewing begins! Reviewers have roughly 4 weeks (from June 18th to July 15th) to perform this crucial and time consuming task. As a reward for their hard work, using feedback from ACs, the 400 or so best reviewers will be awarded free NeurIPS registrations and the next 2000 or so reviewers will have registrations reserved for them.
Indeed, the time and effort that reviewers put into evaluating each submission is what makes NeurIPS possible!
Alina Beygelzimer, Emily Fox, Florence d’Alché-Buc, Hugo Larochelle
NeurIPS 2019 Program Chairs