The NeurIPS submission deadline is almost here (abstract by May 16th at 4pm ET / 1pm PT, full paper by May 23rd at 4pm ET / 1pm PT)!
We’ve finished recruiting all the reviewers we’ll need to evaluate the submissions. 4,534 academic and industrial researchers will be acting as reviewers. The CMT website is now accepting submissions (as of this post, 1355 abstracts have been submitted). We have also matched all area chairs (ACs) with their senior area chair (SAC) who will be overseeing their work during the review process. Finally, Zhenyu (Sherry) Xue has joined the organizing committee as our second workflow manager, alongside Mathieu Germain (see our previous post on workflow managers).
Today we wanted to touch on how reviews will work for NeurIPS 2019.
Reviewer and AC/SAC guidelines
We’ve been working on refining the details of the review process. We’ve just published guidelines for reviewers and ACs/SACs. These guidelines describe the review process, list best practices, and our expectations for everyone participating in the process. It’s a great resource not only for members of the program committee, but also potential authors less familiar with the nature of these roles.
Notably, this year we’re slightly expanding the review form. Last year, NeurIPS reviewers were asked to simply provide an overall score (on a scale from 1 to 10), a confidence score (from 1 to 5) and free-form comments detailing and justifying their evaluation (usually along the lines of quality, clarity, originality, and significance of the submission).
This year, we added two questions to the review form to encourage reviewers to point out the main arguments in favor of a submission.
Here they are, in full:
- Contributions: Please list three things this paper contributes (e.g., theoretical, methodological, algorithmic, empirical contributions; bridging fields; or providing an important critical analysis). For each contribution, briefly state the level of significance (i.e., how much impact will this work have on researchers and practitioners in the future?). If you cannot think of three things, please explain why. Not all good papers will have three contributions.
We think that by adding this question, reviewers will be more likely to actively reflect on the strengths of a submission, and not merely focus on finding issues or weaknesses. We hope it will also help to better guide the discussion period between reviewers, as any disagreement between reviewers on the contributions of a submission would certainly be important to address.
- Improvements: What would the authors have to do for you to increase your score?
This question was added to help the authors better understand what rebuttal this reviewer could find convincing. We also hope that reviewers will write more constructive reviews if they are explicitly prompted to be specific about what the authors could do to improve their submission.
The full review form will also include other questions that are not meant to provide evaluation information about each submission, but instead to gather statistics that could inform future NeurIPS policies (usefulness of the reproducibility checklist or usefulness of the optionally provided code, if applicable). For more information on the reviewing form, please visit section “Review content” of the reviewer guidelines.
We’ll have more posts later this summer. Best of luck polishing your NeurIPS submissions!
Alina Beygelzimer, Emily Fox, Florence d’Alché-Buc, Hugo Larochelle
NeurIPS 2019 Program Chairs