What we learned from NeurIPS 2019 data

Part I: Dissecting the NeurIPS community

There were 15,920 authors of submitted papers. (We made a careful attempt to merge profiles to keep a single entity per person in this analysis.) Three quarters were not on the invitation list for the program committee in any capacity — as reviewers, area chairs (ACs), or senior area chairs (SACs). Slightly over 70% of the remaining quarter accepted our invitation to serve on the program committee, which is a good sign. Moreover, the majority of those reviewing also submitted papers — also a good sign.

Part II: Speculative experiments on reducing or limiting the number of submissions

There have been many discussions about changing the NeurIPS reviewing model to better handle the growing number of submissions. Let’s have a little fun and use the NeurIPS 2019 data to estimate the consequences of some of the proposals we’ve heard.

Editorial screening

As an experiment, we wanted to measure the ability of ACs to predict, before seeing reviews, which of their assigned submissions were going to be rejected (for example, due to their insufficient novelty or poor scholarship). The question is whether NeurIPS should consider allowing ACs to reject submissions without review in order to reduce the reviewing load — such editorial screening is common practice at top journals.

Capping the number of submissions

Another often mentioned proposal is to cap the number of papers that anyone can submit. The AAAI conference even introduced a cap of 15 submissions per author for 2020 (see their Call for Papers).

Supply-and-Demand Reviewing

Another proposal (here and here) is to use a market system to control reviewing. Only submissions that gather sufficient interest from reviewers are reviewed.

Open Dissemination of Submissions under Review

A majority (54%) of all submissions were posted on arXiv; 21% of these submissions were seen by at least one reviewer. The acceptance rate in this latter category was 34%, significantly higher than the base rate of 21.6%. For comparison, the acceptance rate for submissions that were not posted was 17%.

Part III: Review quality

Reviewer Assignment

What’s a good proxy for review quality that we can objectively measure? One proxy suggested to us was whether an assigned reviewer is cited in the paper. So what is the fraction of NeurIPS 2019 submissions with at least one cited reviewer?

Distribution of Review Length

Given frequent complaints about short reviews at NeurIPS, we looked at the distribution of review lengths for NeurIPS 2019, ICLR 2019, and COLT 2019.

Rebuttals, Discussions, and Acceptance Statistics

As an author composing your rebuttal, you probably want to know the probability of your paper getting accepted given its initial scores. What are the odds of your rebuttal shifting the outcome?

Summary

Though the data still leaves a lot of questions unanswered, we personally notice the following takeaways:

  1. No free-loader problem: Relatively few papers are submitted where none of the authors invited to participate in the review process accepted the invitation
  2. Unclear how to rapidly filter papers prior to full review: Allowing for early desk rejects by ACs is unlikely to have a significant impact on reviewer load without producing inappropriate decisions. Likewise, the eagerness of reviewers to review a particular paper is not a strong signal, either.
  3. No clear evidence that review quality as measured by length is lower for NeurIPS: NeurIPS is surprisingly not much different from other conferences of smaller sizes when it comes to review length.
  4. Impact of engagement in rebuttal/discussion period: Overall engagement seemed to be higher than in 2018.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Neural Information Processing Systems Conference

Neural Information Processing Systems Conference

Tweets sent to this account are not actively monitored. To contact us please go to http://neurips.cc/Help/Contact