Behind the Selection of the NeurIPS 2019 Workshops

NeurIPS 2019 workshop decisions went out just moments ago! This guest post is brought to you by this year’s Workshop Chairs, Jenn Wortman Vaughan, Marzyeh Ghassemi, Shakir Mohamed, and Bob Williamson. We want to give you some highlights from the innovative proposals we saw and describe the process that we used for selection. In addition to providing transparency and feedback to everyone who submitted proposals this year, we hope this post will be valuable to those who are submitting workshop proposals in the future.

Before launching into this, we want to thank all the folks who put in the effort to propose workshops. As Workshop Chairs, all we can do is guide the development of these proposals. The work is done by the organizers. So thank you!

What Changed This Year: New Guidelines, Expectations, and Selection Criteria

NeurIPS is growing fast. To accommodate this, the number of Workshop Chairs doubled this year, from two to four. The first thing that the four of us did—and perhaps the most important component of our role—was to rethink and substantially revise the list of criteria used to select workshops. The new criteria were designed to make the selection process transparent and to encourage workshop proposals that we thought would be most valuable to the community. We took into account of difficulties encountered in previous years and crafted the criteria in an attempt to mitigate some of the perceived problems of the past.

Our new set of criteria, which was included in the Call for Workshops, is as follows:

  1. Degree to which the proposal is focused on an important and topical problem, and the degree to which it is expected that the community will find the workshop interesting, exciting, and valuable.

Beyond the selection criteria, we made a few other logistical changes designed to address specific problems that arose in previous years.

First, because of visa issues, we set a global notification deadline of October 1. We asked workshop organizers to commit to sending out accept/reject notifications for workshop contributions before this date to allow time for visa acquisition. (It also made it easy to tell which organizers did not read our guidelines when we saw proposals with notification deadlines in mid-November!) How will we enforce this? In recent years, NeurIPS has sold out quickly. Each workshop is allocated a number of reserved registrations for speakers. This year, these reserved registrations will be contingent on meeting the October 1 notification deadline.

Second, we set out clear policies on conflicts of interest, both around how workshops are selected and around the choice of invited speakers. For example, unlike in previous years, workshop organizers cannot give “invited” talks at the workshops they organize.

Third, we decided up front that we would not forcibly merge proposals or share proposal information if multiple proposals were submitted on very similar topics, as has sometimes been done in past years.

Finally, we set a policy of neither encouraging nor discouraging workshops on topics that have appeared before. Workshop proposals would be evaluated solely on their merits for this year’s conference.

The Decision Process

We created a Workshop Program Committee with 37 reviewers. We followed our own diversity guidelines, but included more senior researchers who have been involved with the NeurIPS community for a number of years, as a certain amount of “community wisdom” is needed to do the role well. Our final list was reviewed and approved by the NeurIPS General Chair and Diversity Chairs.

Each workshop proposal was assigned to two reviewers. In a small number of cases, one of us filled in when a review was late or missing. Reviewers were asked to provide a summary and overall rating for each workshop, a detailed list of pros and cons, and specific ratings for each of the criteria outlined above. We found that these detailed ratings were useful for getting reviewers to think carefully about the selection criteria, but the pros and cons lists were most useful for us in forming an understanding of each workshop’s merits.

After all reviews were submitted, each proposal was assigned to two of the four of us. (In a small number of cases, three of us had conflicts of interest, so only one of us was assigned.) We looked through our assigned proposals and their reviews to form an educated assessment of pros and cons of each.

Finally, the four of us held a series of meetings to walk through and discuss every submitted proposal and make decisions. In some cases, especially when there was substantial overlap in topic between proposals, we all went back and read them one more time during this last phase before reaching our final decisions.

Reflections on This Year’s Proposals

After weeding out erroneous submissions (including a surprising number of research papers inadvertently submitted to the workshops track on CMT), we received a total of 111 true workshop submissions. This number is down from the roughly 140 submissions received in 2018, closer to the roughly 110 submissions in 2017. It’s not clear whether this dip in submissions is due to the change in guidelines and expectations or to other factors.

Of these 111 submissions, we selected 51 workshops to be held in Vancouver this December. Overall, we are thrilled with the selection! We want to commend those workshop organizers who were thoughtful, deliberate, and innovative in their proposals. A few highlights:

  • Diversity as a first-order priority. We saw organizers taking diversity seriously this year, and we were seriously impressed by some of the innovative ideas that came out of this. Highlights include partnerships with existing affinity groups to provide travel scholarships, sessions showcasing the contributions of junior researchers, and awesome invited speakers from government, industry, or other disciplines who have likely never attended NeurIPS before. So much good stuff! We are super excited about this. We saw our reviewers taking diversity seriously too, and calling out organizers who were only giving diversity lip service. (Same goes for access for those who cannot attend in person.) We are thrilled that our guidelines had this effect, and hope this will help the NeurIPS community move toward new norms.

It is also worth noting some common pitfalls that we saw in proposals:

  • Leaning too heavily on past success. Proposals for workshops that are part of series sometimes leaned too heavily on the declared popularity of previous workshops. In some cases, this led to proposals that were less creative and innovative than what we had hoped to see.

In terms of workshop topics, we saw a few clear emerging trends. Some common themes we saw were ML + health, ML + systems, fairness in ML, and privacy in ML. Reinforcement learning also continues to be huge this year. For these popular categories, we did not artificially limit ourselves to selecting just one, but did attempt to ensure that each workshop selected brought something unique to the program.

NeurIPS 2019 Accepted Workshops

On to the best part: the preliminary list of accepted workshops for 2019!

  • Visually Grounded Interaction and Language

Looking forward to seeing you in Vancouver!

Jenn Wortman Vaughan, Marzyeh Ghassemi, Shakir Mohamed, and Bob Williamson, NeurIPS 2019 Workshop Chairs

Tweets sent to this account are not actively monitored. To contact us please go to http://neurips.cc/Help/Contact