Behind the Selection of the NeurIPS 2019 Workshops

NeurIPS 2019 workshop decisions went out just moments ago! This guest post is brought to you by this year’s Workshop Chairs, Jenn Wortman Vaughan, Marzyeh Ghassemi, Shakir Mohamed, and Bob Williamson. We want to give you some highlights from the innovative proposals we saw and describe the process that we used for selection. In addition to providing transparency and feedback to everyone who submitted proposals this year, we hope this post will be valuable to those who are submitting workshop proposals in the future.

Before launching into this, we want to thank all the folks who put in the effort to propose workshops. As Workshop Chairs, all we can do is guide the development of these proposals. The work is done by the organizers. So thank you!

What Changed This Year: New Guidelines, Expectations, and Selection Criteria

NeurIPS is growing fast. To accommodate this, the number of Workshop Chairs doubled this year, from two to four. The first thing that the four of us did—and perhaps the most important component of our role—was to rethink and substantially revise the list of criteria used to select workshops. The new criteria were designed to make the selection process transparent and to encourage workshop proposals that we thought would be most valuable to the community. We took into account of difficulties encountered in previous years and crafted the criteria in an attempt to mitigate some of the perceived problems of the past.

Our new set of criteria, which was included in the Call for Workshops, is as follows:

1. Degree to which the proposal is focused on an important and topical problem, and the degree to which it is expected that the community will find the workshop interesting, exciting, and valuable.
2. Intellectual excitement of the topic. Is it likely to break new ground, or merely reiterate tired, old debates?
3. Diversity and inclusion, in all forms. This includes diversity of viewpoint and thinking regarding the topics discussed at the workshop, plus diversity in terms of gender, race, affiliations, seniority, and so on. We encouraged organizers to include an explicit diversity commitment in their proposals.
4. Degree to which the proposed program offers opportunity for discussion.
5. Quality of proposed invited speakers.
6. Degree to which the organizers offered means to engage in the workshop for those unable to attend in person.
7. Organizational experience and ability of the team, though we encouraged involving junior researchers in workshop organization too.
8. Other dimensions in our Guidance and Expectations for Workshop Organizers not explicitly listed in these criteria.
9. Points of difference. What makes this workshop enticingly different to the hundreds of NeurIPS workshops held previously?

Beyond the selection criteria, we made a few other logistical changes designed to address specific problems that arose in previous years.

First, because of visa issues, we set a global notification deadline of October 1. We asked workshop organizers to commit to sending out accept/reject notifications for workshop contributions before this date to allow time for visa acquisition. (It also made it easy to tell which organizers did not read our guidelines when we saw proposals with notification deadlines in mid-November!) How will we enforce this? In recent years, NeurIPS has sold out quickly. Each workshop is allocated a number of reserved registrations for speakers. This year, these reserved registrations will be contingent on meeting the October 1 notification deadline.

Second, we set out clear policies on conflicts of interest, both around how workshops are selected and around the choice of invited speakers. For example, unlike in previous years, workshop organizers cannot give “invited” talks at the workshops they organize.

Third, we decided up front that we would not forcibly merge proposals or share proposal information if multiple proposals were submitted on very similar topics, as has sometimes been done in past years.

Finally, we set a policy of neither encouraging nor discouraging workshops on topics that have appeared before. Workshop proposals would be evaluated solely on their merits for this year’s conference.

The Decision Process

We created a Workshop Program Committee with 37 reviewers. We followed our own diversity guidelines, but included more senior researchers who have been involved with the NeurIPS community for a number of years, as a certain amount of “community wisdom” is needed to do the role well. Our final list was reviewed and approved by the NeurIPS General Chair and Diversity Chairs.

Each workshop proposal was assigned to two reviewers. In a small number of cases, one of us filled in when a review was late or missing. Reviewers were asked to provide a summary and overall rating for each workshop, a detailed list of pros and cons, and specific ratings for each of the criteria outlined above. We found that these detailed ratings were useful for getting reviewers to think carefully about the selection criteria, but the pros and cons lists were most useful for us in forming an understanding of each workshop’s merits.

After all reviews were submitted, each proposal was assigned to two of the four of us. (In a small number of cases, three of us had conflicts of interest, so only one of us was assigned.) We looked through our assigned proposals and their reviews to form an educated assessment of pros and cons of each.

Finally, the four of us held a series of meetings to walk through and discuss every submitted proposal and make decisions. In some cases, especially when there was substantial overlap in topic between proposals, we all went back and read them one more time during this last phase before reaching our final decisions.

Reflections on This Year’s Proposals

After weeding out erroneous submissions (including a surprising number of research papers inadvertently submitted to the workshops track on CMT), we received a total of 111 true workshop submissions. This number is down from the roughly 140 submissions received in 2018, closer to the roughly 110 submissions in 2017. It’s not clear whether this dip in submissions is due to the change in guidelines and expectations or to other factors.

Of these 111 submissions, we selected 51 workshops to be held in Vancouver this December. Overall, we are thrilled with the selection! We want to commend those workshop organizers who were thoughtful, deliberate, and innovative in their proposals. A few highlights:

• Diversity as a first-order priority. We saw organizers taking diversity seriously this year, and we were seriously impressed by some of the innovative ideas that came out of this. Highlights include partnerships with existing affinity groups to provide travel scholarships, sessions showcasing the contributions of junior researchers, and awesome invited speakers from government, industry, or other disciplines who have likely never attended NeurIPS before. So much good stuff! We are super excited about this. We saw our reviewers taking diversity seriously too, and calling out organizers who were only giving diversity lip service. (Same goes for access for those who cannot attend in person.) We are thrilled that our guidelines had this effect, and hope this will help the NeurIPS community move toward new norms.
• Innovation in programming. We saw some admirable and innovative ideas for allowing the community to contribute that went far beyond the typical contributed talks or poster sessions. Examples include breakout sessions dedicated to making progress on particular questions, prep work done by the organizers to focus discussions, competitions, allowing attendees to submit “challenge questions” to speakers in advance, and open problem sessions.
• Careful thought and reflection. We saw some proposals in which it was clear that the organizers had put detailed, careful thought into the problems they were trying to solve and how holding the workshop would help them address those problems. We were especially impressed by workshops that addressed big, impactful problems for which the organizers had made a clear case that progress was feasible in the near term.
• Unexpected topics. To our delight, there were a few real surprises in terms of topics. One of our favorite proposals brought together two communities that typically don’t interact much to address a common challenge, and was deemed by one reviewer to be “wild and crazy” in a good way.

It is also worth noting some common pitfalls that we saw in proposals:

• Leaning too heavily on past success. Proposals for workshops that are part of series sometimes leaned too heavily on the declared popularity of previous workshops. In some cases, this led to proposals that were less creative and innovative than what we had hoped to see.
• Unconfirmed or irrelevant speakers. The vast majority of proposals included lists of confirmed invited speakers. This made it hard to champion any workshop that didn’t have at least a few speakers confirmed, especially when many unconfirmed big-name speakers were listed (it’s unlikely all would say yes), or when the diversity statement centered on the assumed presence of unconfirmed speakers. There were also several proposals featuring long lists of “celebrity” speakers without clear relevance to the topic of the workshop.
• Insufficient time for discussion. Too many invited speakers—some proposals listed a dozen or more—does not make for a great audience experience, and a workshop with nothing but long-form talks is unlikely to lead to new breakthroughs. We encourage organizers to allocate a larger amount of time to contributed content and open discussion.
• Going too big. We saw only a few proposals that we felt were too narrow, but many we found too broad. There seems to be a tendency to overreach for the sake of going big, while we’d prefer to see more focused workshops.
• Too many organizers. Several proposals had remarkably large organizing committees. It’s not clear why more than five or six organizers would be necessary for a workshop, and it raises concerns about name dropping or organizers added just for an appearance of diversity.
• Diversity lip service. While we were pleased overall by the effort that organizers put into diversity, lack of diversity in a proposal could be fatal. We were particularly wary of proposals that claimed to be big on diversity while having a full lineup of North American white male speakers or a list of organizers who all recently graduated from the same institution.

In terms of workshop topics, we saw a few clear emerging trends. Some common themes we saw were ML + health, ML + systems, fairness in ML, and privacy in ML. Reinforcement learning also continues to be huge this year. For these popular categories, we did not artificially limit ourselves to selecting just one, but did attempt to ensure that each workshop selected brought something unique to the program.

NeurIPS 2019 Accepted Workshops

On to the best part: the preliminary list of accepted workshops for 2019!

• Visually Grounded Interaction and Language
• Bayesian Deep Learning
• Machine Learning for the Developing World (ML4D): Challenges and Risks
• Tackling Climate Change with ML
• Real Neurons & Hidden Units: Future Directions at the Intersection of Neuroscience and AI
• Learning with Temporal Point Processes
• Beyond First Order Methods in Machine Learning Systems
• Medical Imaging Meets NeurIPS
• KR2ML: Knowledge Representation and Reasoning Meets Machine Learning
• CiML 2019: Machine Learning Competitions for All
• Meta-Learning
• Learning with Rich Experience: Integration of Learning Paradigms
• Machine Learning with Guarantees
• Machine Learning for Autonomous Driving
• Workshop on Human-Centric Machine Learning
• Graph Representation Learning
• Science Meets Engineering of Deep Learning
• Perception as Generative Reasoning: Structure, Causality, Probability
• Machine Learning for Health (ML4H): What Makes Machine Learning in Medicine Different?
• Program Transformations for ML
• Robust AI in Financial Services: Data, Fairness, Explainability, Trustworthiness, and Privacy
• Privacy in Machine Learning (PiML)
• Robot Learning: Control and Interaction in the Real World
• Machine Learning and the Physical Sciences
• Context and Compositionality in Biological and Artificial Neural Systems
• Deep Reinforcement Learning
• ML For Systems
• NeurIPS Workshop on Machine Learning for Creativity and Design 3.0
• “Do the Right Thing:” Machine Learning and Causal Inference for Improved Decision Making
• Information Theory and Machine Learning
• Bridging Game Theory and Deep Learning
• Optimal Transport for Machine Learning
• Biological and Artificial Reinforcement Learning
• Joint Workshop on AI for Social Good
• MLSys: Workshop on Systems for ML
• Learning Transferable Skills
• EMC2: Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (5th edition)
• Document Intelligence
• Emergent Communication: Towards Natural Language
• Workshop on Federated Learning for Data Privacy and Confidentiality
• Fair ML in Healthcare
• AI for Humanitarian Assistance and Disaster Response
• The Third Conversational AI Workshop: Today’s Practice and Tomorrow’s Potential
• The Optimization Foundations of Reinforcement Learning
• Safety and Robustness in Decision-Making
• Shared Visual Representations in Human and Machine Intelligence
• Learning Meaningful Representations of Life
• Minding the Gap: Between Fairness and Ethics
• Retrospectives: A Venue for Self-Reflection in ML Research
• Sets and Partitions
• Solving Inverse Problems with Deep Networks: New Architectures, Theoretical Foundations, and Applications

Looking forward to seeing you in Vancouver!

Jenn Wortman Vaughan, Marzyeh Ghassemi, Shakir Mohamed, and Bob Williamson, NeurIPS 2019 Workshop Chairs