Behind the Selection of the NeurIPS 2019 Workshops

What Changed This Year: New Guidelines, Expectations, and Selection Criteria

NeurIPS is growing fast. To accommodate this, the number of Workshop Chairs doubled this year, from two to four. The first thing that the four of us did—and perhaps the most important component of our role—was to rethink and substantially revise the list of criteria used to select workshops. The new criteria were designed to make the selection process transparent and to encourage workshop proposals that we thought would be most valuable to the community. We took into account of difficulties encountered in previous years and crafted the criteria in an attempt to mitigate some of the perceived problems of the past.

  1. Degree to which the proposal is focused on an important and topical problem, and the degree to which it is expected that the community will find the workshop interesting, exciting, and valuable.
  2. Intellectual excitement of the topic. Is it likely to break new ground, or merely reiterate tired, old debates?
  3. Diversity and inclusion, in all forms. This includes diversity of viewpoint and thinking regarding the topics discussed at the workshop, plus diversity in terms of gender, race, affiliations, seniority, and so on. We encouraged organizers to include an explicit diversity commitment in their proposals.
  4. Degree to which the proposed program offers opportunity for discussion.
  5. Quality of proposed invited speakers.
  6. Degree to which the organizers offered means to engage in the workshop for those unable to attend in person.
  7. Organizational experience and ability of the team, though we encouraged involving junior researchers in workshop organization too.
  8. Other dimensions in our Guidance and Expectations for Workshop Organizers not explicitly listed in these criteria.
  9. Points of difference. What makes this workshop enticingly different to the hundreds of NeurIPS workshops held previously?

The Decision Process

We created a Workshop Program Committee with 37 reviewers. We followed our own diversity guidelines, but included more senior researchers who have been involved with the NeurIPS community for a number of years, as a certain amount of “community wisdom” is needed to do the role well. Our final list was reviewed and approved by the NeurIPS General Chair and Diversity Chairs.

Reflections on This Year’s Proposals

After weeding out erroneous submissions (including a surprising number of research papers inadvertently submitted to the workshops track on CMT), we received a total of 111 true workshop submissions. This number is down from the roughly 140 submissions received in 2018, closer to the roughly 110 submissions in 2017. It’s not clear whether this dip in submissions is due to the change in guidelines and expectations or to other factors.

  • Diversity as a first-order priority. We saw organizers taking diversity seriously this year, and we were seriously impressed by some of the innovative ideas that came out of this. Highlights include partnerships with existing affinity groups to provide travel scholarships, sessions showcasing the contributions of junior researchers, and awesome invited speakers from government, industry, or other disciplines who have likely never attended NeurIPS before. So much good stuff! We are super excited about this. We saw our reviewers taking diversity seriously too, and calling out organizers who were only giving diversity lip service. (Same goes for access for those who cannot attend in person.) We are thrilled that our guidelines had this effect, and hope this will help the NeurIPS community move toward new norms.
  • Innovation in programming. We saw some admirable and innovative ideas for allowing the community to contribute that went far beyond the typical contributed talks or poster sessions. Examples include breakout sessions dedicated to making progress on particular questions, prep work done by the organizers to focus discussions, competitions, allowing attendees to submit “challenge questions” to speakers in advance, and open problem sessions.
  • Careful thought and reflection. We saw some proposals in which it was clear that the organizers had put detailed, careful thought into the problems they were trying to solve and how holding the workshop would help them address those problems. We were especially impressed by workshops that addressed big, impactful problems for which the organizers had made a clear case that progress was feasible in the near term.
  • Unexpected topics. To our delight, there were a few real surprises in terms of topics. One of our favorite proposals brought together two communities that typically don’t interact much to address a common challenge, and was deemed by one reviewer to be “wild and crazy” in a good way.
  • Leaning too heavily on past success. Proposals for workshops that are part of series sometimes leaned too heavily on the declared popularity of previous workshops. In some cases, this led to proposals that were less creative and innovative than what we had hoped to see.
  • Unconfirmed or irrelevant speakers. The vast majority of proposals included lists of confirmed invited speakers. This made it hard to champion any workshop that didn’t have at least a few speakers confirmed, especially when many unconfirmed big-name speakers were listed (it’s unlikely all would say yes), or when the diversity statement centered on the assumed presence of unconfirmed speakers. There were also several proposals featuring long lists of “celebrity” speakers without clear relevance to the topic of the workshop.
  • Insufficient time for discussion. Too many invited speakers—some proposals listed a dozen or more—does not make for a great audience experience, and a workshop with nothing but long-form talks is unlikely to lead to new breakthroughs. We encourage organizers to allocate a larger amount of time to contributed content and open discussion.
  • Going too big. We saw only a few proposals that we felt were too narrow, but many we found too broad. There seems to be a tendency to overreach for the sake of going big, while we’d prefer to see more focused workshops.
  • Too many organizers. Several proposals had remarkably large organizing committees. It’s not clear why more than five or six organizers would be necessary for a workshop, and it raises concerns about name dropping or organizers added just for an appearance of diversity.
  • Diversity lip service. While we were pleased overall by the effort that organizers put into diversity, lack of diversity in a proposal could be fatal. We were particularly wary of proposals that claimed to be big on diversity while having a full lineup of North American white male speakers or a list of organizers who all recently graduated from the same institution.

NeurIPS 2019 Accepted Workshops

On to the best part: the preliminary list of accepted workshops for 2019!

  • Visually Grounded Interaction and Language
  • Bayesian Deep Learning
  • Machine Learning for the Developing World (ML4D): Challenges and Risks
  • Tackling Climate Change with ML
  • Real Neurons & Hidden Units: Future Directions at the Intersection of Neuroscience and AI
  • Learning with Temporal Point Processes
  • Beyond First Order Methods in Machine Learning Systems
  • Medical Imaging Meets NeurIPS
  • KR2ML: Knowledge Representation and Reasoning Meets Machine Learning
  • CiML 2019: Machine Learning Competitions for All
  • Meta-Learning
  • Learning with Rich Experience: Integration of Learning Paradigms
  • Machine Learning with Guarantees
  • Machine Learning for Autonomous Driving
  • Workshop on Human-Centric Machine Learning
  • Graph Representation Learning
  • Science Meets Engineering of Deep Learning
  • Perception as Generative Reasoning: Structure, Causality, Probability
  • Machine Learning for Health (ML4H): What Makes Machine Learning in Medicine Different?
  • Program Transformations for ML
  • Robust AI in Financial Services: Data, Fairness, Explainability, Trustworthiness, and Privacy
  • Privacy in Machine Learning (PiML)
  • Robot Learning: Control and Interaction in the Real World
  • Machine Learning and the Physical Sciences
  • Context and Compositionality in Biological and Artificial Neural Systems
  • Deep Reinforcement Learning
  • ML For Systems
  • NeurIPS Workshop on Machine Learning for Creativity and Design 3.0
  • “Do the Right Thing:” Machine Learning and Causal Inference for Improved Decision Making
  • Information Theory and Machine Learning
  • Bridging Game Theory and Deep Learning
  • Optimal Transport for Machine Learning
  • Biological and Artificial Reinforcement Learning
  • Joint Workshop on AI for Social Good
  • MLSys: Workshop on Systems for ML
  • Learning Transferable Skills
  • EMC2: Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (5th edition)
  • Document Intelligence
  • Emergent Communication: Towards Natural Language
  • Workshop on Federated Learning for Data Privacy and Confidentiality
  • Fair ML in Healthcare
  • AI for Humanitarian Assistance and Disaster Response
  • The Third Conversational AI Workshop: Today’s Practice and Tomorrow’s Potential
  • The Optimization Foundations of Reinforcement Learning
  • Safety and Robustness in Decision-Making
  • Shared Visual Representations in Human and Machine Intelligence
  • Learning Meaningful Representations of Life
  • Minding the Gap: Between Fairness and Ethics
  • Retrospectives: A Venue for Self-Reflection in ML Research
  • Sets and Partitions
  • Solving Inverse Problems with Deep Networks: New Architectures, Theoretical Foundations, and Applications



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store