Organizing the NeurIPS 2020 Competitions Track
Competitions are returning to NeurIPS 2020! This year’s co-organizers are Dr. Hugo Jair Escalante, Principal Researcher at the National Institute of Astrophysics, Optics and Electronics (INAOE), Mexico, and Dr. Katja Hofmann, Principal Researcher at Microsoft Research in Cambridge, UK, and head of the Game Intelligence group.
Hugo is a seasoned expert in this role. He has been involved with competitions for more than a decade and co-organized the previous year’s NeurIPS Competition and Demonstration tracks.
In his experience organizing competitions, Hugo was positively surprised by the response from the NeurIPS community to the 2019 edition of the track: 16 challenges were featured all of them dealing with open problems in machine learning; participation was massive, with more than 15,000 entries submitted by more than 4,000 teams! During the competition session amazing progress made by participants, many proposed solutions pushed the state of the art and significantly outperformed baseline models. Interestingly, a particularly popular focus used to address the problems posed by competitions was reinforcement learning (together with deep learning, of course), where each competition had its own twist. Likewise, “live competitions” were very attractive for participants and even attendees who filled the room during the sessions.
Katja brings her experience co-organizing three competitions centered around the Project Malmo experimentation platform based on the popular video game Minecraft. The most recent of these was the MineRL competition, which she co-organized with a team from CMU and other partners. It focused on sample efficient reinforcement learning approaches with human priors.
Katja found the collaboration to organize the MineRL competition hugely inspiring. Led by William Guss and others at Carnegie Mellon University, the organizers rethought the experiment setup and evaluation for reinforcement learning competitions. Their goal was to find a setup that would impose strict limits on compute, data and training time.This would encourage participants to develop more sample efficient reinforcement learning algorithms. Given this ambitious goal, it is no surprise that the competition demanded an organizing team to match: more than 20 co-organizers and advisors worked hard to make the competition a success.
The most rewarding aspect of the competition track for Katja was meeting the organizing team and many participants in person at NeurIPS. The enthusiasm and passion of everyone involved blew her away. The results achieved by participants were extremely strong, in fact much more progress was made towards solving the challenge at hand than many of the organizers had anticipated. To Katja, one of the key benefits of the competition setup that enabled this was its low barrier to entry. People with a wide range of expertise and experience participated, including some who would not traditionally participate or join NeurIPS. This meant that between themselves, participants tested a wide range of ideas, enabling this impressive progress.
Both Hugo and Katja’s experience has armed them with best practices. In Katja’s experience having a sufficient organizing team is key. While many competitions are organized by much smaller teams, it is worth investing in key areas that are particularly likely to enable broad participation, such as clearly documented competition rules, easy to use starter code, a forum or channel for participant questions and discussion. For Hugo, diversity and inclusion from both sides, the organizing team and participants, should be further encouraged, challenges are far reaching efforts that make things fair for everyone, this is definitely a promising way to tackle disadvantages for underrepresented groups to learn, contribute and eventually solve problems with a huge impact into machine learning. Likewise, challenges with societal and humanitarian impact have priority for the track. This reflects the commitment that NeurIPS have in helping to build a better world throughout machine learning.
The 2020 NeurIPS competition track will be shaped by its organizers’ experience in several ways. There will be a joint poster session at the end of the first competition day. This will provide additional opportunities for all attending competition organizers and participants to exchange ideas and learn from each others’ insights. We are also considering potential ways to publish competition proceedings with the most relevant findings from the competition track. We cannot wait to see the exciting competition proposals that will be submitted this year, and the progress they will drive in machine learning research and practice.
