The aim of the workshop is to discuss and propose standards for evaluating ML research, in order to better identify promising new directions and to accelerate real progress in the field of ML research. The problem requires understanding the kinds of practices that add or detract from the generalizability or reliability of results reported, and incentives for researchers to follow best practices. We may draw inspiration from adjacent scientific fields, from statistics, or history of science. Acknowledging that there is no consensus on best practices for ML, the workshop will have a focus on panel discussions and a few invited talks representing a variety of perspectives. The call to papers will welcome opinion papers as well as more technical papers on evaluation of ML methods. We plan to summarize the findings and topics that emerged during our workshop in a short report.

Call for papers

We invite two types of papers – opinion papers (up to 4 pages) stating positions on the topics related to those listed above, and methodology papers (up to 8 pages excluding references) about evaluation in ML. These topics may include:

• Establishing benchmarking standards for ML research
• Reliable tools/protocols for benchmarking and evaluation
• Understanding and defining reproducibility for machine learning
• Meta analyses thoroughly evaluating existing claims across papers
• Incentives for doing better evaluation and reporting results

Submission Site: https://cmt3.research.microsoft.com/SMILES2022

Speakers

Thomas Wolf Hugginface Inc.	Frank Schneider University of Tübingen	Rotem Dror University of Pennsylvania	James Evans University of Chicago
Melanie Mitchell Sante Fe Institute	Katherine Heller Google Brain	Corinna Cortes Google Research NYC

Panels

Reproducibility and Rigor in ML

Rotem Dror University of Pennsylvania	Sara Hooker Google Brain	Koustuv Sinha Mila, McGill University
Frank Schneider University of Tübingen	Gaël Varoquaux INRIA

Slow vs Fast Science

Chelsea Finn Stanford University	Michela Paganini DeepMind	James Evans University of Chicago
Russel Poldrack Stanford University	Oriol Vinyals DeepMind

Incentives for Better Evaluation

Corinna Cortes Google Research NYC	Yoshua Bengio Mila, Université de Montréal	John Langford Microsoft Research
Kyunghyun Cho New York University

Organizers

Stephanie Chan DeepMind	Rishabh Agarwal Google Brain	Xavier Bouthillier Mila, Université de Montréal
Caglar Gulcehre DeepMind	Jesse Dodge Allen Institute for AI

For any queries, please reach out to the organizers at ml-eval-iclr2022@googlegroups.com .