ATTRIB 2024 Workshop

What makes ML models tick? How do we attribute model behavior to the training data, algorithm, architecture, or scale used in training?

Recently-developed algorithmic innovations and large-scale datasets have given rise to machine learning models with impressive capabilities. However, there is much left to understand in how these different factors combine to give rise to observed behaviors. For example, we still do not fully understand how the composition of training datasets influence downstream model capabilities, how to attribute model capabilities to subcomponents inside the model, and which algorithmic choices really drive performance.

A common theme underlying all these challenges is model behavior attribution. That is, the need to tie model behavior back to factors in the machine learning pipeline—such as the choice of training dataset or particular training algorithm—that we can control or reason about. This workshop aims to bring together researchers and practitioners with the goal of advancing our understanding of model behavior attribution.

Call for Papers

Submissions open August 1st!

We are soliciting papers along two tracks:

Main track papers: 3-6 page submissions on attributing model behaviors (see below for example topics).
Idea track papers: 2-4 page submissions, on a specific topic or on the field of attribution as a whole. Vision papers, unifying ideas, documentation of failed experiments are all welcome. Papers in this track will be held to the same standard as the main track, but can be opinionated (so long as opinions are clearly demarcated from facts) and need not have experiments (although lack of experimentation should be justified).

Along these tracks, we welcome submissions pertaining to any aspect of model behavior attribution. For example:

Data: Models are trained on large-scale datasets collected from disparate (and often arbitrarily chosen) sources. How can we understand how the composition training data affects model behavior? This includes:
- Data attribution and selection: How can we (efficiently) attribute model outputs back to specific training examples? How can we select data to optimize downstream performance/capabilities?
- Data leakage/contamination: How can we monitor and fix data leakage at internet scale? How do data feedback loops (e.g., training on LLM-generated outputs) influence model biases?
Trained models: Large models remain black boxes—how do we attribute a model's behavior to its subcomponents? Directions include:
- Mechanistic interpretability: How do individual neurons combine to yield model predictions?
- Concept-based interpretability: Can we attribute predictions to human-identifiable concepts? Can we attribute these concepts or other biases to subnetworks inside a DNN?
Learning algorithms: Designing a ML model involves dozens of choices, ranging from choice of model architecture, optimizer, to learning algorithm. How do these choices influence model behavior? For example, exploring issues such as:
- Understanding algorithmic choices: How do algorithmic choices affect model capabilities? What parts of model behavior can we attribute to specific algorithmic choices?
- Scalings laws/emergence: What emergent capabilities (if any) can we actually attribute to scale alone?

Submission Instructions

Format submissions as follows:
- 3-6 pages (main track) or 2-4 pages (idea track)
- NeurIPS 2024 paper formatting (download from here)
- Appendix included in the same PDF as the main body
- No Appendix page limit
When ready, submit to OpenReview (note our workshop is non-archival; published papers are fine to submit provided they meet the formatting requirements above)
(Optional) Camera-ready: TBA

Important Dates

August 1: Submission portal opens

September 25 (AOE): Deadline for both idea and main track papers

October 4 (AOE): Final deadline for papers (if >= 1 author agrees to emergency review)

October 10: Decision notifications

December 14: Workshop!