Mechanism of In-Context Learning in Transformers
Comparing three hypothesized mechanisms for ICL without parameter updates
0.936
Best ICL Accuracy (Task Retrieval, k=8)
3
Mechanisms Compared
4
Layers Analyzed
50
Tasks Evaluated
ICL Accuracy vs Demonstrations
Layer-wise Mechanism Contributions
Task Retrieval Probability
Mechanism Comparison at k=8
Key Findings
Task retrieval achieves the highest individual accuracy, matching oracle at k=8 demonstrations.
Implicit gradient descent shows steady improvement but requires many more examples.
Induction heads provide a robust but lower-accuracy baseline via pattern matching.
Layer-wise analysis reveals depth-dependent specialization: early layers for task ID, later layers for pattern matching.
ICL is best understood as a multi-mechanism process, not a single algorithm.