CausalLM is not optimal for in-context learning
Nan Ding, Tomer Levinboim, Jialin Wu +2 authors
Theoretical analysis shows that prefix language models outperform causal language models in in-context learning by converging to optimal solutions, while causal models exhibit dynamics similar to online gradient descent.