返回文献探索AI 摘要
Paper · arXiv 2306.07042
Transformers learn through gradual rank increase
Enric Boix-Adsera, Etai Littwin, Emmanuel Abbe, Samy Bengio, Joshua Susskind
10 upvotesJune 12, 2023arXiv 预印本
The study examines incremental learning in transformers, proving that the difference in rank between trained and initial weights increases progressively.
incremental learningtransformersdiagonal weight matricessmall initialization
Abstract
We identify incremental learning dynamics in transformers, where the difference between trained and initial weights progressively increases in rank. We rigorously prove this occurs under the simplifying assumptions of diagonal weight matrices and small initialization. Our experiments support the theory and also show that phenomenon can occur in practice without the simplifying assumptions.