Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training
Wenyu Du, Tongxu Luo, Zihan Qiu +5 authors
A depthwise stacking operator called $G_{\text{stack}}$ accelerates LLM pre-training, improving performance with less computational expense and providing guidelines for effective use.