Frac-Connections: Fractional Extension of Hyper-Connections

Defa Zhu; Hongzhi Huang; Jundong Zhou; Zihao Huang; Yutao Zeng; Banggu Wu; Qiyang Min; Xun Zhou

返回文献探索

Paper · arXiv 2503.14125

Frac-Connections: Fractional Extension of Hyper-Connections

Defa Zhu, Hongzhi Huang, Jundong Zhou, Zihao Huang, Yutao Zeng, Banggu Wu, Qiyang Min, Xun Zhou

22 upvotesMarch 18, 2025arXiv 预印本

AI 摘要

Frac-Connections, a technique that divides hidden states, outperform traditional residual connections in deep learning by balancing memory consumption and performance, validated through experiments on large language models.

residual connectionsgradient vanishingHyper-Connectionsrepresentation collapseFrac-Connectionshidden statesmemory consumptionlarge-scale experimentslanguage tasks7B MoE model3T tokens

Abstract

Residual connections are central to modern deep learning architectures, enabling the training of very deep networks by mitigating gradient vanishing. Hyper-Connections recently generalized residual connections by introducing multiple connection strengths at different depths, thereby addressing the seesaw effect between gradient vanishing and representation collapse. However, Hyper-Connections increase memory access costs by expanding the width of hidden states. In this paper, we propose Frac-Connections, a novel approach that divides hidden states into multiple parts rather than expanding their width. Frac-Connections retain partial benefits of Hyper-Connections while reducing memory consumption. To validate their effectiveness, we conduct large-scale experiments on language tasks, with the largest being a 7B MoE model trained on up to 3T tokens, demonstrating that Frac-Connections significantly outperform residual connections.