MH-MoE:Multi-Head Mixture-of-Experts

Shaohan Huang; Xun Wu; Shuming Ma; Furu Wei

返回文献探索

Paper · arXiv 2411.16205

MH-MoE:Multi-Head Mixture-of-Experts

Shaohan Huang, Xun Wu, Shuming Ma, Furu Wei

26 upvotesNovember 25, 2024arXiv 预印本

AI 摘要

A novel implementation of Multi-Head Mixture-of-Experts maintains efficiency and surpasses traditional models in language performance, even with 1-bit Large Language Models.

Multi-Head Mixture-of-ExpertsMH-MoErepresentation spacesexpertsFLOPsparameter paritysparse Mixture of Expertsvanilla MoEfine-grained MoEbitwise Large Language ModelsBitNet

Abstract

Multi-Head Mixture-of-Experts (MH-MoE) demonstrates superior performance by using the multi-head mechanism to collectively attend to information from various representation spaces within different experts. In this paper, we present a novel implementation of MH-MoE that maintains both FLOPs and parameter parity with sparse Mixture of Experts models. Experimental results on language models show that the new implementation yields quality improvements over both vanilla MoE and fine-grained MoE models. Additionally, our experiments demonstrate that MH-MoE is compatible with 1-bit Large Language Models (LLMs) such as BitNet.