MambaByte: Token-free Selective State Space Model

Junxiong Wang; Tushaar Gangavarapu; Jing Nathan Yan; Alexander M Rush

返回文献探索

Paper · arXiv 2401.13660

MambaByte: Token-free Selective State Space Model

Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, Alexander M Rush

59 upvotesJanuary 24, 2024arXiv 预印本

AI 摘要

MambaByte, a token-free byte-level autoregressive model, demonstrates computational efficiency and competitive performance compared to subword token-based models, with the added benefit of fast inference due to linear length scaling.

MambaBytetoken-free language modelssubword tokenizationautoregressive Transformersbyte sequencesstate space modelMamba state space model

Abstract

Token-free language models learn directly from raw bytes and remove the bias of subword tokenization. Operating on bytes, however, results in significantly longer sequences, and standard autoregressive Transformers scale poorly in such settings. We experiment with MambaByte, a token-free adaptation of the Mamba state space model, trained autoregressively on byte sequences. Our experiments indicate the computational efficiency of MambaByte compared to other byte-level models. We also find MambaByte to be competitive with and even outperform state-of-the-art subword Transformers. Furthermore, owing to linear scaling in length, MambaByte benefits from fast inference compared to Transformers. Our findings establish the viability of MambaByte in enabling token-free language modeling.