mLongT5: A Multilingual and Efficient Text-To-Text Transformer for
  Longer Sequences

David Uthus; Santiago Ontañón; Joshua Ainslie; Mandy Guo

返回文献探索

Paper · arXiv 2305.11129

mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences

David Uthus, Santiago Ontañón, Joshua Ainslie, Mandy Guo

2 upvotesMay 18, 2023arXiv 预印本

AI 摘要

A multilingual text-to-text transformer called mLongT5, combining LongT5 with datasets from mT5 and UL2, outperforms existing models like mBART and M-BERT in summarization and question-answering tasks.

multilingualtext-to-text transformerLongT5mT5UL2summarizationquestion-answering

Abstract

We present our work on developing a multilingual, efficient text-to-text transformer that is suitable for handling long inputs. This model, called mLongT5, builds upon the architecture of LongT5, while leveraging the multilingual datasets used for pretraining mT5 and the pretraining tasks of UL2. We evaluate this model on a variety of multilingual summarization and question-answering tasks, and the results show stronger performance for mLongT5 when compared to existing multilingual models such as mBART or M-BERT.