Otter: A Multi-Modal Model with In-Context Instruction Tuning

Bo Li; Yuanhan Zhang; Liangyu Chen; Jinghao Wang; Jingkang Yang; Ziwei Liu

返回文献探索

Paper · arXiv 2305.03726

Otter: A Multi-Modal Model with In-Context Instruction Tuning

Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, Ziwei Liu

7 upvotesMay 5, 2023arXiv 预印本

AI 摘要

Multimodal instruction tuning improves the performance of models like Otter, based on OpenFlamingo, with an optimized dataset and reduced training resource requirements.

large language modelsfew/zero-shot learnerstext dataGPT-3InstrctGPTChatGPTinstruction tuningmulti-modal modelsFlamingo modelupstream interleaved format pretrainingMultI-Modal In-Context Instruction TuningOtterOpenFlamingoin-context learningHuggingface Transformers

Abstract

Large language models (LLMs) have demonstrated significant universal capabilities as few/zero-shot learners in various tasks due to their pre-training on vast amounts of text data, as exemplified by GPT-3, which boosted to InstrctGPT and ChatGPT, effectively following natural language instructions to accomplish real-world tasks. In this paper, we propose to introduce instruction tuning into multi-modal models, motivated by the Flamingo model's upstream interleaved format pretraining dataset. We adopt a similar approach to construct our MultI-Modal In-Context Instruction Tuning (MIMIC-IT) dataset. We then introduce Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following ability and in-context learning. We also optimize OpenFlamingo's implementation for researchers, democratizing the required training resources from 1times A100 GPU to 4times RTX-3090 GPUs, and integrate both OpenFlamingo and Otter into Huggingface Transformers for more researchers to incorporate the models into their customized training and inference pipelines.