Lumos : Empowering Multimodal LLMs with Scene Text Recognition

Ashish Shenoy; Yichao Lu; Srihari Jayakumar; Debojeet Chatterjee; Mohsen Moslehpour; Pierce Chuang; Abhay Harpale; Vikas Bhardwaj; Di Xu; Shicong Zhao; Longfang Zhao; Ankit Ramchandani; Xin Luna Dong; Anuj Kumar

Paper · arXiv 2402.08017

Lumos : Empowering Multimodal LLMs with Scene Text Recognition

Ashish Shenoy, Yichao Lu, Srihari Jayakumar, Debojeet Chatterjee, Mohsen Moslehpour, Pierce Chuang, Abhay Harpale, Vikas Bhardwaj, Di Xu, Shicong Zhao, Longfang Zhao, Ankit Ramchandani, Xin Luna Dong, Anuj Kumar

27 upvotesFebruary 12, 2024arXiv 预印本

AI 摘要

Lumos is an end-to-end multimodal question-answering system that integrates Scene Text Recognition and a Multimodal Large Language Model to improve text understanding and efficiency from first-person images.

Scene Text RecognitionSTRMultimodal Large Language ModelMM-LLMfirst person point-of-viewmultimodal question-answering

Abstract

We introduce Lumos, the first end-to-end multimodal question-answering system with text understanding capabilities. At the core of Lumos is a Scene Text Recognition (STR) component that extracts text from first person point-of-view images, the output of which is used to augment input to a Multimodal Large Language Model (MM-LLM). While building Lumos, we encountered numerous challenges related to STR quality, overall latency, and model inference. In this paper, we delve into those challenges, and discuss the system architecture, design choices, and modeling techniques employed to overcome these obstacles. We also provide a comprehensive evaluation for each component, showcasing high quality and efficiency.