Subobject-level Image Tokenization

Delong Chen; Samuel Cahyawijaya; Jianfeng Liu; Baoyuan Wang; Pascale Fung

返回文献探索

Paper · arXiv 2402.14327

Subobject-level Image Tokenization

Delong Chen, Samuel Cahyawijaya, Jianfeng Liu, Baoyuan Wang, Pascale Fung

18 upvotesFebruary 22, 2024arXiv 预印本

AI 摘要

Subobject-level tokenization in vision models improves image description learning by using semantically meaningful segments instead of fixed-size patches.

Transformer-based vision modelsimage tokenizersubobject tokenizationsegment anything modelsSequence-to-sequence AutoEncoder (SeqAE)compact embedding vectorsvision language learningpatch-level tokenization

Abstract

Transformer-based vision models typically tokenize images into fixed-size square patches as input units, which lacks the adaptability to image content and overlooks the inherent pixel grouping structure. Inspired by the subword tokenization widely adopted in language models, we propose an image tokenizer at a subobject level, where the subobjects are represented by semantically meaningful image segments obtained by segmentation models (e.g., segment anything models). To implement a learning system based on subobject tokenization, we first introduced a Sequence-to-sequence AutoEncoder (SeqAE) to compress subobject segments of varying sizes and shapes into compact embedding vectors, then fed the subobject embeddings into a large language model for vision language learning. Empirical results demonstrated that our subobject-level tokenization significantly facilitates efficient learning of translating images into object and attribute descriptions compared to the traditional patch-level tokenization. Codes and models will be open-sourced at https://github.com/ChenDelong1999/subobjects.