Semantica: An Adaptable Image-Conditioned Diffusion Model

Manoj Kumar; Neil Houlsby; Emiel Hoogeboom

返回文献探索

Paper · arXiv 2405.14857

Semantica: An Adaptable Image-Conditioned Diffusion Model

Manoj Kumar, Neil Houlsby, Emiel Hoogeboom

9 upvotesMay 23, 2024arXiv 预印本

AI 摘要

Semantica, an image-conditioned diffusion model, generates high-quality images based on semantic information and adapts to different datasets without fine-tuning.

image-conditioned diffusion modelweb-scale image pairspretrained image encoderssemantic-based data filteringtransfer properties

Abstract

We investigate the task of adapting image generative models to different datasets without finetuneing. To this end, we introduce Semantica, an image-conditioned diffusion model capable of generating images based on the semantics of a conditioning image. Semantica is trained exclusively on web-scale image pairs, that is it receives a random image from a webpage as conditional input and models another random image from the same webpage. Our experiments highlight the expressivity of pretrained image encoders and necessity of semantic-based data filtering in achieving high-quality image generation. Once trained, it can adaptively generate new images from a dataset by simply using images from that dataset as input. We study the transfer properties of Semantica on ImageNet, LSUN Churches, LSUN Bedroom and SUN397.