Score Distillation Sampling with Learned Manifold Corrective

Thiemo Alldieck; Nikos Kolotouros; Cristian Sminchisescu

返回文献探索

Paper · arXiv 2401.05293

Score Distillation Sampling with Learned Manifold Corrective

Thiemo Alldieck, Nikos Kolotouros, Cristian Sminchisescu

12 upvotesJanuary 10, 2024arXiv 预印本

AI 摘要

The paper addresses issues in the Score Distillation Sampling loss function by training a shallow network to remove noisy gradients introduced by high text guidance, demonstrating improvements in various applications like image synthesis, network training, and 3D synthesis.

image diffusion modelScore Distillation Sampling (SDS)loss functionnoisy gradientstimestep-dependent denoising deficiencyoptimization-based image synthesiszero-shot image translationtext-to-3D synthesis

Abstract

Score Distillation Sampling (SDS) is a recent but already widely popular method that relies on an image diffusion model to control optimization problems using text prompts. In this paper, we conduct an in-depth analysis of the SDS loss function, identify an inherent problem with its formulation, and propose a surprisingly easy but effective fix. Specifically, we decompose the loss into different factors and isolate the component responsible for noisy gradients. In the original formulation, high text guidance is used to account for the noise, leading to unwanted side effects. Instead, we train a shallow network mimicking the timestep-dependent denoising deficiency of the image diffusion model in order to effectively factor it out. We demonstrate the versatility and the effectiveness of our novel loss formulation through several qualitative and quantitative experiments, including optimization-based image synthesis and editing, zero-shot image translation network training, and text-to-3D synthesis.