Unconditional Priors Matter! Improving Conditional Generation of
  Fine-Tuned Diffusion Models

Prin Phunyaphibarn; Phillip Y. Lee; Jaihoon Kim; Minhyuk Sung

返回文献探索

Paper · arXiv 2503.20240

Unconditional Priors Matter! Improving Conditional Generation of Fine-Tuned Diffusion Models

Prin Phunyaphibarn, Phillip Y. Lee, Jaihoon Kim, Minhyuk Sung

22 upvotesMarch 26, 2025arXiv 预印本

AI 摘要

Replacing unconditional noise predictions in Classifier-Free Guidance with those from a pre-trained model improves conditional generation quality for diffusion models.

Classifier-Free Guidanceconditional diffusion modelsnoise predictiondropout ratepriorsfine-tuningbase modelimage generationvideo generationZero-1-to-3Versatile DiffusionDiTDynamiCrafterInstructPix2Pix

Abstract

Classifier-Free Guidance (CFG) is a fundamental technique in training conditional diffusion models. The common practice for CFG-based training is to use a single network to learn both conditional and unconditional noise prediction, with a small dropout rate for conditioning. However, we observe that the joint learning of unconditional noise with limited bandwidth in training results in poor priors for the unconditional case. More importantly, these poor unconditional noise predictions become a serious reason for degrading the quality of conditional generation. Inspired by the fact that most CFG-based conditional models are trained by fine-tuning a base model with better unconditional generation, we first show that simply replacing the unconditional noise in CFG with that predicted by the base model can significantly improve conditional generation. Furthermore, we show that a diffusion model other than the one the fine-tuned model was trained on can be used for unconditional noise replacement. We experimentally verify our claim with a range of CFG-based conditional models for both image and video generation, including Zero-1-to-3, Versatile Diffusion, DiT, DynamiCrafter, and InstructPix2Pix.