Style transfer involves transferring the style from a reference image to the content of a target image. Recent advancements in LoRA-based (Low-Rank Adaptation) methods have shown promise in effectively capturing the style of a single image. However, these approaches still face significant challenges such as content inconsistency, style misalignment, and content leakage. In this paper, we comprehensively analyze the limitations of the standard diffusion parameterization, which learns to predict noise, in the context of style transfer. To address these issues, we introduce ConsisLoRA, a LoRA-based method that enhances both content and style consistency by optimizing the LoRA weights to predict the original image rather than noise. We also propose a two-step training strategy that decouples the learning of content and style from the reference image. To effectively capture both the global structure and local details of the content image, we introduce a stepwise loss transition strategy. Additionally, we present an inference guidance method that enables continuous control over content and style strengths during inference. Through both qualitative and quantitative evaluations, our method demonstrates significant improvements in content and style consistency while effectively reducing content leakage.
We introduce ConsisLoRA, a LoRA-based method designed to enhance content and style consistency in style transfer. ConsisLoRA is based on three main ideas. First, we replace the standard \epsilon-prediction loss with x_0-prediction loss. Second, we introduce a two-step training strategy that more effectively separates the content and style representations within the style image. Third, we propose a stepwise loss transition strategy to simultaneously capture the overall structure and fine details of the content image.
Results generated by ConsisLoRA for three image stylization tasks: (Top) Transferring the style from a reference image to the content of a target image; (Middle) Applying the style described by prompts to a content image; (Bottom) Generating objects described by prompts with the style extracted from a reference image.
