Abstract

Portrait harmonization aims to composite a subject into a new background, adjusting its lighting and color to ensure harmony with the background scene. Existing harmonization techniques often only focus on adjusting the global color and brightness of the foreground and ignore crucial illumination cues from the background such as apparent lighting direction, leading to unrealistic compositions. We introduce Relightful Harmonization, a lighting-aware diffusion model designed to seamlessly harmonize sophisticated lighting effect for the foreground portrait using any background image. Our method outperforms existing benchmarks in visual fidelity and lighting coherence, showing superior generalization in real-world testing scenarios, highlighting its versatility and practicality.

Method

Our approach unfolds in three stages. First, we introduce a lighting representation module that allows our diffusion model to encode lighting information from target image background. Second, we introduce an alignment network that aligns lighting features learned from image background with lighting features learned from panorama environment maps, which is a complete representation for scene illumination. Last, to further boost the photorealism of the proposed method, we introduce a novel data simulation pipeline that generates synthetic training pairs from a diverse range of natural images, which are used to refine the model.

Real-world Harmonization

Lighting Effects: In cases where the target backgrounds offer clear lighting cues, our method generates visually convincing lighting effects (Left: Composed. Right: Harmonized). Additionally, upon flipping the background, we note consistent and appropriate adjustments to the lighting direction in the output.

Shadow Neutralization: Our method effectively neutralizes pronounced shadows in the input while accommodating the ambient lighting of the background.

Shadow Casting: When applied to backgrounds with intense lighting conditions, e.g., with overhead sunlight, our method casts plausible self-occlusion shadows.

Temporally-changing backgrounds: Our method harmonizes images with backgrounds captured across timelapse sequences (source) featuring changing lighting conditions.

Spatially-moving backgrounds: Our method consistently adjusts lighting when applied to moving backgrounds with spatially changing lighting directions .

Reference-based harmonization: Our approach allows for reference-based background replacement. This involves removing the subject from the reference image (upper left) to create a background (lower left) for composition. The harmonized results (right) achieve lighting effects closely resembling those in the reference.

Benchmarking

Compared with both relighting-based methods and harmonization-based methods, our method provides a more versatile solution for arbitraty backgrounds, and more effectively harmonizes incoherent foreground color, lighting and shadow.