Relightful Harmonization: Lighting-aware Portrait Background Replacement
CVPR 2024



Portrait harmonization aims to composite a subject into a new background, adjusting its lighting and color to ensure harmony with the background scene. Existing harmonization techniques often only focus on adjusting the global color and brightness of the foreground and ignore crucial illumination cues from the background such as apparent lighting direction, leading to unrealistic compositions. We introduce Relightful Harmonization, a lighting-aware diffusion model designed to seamlessly harmonize sophisticated lighting effect for the foreground portrait using any background image. Our method outperforms existing benchmarks in visual fidelity and lighting coherence, showing superior generalization in real-world testing scenarios, highlighting its versatility and practicality.


Our approach unfolds in three stages. First, we introduce a lighting representation module that allows our diffusion model to encode lighting information from target image background. Second, we introduce an alignment network that aligns lighting features learned from image background with lighting features learned from panorama environment maps, which is a complete representation for scene illumination. Last, to further boost the photorealism of the proposed method, we introduce a novel data simulation pipeline that generates synthetic training pairs from a diverse range of natural images, which are used to refine the model.


Real-world Harmonization

Lighting Effects: In cases where the target backgrounds offer clear lighting cues, our method generates visually convincing lighting effects (Left: Composed. Right: Harmonized). Additionally, upon flipping the background, we note consistent and appropriate adjustments to the lighting direction in the output.


Shadow Neutralization: Our method effectively neutralizes pronounced shadows in the input while accommodating the ambient lighting of the background.


Shadow Casting: When applied to backgrounds with intense lighting conditions, e.g., with overhead sunlight, our method casts plausible self-occlusion shadows. scales

Temporally-changing backgrounds: Our method harmonizes images with backgrounds captured across timelapse sequences (source) featuring changing lighting conditions. scales

Spatially-moving backgrounds: Our method consistently adjusts lighting when applied to moving backgrounds with spatially changing lighting directions . scales

Reference-based harmonization: Our approach allows for reference-based background replacement. This involves removing the subject from the reference image (upper left) to create a background (lower left) for composition. The harmonized results (right) achieve lighting effects closely resembling those in the reference. scales


Compared with both relighting-based methods and harmonization-based methods, our method provides a more versatile solution for arbitraty backgrounds, and more effectively harmonizes incoherent foreground color, lighting and shadow.



title={Relightful Harmonization: Lighting-aware Portrait Background Replacement}, 
author={Mengwei Ren and Wei Xiong and Jae Shin Yoon and Zhixin Shu and Jianming Zhang and HyunJoon Jung and Guido Gerig and He Zhang},


We are grateful for Yannick Hold-Geoffroy who provides the panorama environment maps and set up the rendering scripts. We thank Chaowei Company for the support of light stage dataset. We thank David Futschik for running the testing on Total Relighting codebase. We thank Scott Cohen for providing great inspiration for our project name.

scales scales

*Generated from Relightful Harmonization

The website template was borrowed from Nerfies.