IFS-Light: An Interactive Framework for Single-view Face Relighting with both Facial and Lighting Consistency

摘要

Single-view face relighting aims to adjust the portrait lighting while preserving the original background. Although recent diffusion-based methods achieve great relit results by using reference lighting and facial features as conditions for the diffusion relighting process, they are limited by the incompleteness of these conditions, such as the absence of explicit constraints on skin tone and the lack of spatial coverage for hard shadows, which results in facial and lighting inconsistencies. To address these challenges, we propose IFS-Light, an interactive framework that leverages spatial-nonspatial conditioning mechanism to localize facial features and reference lighting, then optimize their interplay in the relighting process. To ensure facial consistency, we first combine skin-tone-scaled conditions with shape information for tone adjustment, enhanced by a detail mask that identifies modifiable facial regions. Skin and shape parameters are then optimized to preserve both skin tone and fine details. To maintain lighting consistency, we propose a ray-tracing-based formulation that decomposes reference lighting into diffuse and non-diffuse components. These, integrated with shape information, assist in positioning light and shadow regions. Both components are then encoded for precise control over color and intensity. In addition, we propose an innovative and user-friendly solution for adjusting light conditions, which enables the user to precisely adjust the position of the light source to flexibly control both light intensity and direction, thereby making it easier to achieve the desired relighting results. Extensive experiments show that IFS-Light achieves superior relighting results compared to state-of-the-art methods. The code and appendix are available https://github.com/mRobotit/IFS-Light