RELATE3D: REfocusing Latent Adapter for Targeted local Enhancement and Editing in 3D Generation

摘要

Recent advancements in 3D generation techniques have simplified the tedious manual process of 3D asset production. Among these methods, 3D native latent diffusion models are particularly effective in generating high-quality geometric details. However, achieving local enhancement and editing of the generated 3D models remains a challenge due to the limited understanding of the relationship between text,images,and 3D in terms of local semantics and feature space.We explore and reveal the characteristics of the native 3D latent space, make it decomposable and low-rank, thereby enabling efficient and effective learning for multimodal local alignment. Based on this, we introduce RELATE3D, a novel approach that combines a Refocusing Adapter with part-to-latent correspondence guided training for precise local enhancement and part-level editing of 3D geometry. The Refocusing Adapter incorporates partial image and caption signals, and, combined with part-to-latent mapping, directs modifications to the relevant latent dimensions during latent diffusion process. We validate the effectiveness of our approach through extensive experiments and ablation studies, showcasing the capabilities of our generative local enhancement and editing process, as well as global refinement.