CaRaFe: Camera-Radar Radiance Fields for Scene Reconstruction

摘要

Radar neural reconstruction methods have recently achieved robust 3D scene occupancy from radar measurements alone, as they provide metric depth and are insensitive to adverse weather and low light. However, while these methods can recover some 3D geometry, their input radar data mixes information across elevation into a 2D range-azimuth measurement. This fundamentally limits their elevation resolution, especially in automotive scenes with limited vertical baselines. Camera images offer the opposite tradeoff: they contain strong, high resolution cues for object elevation but struggle with accurate depth and in adverse conditions. We propose CaRaFe, a method that leverages the complementary strengths of camera and radar for 3D reconstruction in challenging urban settings. CaRaFe employs a single multi-modal neural field, relying on conventional novel view synthesis as a supervision signal. Radar supervision provides a valuable geometric constraint to camera rendering that reduces shape-radiance ambiguity, while camera supervision allows for more accurate object elevation disambiguation and fewer missing structures in regions weakly observed by radar. We validate CaRaFe across diverse in-the-wild driving scenes, demonstrating favorable reconstruction quality over both radar and camera methods. Code for this paper is available at light.princeton.edu/carafe.