Noise-Robust Cross-modal Learning for Reliable 2D-3D Retrieval

摘要

With the rapid proliferation of 2D and 3D data, driven by advances in virtual environments and AI-generated content, cross-modal 2D-3D retrieval has attracted growing attention. However, it is easy to introduce noisy labels due to the spatial complexity of 3D content. Although various methods have been proposed to address this issue, they still struggle to handle or effectively re-exploit noisy samples. Moreover, existing approaches are prone to error accumulation due to the self-reinforcement of the model during training. To address these issues, we propose a Noise-Robust Cross-modal Learning (NRCL) framework based on the hybrid strategy. Specifically, NRCL introduces a Robust Cross-modal Co-separator (RCC), which separates noisy samples from clean ones by leveraging modality complementarity and adopting a co-teaching paradigm to mitigate potential error accumulation of the single model during training. Besides, a Reliable Soft Rectification (RSR) method is adopted to correct noisy labels by aggregating historical and dual-model predictions, exploiting the discriminative information from noisy samples. Finally, a Robust Cross-modal Prototype Learning (RCPL) is proposed to improve the discriminability of inter-class and alleviate the inherent gaps across modalities in the shared common space, which jointly leverages clean and rectified labels, thereby mitigating the detrimental impact of noisy samples. Extensive experiments are conducted on three 3D multimodal datasets to verify the effectiveness of our method by comparing it with 10 state-of-the-art methods. The code is available at https://github.com/yangaonidaye123/NRCL.