Discrepancy-Aware Attention Network for Enhanced Audio-Visual Generalized Zero-Shot Learning

摘要

Audio-visual Generalized Zero-Shot Learning ((G)ZSL) has attracted significant attention for its ability to identify unseen classes in general video classification tasks. However, modality imbalance in (G)ZSL leads to over-reliance on the optimal modality, reducing discriminative capabilities for unseen classes. Though recent studies have attempted to address this issue, two challenges still remain unsolved: (a) Quality discrepancies, where modalities offer differing quantities and qualities of information for the same concept. (b) Content discrepancies, where the contributions of different samples within the same modality exhibit significant differences. To address these challenges, we propose a Discrepancy-Aware Attention Network (DAAN) for Enhanced Audio-Visual (G)ZSL. Our approach introduces a Redundant-Noise Mitigation Attention (RNMA) unit to minimize content discrepancies by mitigating redundant information in modalities and a Contrastive Sample Gradient Modulation (CSGM) mechanism to adjust gradient magnitudes and balance quality discrepancies. We quantify modality contributions by integrating optimization and convergence rate for more precise gradient modulation in CSGM. Experiments demonstrate DAAN achieves state-of-the-art performance on benchmark datasets, with ablation studies validating the effectiveness of individual modules. Code is available at https://github.com/xiaoxinning/DAAN-GZSL.