DFCNet: Dual-Factor Compensatory Clustering Network for Modality-Imbalanced Generalized Zero-Shot Learning

摘要

In audio-visual joint analysis, generalized zero-shot learning (GZSL) aims to recognize unseen categories by aligning semantic information across modalities. However, significant challenges arise from temporal and semantic discrepancies between audio and video modalities. Traditional methods typically depend on static semantic embeddings, thereby overlooking the dynamic nature of these modalities and often resulting in modality imbalance. We propose the Dual-Factor Compensatory Clustering Network (DFCNet), an end-to-end framework for dynamic fusion and optimized alignment of heterogeneous modal information to address these limitations. DFCNet employs a multi-branch architecture, integrating a parallel multi-layer perceptron (MLP) for semantic modeling and a Bidirectional Long Short-Term Memory (BiLSTM) network for capturing temporal consistency. The Compensatory Fusion Block (CFB) employs tensor decomposition to facilitate cross-modal coupling, where the low-rank representation decomposition aligns intra-modal feature distributions. Additionally, we introduce the Dual-Factor Clustering Multi-objective Optimization Framework (DCMOF), which ensures gradient equilibrium and adaptively adjusts modality contribution weights to strengthen robust cross-modal alignment. Designed as a pluggable module, DFCNet can be seamlessly integrated into existing base models. Experimental results demonstrate that our framework significantly improves the performance of the Audio-Visual Cross-Modal Alignment (AVCA) model across multiple GZSL datasets. Ablation studies further validate the critical role of CFB in cross-modal alignment and highlight the significance of DCMOF in optimizing modality coordination. The code is available at https://github.com/ATKEROM/DFCNet.