VLHP: Learning Discriminative Vision-Language Hybrid Prototypes for Weakly Supervised Semantic Segmentation

摘要

Recent advances in Weakly Supervised Semantic Segmentation (WSSS) focus on generating high-quality Class Activation Maps (CAMs) using image-level labels. However, the co-occurrence of foreground-background concepts in a single image often induces semantic confusion, which degrades the quality of conventional CAM-based approaches. In this paper, we propose VLHP, a novel framework that leverages vision-language hybrid prototypes to overcome semantic confusion. Specifically, VLHP constructs hybrid prototypes through cross-modal association between textual embeddings and visual features, generating discriminative semantic representations while effectively bridging the modality gap. To further improve discriminability, we introduce two dedicated strategies: Discriminative Explicit Alignment (DEA) to explore cross-modal consistent discrimination and Confounding Background Decoupling (CBD) to model co-occurring backgrounds and decouple them. Finally, a Prototype-driven Class-aware Decoder (PCD) employs these refined prototypes as category-specific priors to generate precise segmentation masks in a single-stage framework. Extensive experiments on PASCAL VOC and MS COCO benchmarks demonstrate that VLHP outperforms state-of-the-art alternatives. The code is available at https://github.com/fjy0105/VLHP.