Open-vocabulary semantic segmentation (OVSS) conducts pixel-level classification via text-driven alignment, where the domain discrepancy between base category training and open-vocabulary inference poses challenges in discriminative modeling of latent unseen category. To address this challenge, existing vision-language model (VLM)-based approaches demonstrate commendable performance through pre-trained multi-modal representations. However, the fundamental mechanisms of latent semantic comprehension remain underexplored, making the bottleneck for OVSS. In this work, we initiate a probing experiment to explore distribution patterns and dynamics of latent semantics in VLMs under inductive learning paradigms. Building on these insights, we propose X-Agent, an innovative OVSS framework employing latent semantic-aware ''agent'' to orchestrate cross-modal attention mechanisms, simultaneously optimizing latent semantic dynamic and amplifying its perceptibility. Extensive benchmark evaluations demonstrate that X-Agent achieves state-of-the-art performance while effectively enhancing the latent semantic saliency.
论文检索
输入标题、作者或关键词，从 7,876 篇学术成果中精准定位
Open vocabulary image segmentation tackles the challenge of recognizing dynamically adjustable, predefined novel categories at inference time by leveraging vision-language alignment. However, existing paradigms typically perform class-agnostic region segmentation followed by category matching, which deviates from the human visual system's process of recognizing objects based on semantic concepts, leading to poor alignment between region segmentation and object concepts. To bridge this gap, we propose a novel Cognition-Inspired Framework for open vocabulary image segmentation that emulates the human visual recognition process: first forming a conceptual understanding of an object, then perceiving its spatial extent. The framework consists of three core components: (1) A Generative Vision-Language Model (G-VLM) that mimics human cognition by generating object concepts to provide semantic guidance for region segmentation. (2) A Concept-Aware Visual Enhancer module that fuses textual concept features with global visual representations, enabling adaptive visual perception based on object concepts. (3) A Cognition-Inspired Mask Decoder that integrates local instance features with G-VLM-provided semantic cues, allowing selective classification over a subset of relevant categories. Extensive experiments demonstrate that our framework achieves significant improvements, reaching 27.2 PQ, 17.0 mAP, and 35.3 mIoU on A-150. It further attains 56.2, 28.2, 15.4, 59.2, 18.7, and 95.8 mIoU on Cityscapes, Mapillary Vistas, A-847, PC-59, PC-459, and PAS-20, respectively. In addition, our framework supports vocabulary-free image segmentation, offering enhanced flexibility in recognizing unseen categories.
DichotomyIR: Universal Image Reconstruction via Dichotomy Classification and Uncertainty Elimination
L1 loss is a classical regression loss function and achieves remarkable success in image reconstruction (IR) tasks. Theoretically, the L1 loss is the maximum likelihood of images with the Laplace prior distribution, which is simple and not that suitable for digital images with integer pixel values. This phenomenon results in high reconstruction error in detailed areas (such as boundary areas). For this, we investigate the effectiveness of classification loss functions like Cross Entropy (CE) loss in IR tasks, which is the maximum likelihood of Multinomial (or Bernoulli) prior distribution, and propose a novel image reconstruction framework named DichotomyIR. To apply CE loss to image reconstruction, we adopt the dichotomy method to transfer integer pixel values into 8-bit labels and further design a dual-branch Dichotomy Decoder (D-Decoder) in DichotomyIR to reconstruct high-quality (HQ) images I_L1 and I_CE supervised with L1 loss and CE loss, respectively. Next, we analyze the reconstructed uncertainties of these 2 images with different prior distributions and design an iterative uncertainty elimination (IUE) processing. Integrating the popular status spatial model with the IUE processing, we propose the Uncertainty Elimination Mamba (UEM) to eliminate the reconstructed uncertainty iteratively. With the D-Decoder and UEM, the DichotomyIR is flexible and feasible to be embedded with any current popular IR methods. Universal experiments on IR tasks demonstrate the effectiveness and efficiency of the proposed DichotomyIR, which strongly supports the importance of uncertainty elimination in IR tasks.
While Retrieval-Augmented Generation (RAG) augments Large Language Models (LLMs) with external knowledge, conventional single-agent RAG remains fundamentally limited in resolving complex queries demanding coordinated reasoning across heterogeneous data ecosystems. We present HM-RAG, a novel Hierarchical Multi-agent Multimodal RAG framework that pioneers collaborative intelligence for dynamic knowledge synthesis across structured, unstructured, and graph-based data. The framework is composed of a three-tiered architecture with specialized agents: a Decomposition Agent that dissects complex queries into contextually coherent sub-tasks via semantic-aware query rewriting and schema-guided context augmentation; Multi-source Retrieval Agents that carry out parallel, modality-specific retrieval using plug-and-play modules designed for vector, graph, and web-based databases; and a Decision Agent that uses consistency voting to integrate multi-source answers and resolve discrepancies in retrieval results through Expert Model Refinement. This architecture attains comprehensive query understanding by combining textual, graph-relational, and web-derived evidence, resulting in a remarkable 12.95% improvement in answer accuracy and a 3.56% boost in question classification accuracy over baseline RAG systems on the ScienceQA and CrisisMMD benchmarks. Notably, HM-RAG establishes state-of-the-art results in zero-shot settings on both datasets. Its modular architecture ensures seamless integration of new data modalities while maintaining strict data governance, marking a significant advancement in addressing the critical challenges of multimodal reasoning and knowledge synthesis in RAG systems.
Driven by the increasing demand for applications in augmented and virtual reality, egocentric action recognition has emerged as a prominent research area. It is typically divided into two subtasks: recognizing the performed behavior (i.e., verb component) and identifying the objects being acted upon (i.e., noun component) from the first-person perspective. However, most existing approaches treat these two components as independent classification tasks, focusing on extracting component-specific knowledge while overlooking their inherent semantic and contextual relationships, leading to fragmented representations and sub-optimal generalization capability. To address these challenges, we propose a prompt learning-based framework, EgoPrompt , to conduct the egocentric action recognition task. Building on the existing prompting strategy to capture the component-specific knowledge, we construct a Unified Prompt Pool space to establish interaction between the two types of component representations. Specifically, the component representations (from verbs and nouns) are first decomposed into fine-grained patterns with the prompt pair form. Then, these pattern-level representations are fused through an attention-based mechanism to facilitate cross-component interaction. To ensure the prompt pool is informative, we further introduce a novel training objective, Diverse Pool Criteria . This objective realizes our goals from two perspectives: Prompt Selection Frequency Regularization and Prompt Knowledge Orthogonalization. Extensive experiments are conducted on the Ego4D, EPIC-Kitchens, and EGTEA datasets. The results consistently show that EgoPrompt achieves state-of-the-art performance across within-dataset, cross-dataset, and base-to-novel generalization benchmarks.
Spiking Neural Network (SNN), as a next-generation neural network technology, use binary spike signals as carriers of information. They offer advantages such as low energy consumption, low computational complexity, and high information transmission rates. However, deep SNNs suffer from the gradient vanishing problem due to issues like the non-differentiability of the step function and neuron dormancy. To address this gradient problem, we first propose the MCLIF neuron, which optimizes the backpropagation mechanism and compensates for gradient information from both temporal and spatial dimensions. Furthermore, we design a spiking attention mechanism tailored to the temporal characteristics of SNNs. By introducing the QK Memory to embed temporal features, we make full use of information from different time steps. Additionally, we propose a gradient correction module to enhance the model's representational power from both temporal and spatial dimensions. The proposed SGM-Transformer achieves state-of-the-art (SOTA) performance in image classification tasks such as CIFAR10, CIFAR100, and CIFAR10-DVS, and also excels in industrial defect classification scenarios. The code will be made available after the paper is accepted.
Existing few-shot action recognition (FSAR) studies predominantly follow a metric learning framework, where prototypes are generated directly from features extracted by an encoder, and classification is performed via distance-based matching. However, due to the limited number of available samples, significant variations exist between different video features of the same class. As a result, the same query video may yield different classification results when matched against different sets of support videos. To address this issue, we propose a novel Hierarchical Meta-Prototypes Network (HMP-Net). The key innovation of our approach lies in the introduction of a category-agnostic and feature-agnostic meta-prototype module, which guides video feature mapping into a more suitable feature space. To optimize this meta-prototype, we design an alternating meta-prototype training strategy, where the model first learns to transform features under a fixed meta-prototype, and then the meta-prototype is refined to better guide feature mapping. Additionally, to adapt image-based metric learning models to video-based FSAR tasks, we introduce a series of lightweight adaptation modules. Specifically, we integrate an adapter into the encoder to improve video frame feature extraction, design a hierarchical prototype generation mechanism to enhance overall video understanding, and incorporate a task-specific perception module to extract unique features for each task. These adaptations make our model better suited for FSAR, significantly improving performance. We evaluate HMP-Net on five challenging benchmarks, and experimental results demonstrate that our model achieves new state-of-the-art performance on HMDB51, UCF101, Kinetics, and SthSthV2-Small. Extensive empirical evaluations further highlight the effectiveness and robustness of HMP-Net.
Multi-view classification aims to leverage information from multiple views of data to improve prediction performance by learning complementary and consistent representations. Therefore, in recent years, multi-view learning has attracted widespread attention in the community. Despite the success of existing multi-view learning methods, there are still some challenges when dealing with large-scale multi-view data. To address this issue, we propose a novel Multi-view Hashing Classification (MHC) framework to encode large-scale multi-view data as binary codes, thereby enhancing the semantic discrimination. Specifically, we leverage class prompts to generate corresponding textual descriptions for each instance and learn the corresponding anchor hash codes. To achieve intra-class compactness and inter-class separability, we propose Class-prompt Contrastive Learning (CCL) to enforce class-wise aggregation and separation in the Hamming space. To mitigate the cross-view heterogeneity gap, we propose a Supervised Cross-view Contrastive (SCC) module to align view-specific hash codes under label supervision. Finally, we present Boundary-aware Independent Hashing (BIH) that introduces boundary-aware constraints to reduce class boundary ambiguity, thereby improving the discrimination of fusion hash codes. Nevertheless, we observe that anchor hash codes could violate the bit independence assumption, which potentially hinders the optimization direction. To this end, we adopt a Bit-level Calibration Mechanism (BCM) to filter out redundant bits, thereby restoring bit independence. Extensive experiments conducted on ten benchmark datasets demonstrate the superiority of the proposed MHC in terms of both classification accuracy and inference efficiency. The code is released at https://github.com/Yuhang-lan04/MHC.
Topological structures in image data, such as connected components and loops, play a crucial role in understanding image content (e.g., biomedical objects). Despite remarkable successes of numerous image processing methods that rely on appearance information, these methods often lack sensitivity to topological structures when used in general deep learning (DL) frameworks. In this paper, we introduce a new general approach, called TopoImages (for Topology Images), which computes a new representation of input images by encoding local topology of patches. In TopoImages, we leverage persistent homology (PH) to encode geometric and topological features inherent in image patches. Our main objective is to capture topological information in local patches of an input image into a vectorized form. Specifically, we first compute persistence diagrams (PDs) of the patches, and then vectorize and arrange these PDs into long vectors for pixels of the patches. The resulting multi-channel image-form representation is called a TopoImage. TopoImages offers a new perspective for data analysis. To garner diverse and significant topological features in image data and ensure a more comprehensive and enriched representation, we further generate multiple TopoImages of the input image using various filtration functions, which we call multi-view TopoImages. The multi-view TopoImages are fused with the input image for DL-based classification, with considerable improvement. Our TopoImages approach is highly versatile and can be seamlessly integrated into common DL frameworks. Experiments on three public medical image classification datasets demonstrate noticeably improved accuracy over state-of-the-art methods.
Small-scale object detection remains a major challenge in semi-supervised object detection (SSOD), particularly in medical image analysis. Conventional teacher models often struggle to accurately capture the features of low-contrast small lesions, leading to noisy pseudo-labels in both localization and classification, which introduces severe uncertainty and degrades detection performance. To address this issue, we propose Dual Teacher, a novel multimodal semi-supervised detection framework designed to enhance pseudo-label reliability and improve small-scale lesion detection. Specifically, we introduce two complementary teacher models: Hybrid-Scale Teacher, which exploits downsampled views to strengthen multi-scale feature learning, and Entropy-Based Multi-Modal Teacher, which leverages entropy maps to refine the quality of small-scale pseudo-labels. To effectively fuse predictions from both teachers and resolve conflicts, we propose a Dempster-Shafer-based Dual-Teacher pseudo-label fusion strategy that explicitly models uncertainty and optimizes classification confidence. Additionally, we introduce a class-adaptive threshold mechanism that dynamically adjusts pseudo-label selection based on dual-teacher predictions, further boosting the recall of small-scale lesions. Extensive experiments on the Dental Disease Dataset, ChestX-Det and M3FD demonstrate that our method consistently surpasses state-of-the-art SSOD approaches. Code is available at: https://github.com/z316910/Dual-Teacher.git.
Low-Rank Adaptation (LoRA) and its variants have delivered strong capability in Parameter-Efficient Transfer Learning (PETL) by minimizing trainable parameters and benefiting from reparameterization. However, their projection matrices remain unrestricted during training, causing high representation redundancy and diminishing the effectiveness of feature adaptation in the resulting subspaces. While existing methods mitigate this by manually adjusting the rank or implicitly applying channel-wise masks, they lack flexibility and generalize poorly across various datasets and architectures. Hence, we propose ReSoRA, a method that explicitly models redundancy between mapping subspaces and adaptively Regularizes Subspace redundancy of Low-Rank Adaptation. Specifically, it theoretically decomposes the low-rank submatrices into multiple equivalent subspaces and systematically applies de-redundancy constraints to the feature distributions across different projections. Extensive experiments validate that our proposed method consistently facilitates existing state-of-the-art PETL methods across various backbones and datasets in vision-language retrieval and standard visual classification benchmarks. Besides, as a training supervision, ReSoRA can be seamlessly integrated into existing approaches in a plug-and-play manner, with no additional inference costs. Code is publicly available at: https://github.com/Lucenova/ReSoRA.
For skeleton-based action recognition, Graph Convolutional Networks (GCNs) are effective models. Still, their reliance on floatingpoint computations leads to high energy consumption, limiting their applicability in battery-powered devices. While energy-efficient, Spiking Neural Networks (SNNs) struggle to model skeleton dynamics, leading to suboptimal solutions. We propose Signal-SGN (Spiking Graph Convolutional Network), which utilizes the temporal dimension of skeleton sequences as the spike time steps and represents features as multi-dimensional discrete stochastic signals for temporal-frequency domain feature extraction. It combines the 1D Spiking Graph Convolution (1D-SGC) module and the Frequency Spiking Convolution (FSC) module to extract features from the skeleton represented as spiking form. Additionally, the Multi-Scale Wavelet Transform Feature Fusion (MWTF) module is proposed to extract dynamic spiking features and capture frequency-specific characteristics, enhancing classification performance. Experiments across three large-scale datasets reveal Signal-SGN exceeding state-of-the-art SNN-based methods in accuracy and computational efficiency while attaining comparable performance with GCN methods and significantly reducing theoretical energy consumption.
Recently, contrastive learning has emerged as a promising approach for multi-view clustering (MVC), as it enforces cross-view consistency and leverages complementary information from different views to enhance the analysis of heterogeneous data. However, traditional contrastive MVC methods suffer from an inherent limitation: their one-to-many contrast mechanism induces the False Negative Problem (FNP), where semantically similar intra-class instances are erroneously repelled. This phenomenon compromises intra-class consistency and ultimately degrades clustering performance. To overcome this issue, we propose a novel Pseudo lA bel gU ided univerS um lE arning (PAUSE) framework for robust multi-view clustering. Specifically, PAUSE operates in two synergistic stages: (1) A warm-up stage that employs dual contrastive learning to generate reliable pseudo-labels, establishing robust semantic relationships; (2) A fine-tuning stage that synthesizes universum samples via Mixup between anchor instances and out-of-class centroids, guided by the acquired pseudo-labels. This unique mechanism constructs generalized negative classes that expand inter-class margins while preserving intra-class cohesion. Crucially, the widened decision boundaries prevent misclassification of displaced intra-class instances, effectively circumventing FNP without requiring explicit negative pair correction. We further devise a robust universum contrastive loss that explicitly enforces cross-view consistency through adaptive boundary constraints. Extensive experiments on five multi-view benchmarks demonstrate that our PAUSE consistently outperforms 11 state-of-the-art multi-view learning methods. Our code is accessible at: https://github.com/xixi-555/PAUSE_main_code.
Sonar image recognition is a key technology in underwater exploration systems. Compared with natural images, sonar images have fewer texture details and are easily affected by heavy noise, making it more challenging for specialists to distinguish the subtle differences among classes. In view of this, studying fine-grained classification methods for sonar images with scarce annotations is of significant importance. To address this issue, we propose a Physics-Guided Teacher-Student (PGTS) framework to explore the unique physical information of sonar images while simultaneously mitigating the effects of limited annotations. First, PGTS reconstructs sonar signals through physical simulation and a specially designed physics-guided feature generation module, which allows it to bypass the time-consuming physical simulation during inference. Then, we design a multi-modal teacher model combines the reconstructed sonar signals and sonar images to extract discriminative features to generate robust pseudo labels for fine-grained target categories. Finally, the knowledge is transferred to a single-modal student model through consistency loss. Under the joint constraints of the teacher model and the reconstructed sonar physical signals, the student model continuously improves its performance in annotation-scarce scenarios. Notably, when merely 1% of the data is labeled, our method outperforms other state-of-the-art approaches by 12.46% in terms of accuracy.
The rapid evolution of face manipulation techniques poses a critical challenge for face forgery detection: cross-domain generalization. Conventional methods, which rely on simple classification objectives, often fail to learn domain-invariant representations. We propose HAMLET-FFD, a cognitively inspired Hierarchical Adaptive Multi-modal Learning framework that tackles this challenge via bidirectional cross-modal reasoning. Building on contrastive vision-language models such as CLIP, HAMLET-FFD introduces a knowledge refinement loop that iteratively assesses authenticity by integrating visual evidence with conceptual cues, emulating expert forensic analysis. A key innovation is a bidirectional fusion mechanism in which textual authenticity embeddings guide the aggregation of hierarchical visual features, while modulated visual features refine text embeddings to generate image-adaptive prompts. This closed-loop process progressively aligns visual observations with semantic priors to enhance authenticity assessment. By design, HAMLET-FFD freezes all pretrained parameters, serving as an external plugin that preserves CLIP's original capabilities. Extensive experiments demonstrate its superior generalization to unseen manipulations across multiple benchmarks, and visual analyses reveal a division of labor among embeddings, with distinct representations specializing in fine-grained artifact recognition.
Graph Neural Networks (GNNs) are effective for processing graph data; however, their heavy reliance on label information limits their generalization across domains. At the same time, Large Language Models (LLMs) have made significant progress across diverse domains, sparking growing interest in their potential for processing and understanding graph data. Nevertheless, their effectiveness is constrained by challenges such as space misalignment and global topology blindness, which arise because LLMs are trained primarily on Euclidean data rather than non-Euclidean structures. To address these challenges, we propose BridgeGLM, a novel framework that bridges graph and language spaces to improve domain generalization. BridgeGLM integrates topology-aware graph representations to capture higher-order structural relationships and employs a semantic-aware tokenizer to generate enriched node representations. Furthermore, we introduce three contrastive learning strategies based on graph interactions to effectively align graph and language representations. During testing, task-specific instruction templates facilitate zero-shot node classification. Extensive experiments on six datasets, covering both academic and recommendation graphs, show that BridgeGLM consistently outperforms state-of-the-art baselines across in-dataset, intra-domain, and cross-domain settings. In cross-domain settings, BridgeGLM achieves a 2-7% improvement in key performance metrics compared to the existing state-of-the-art method.
Multimodal models integrate visual, textual, and other data to achieve human-like understanding, but this fusion creates a conflict between cross-modal alignment and modality-specific expertise.The pursuit of unified feature spaces often undermines specialized knowledge in individual modalities, as shown by performance drops in unimodal tasks. To resolve this contradiction, we propose VAMP (Variational Alignment with Modality Preservation), a novel multimodal framework featuring a Dynamic Feature Diversion mechanism that partitions modal representations into two components-one preserving modality-specific expertise and the other enabling cross-modal alignment. Inspired by Variational Canonical Correlation Analysis, we introduce a shared space projection layer that maps features into a common representational space while preserving modality-specific characteristics. We further implement a Progressive Training Strategy that sequentially freezes different components before full fine-tuning, preventing mode collapse and enhancing generalization capabilities. Experimental results demonstrate VAMP's significant performance improvements across zero-shot image classification, cross-modal retrieval, and visual question answering, while simultaneously outperforming baseline models on unimodal tasks. This research provides an engineered solution to the ''knowledge dilution'' problem in cross-modal alignment.
Multi-view classification has demonstrated its ability to integrate diverse sources of information to significantly boost classification accuracy. To further enhance the reliability of these results, trusted multi-view learning methods have been developed. However, these approaches are designed for closed-set scenarios and fail when novel or unknown categories appear in open-world contexts. To address this limitation, we introduce the concept of Open Multi-View Learning, with the objective of detecting unknown categories with low confidence scores. We propose Trusted Open-World Multi-View Classification method for this problem. Specifically, we employ subjective logic to measure the uncertainty of data views. On these grounds, we propose dynamic opinion aggregation strategy based on their uncertainty measurements and theoretically prove this strategy can effectively detect unknown multi-view categories. The inter-view opinion consistency regularization is also adopted to mitigate conflicts between views. Experiments conducted on various multi-view datasets validate the reliability and robustness of our method.
Accurate nuclei classification serves as a critical cornerstone for disease diagnosis and treatment, yet challenged by the heterogeneity of tissue types, staining procedures, and imaging techniques. Recently, vision-language models (VLMs) have demonstrated impressive success in the natural image field and advanced potential in medical imaging. However, the adaptation of VLMs to nuclei classification still poses several challenges, including limited generalization capability and coarse image-text feature alignment. In this paper, we propose SIGNPrompt, a domain-Specific Interactive Prompt learning framework for Generalized Nuclei classification. Specifically, to unleash the generalization capability of VLM-based models, we introduce a prior-guided domain adapting module that integrates nuclei prior information from the large language model (LLM), enabling flexible and robust adaptation to the inherent heterogeneity across pathological domains. Moreover, we develop a multi-modal interactive prompting mechanism to refine image-text feature alignment by leveraging the interdependence between visual and language prompting, thus enhancing the discriminability of nuclei categories. In addition, a simple yet effective noise-adding strategy is proposed to mitigate the overfitting problem in prompt learning. Extensive experiments on diverse public benchmarks and challenging zero-shot scenarios validate that SIGNPrompt consistently outperforms state-of-the-art (SOTA) methods in both accuracy and generalization.
Audio-visual Generalized Zero-Shot Learning ((G)ZSL) has attracted significant attention for its ability to identify unseen classes in general video classification tasks. However, modality imbalance in (G)ZSL leads to over-reliance on the optimal modality, reducing discriminative capabilities for unseen classes. Though recent studies have attempted to address this issue, two challenges still remain unsolved: (a) Quality discrepancies, where modalities offer differing quantities and qualities of information for the same concept. (b) Content discrepancies, where the contributions of different samples within the same modality exhibit significant differences. To address these challenges, we propose a Discrepancy-Aware Attention Network (DAAN) for Enhanced Audio-Visual (G)ZSL. Our approach introduces a Redundant-Noise Mitigation Attention (RNMA) unit to minimize content discrepancies by mitigating redundant information in modalities and a Contrastive Sample Gradient Modulation (CSGM) mechanism to adjust gradient magnitudes and balance quality discrepancies. We quantify modality contributions by integrating optimization and convergence rate for more precise gradient modulation in CSGM. Experiments demonstrate DAAN achieves state-of-the-art performance on benchmark datasets, with ablation studies validating the effectiveness of individual modules. Code is available at https://github.com/xiaoxinning/DAAN-GZSL.