Multi-label image classification requires simultaneously recognizing multiple objects with complex interdependencies. While existing attention-based methods are prominent, their performance is hampered by two forms of representation entanglement: 1) Spatial entanglement, where contextual interference from backgrounds and co-occurring objects confuses specific object representations; 2) Semantic entanglement, where models overfit label co-occurrence priors, thereby impairing a genuine semantic understanding of the image. To address these challenges, we propose an Object-Purified Representation Learning framework. Concretely, for spatial entanglement, we propose the Spatial-wise Representation Purification Module that employs Spatial-Purified Attention to eliminate object-irrelevant feature activations for contextual interference reduction, combined with Spatial-Aware Supervision to enhance object perception capability. For semantic entanglement, we develop the Semantic-wise Association Purification Module that synergistically integrates our proposed average message with the original co-occurrence-based message. This design effectively models co-occurrence relationships while preventing their overemphasis. Furthermore, we design the Bidirectional Representation Refinement Module to efficiently enhance representations, further boosting classification performance. Extensive experiments on multiple benchmark datasets with different configurations demonstrate that our proposed method achieves state-of-the-art performance.
论文检索
输入标题、作者或关键词，从 1,620 篇学术成果中精准定位
Automatic echocardiography video segmentation is a powerful tool for improving the accuracy of cardiovascular function assessment. However, it remains a challenging task owing to (1) extensive speckle noise and blurred boundaries, (2) dramatic shape variations of targeting structures across frames, and (3) limited labeled data due to the high cost of annotation. In this paper, we present a novel semi-supervised segmentation model based on Vision Mamba (Vim) to comprehensively tackle these challenges; we call it EchoVim. Our framework introduces three technical innovations: First, a bidirectional inference mechanism (BIM) which can propagate label information bidirectionally from end-diastolic (ED) and end-systolic (ES) frames to generate pseudo-labels, coupled with confidence-aware dynamic updating to progressively refine supervision signals. Second, a dynamic interaction temporal alignment (DITA) module that establishes anatomical correspondence across frames by adaptively enhancing features near temporally stable regions while suppressing motion-irrelevant artifacts, effectively addressing variations in cardiac shape. Third, a semantic token-attentive refinement (STR) module that constructs low-rank semantic tokens to encode cardiac structure priors, utilizing attention-guided nonlinear transformations to disentangle speckle noise from true anatomical patterns. We conduct extensive experiments on two benchmarking echocardiography video datasets: CAMUS and EchoNet-Dynamic, and the results demonstrate that our method outperforms existing state-of-the-art approaches with real-time inference. Codes are available at https://github.com/guojx2255/EchoVim.
Contrastive Language-Image Pre-training (CLIP) has achieved success on multiple downstream tasks by aligning image and text modalities. However, the nature of global contrastive learning limits CLIP's ability to comprehend compositional concepts, such as relations and attributes. Although recent studies employ global hard negative samples to improve compositional understanding, these methods significantly compromise the model's inherent general capabilities by forcibly distancing textual negative samples from images in the embedding space. To overcome this limitation, we introduce a Decoupled Global-Local Alignment (DeGLA) framework that improves compositional understanding while substantially mitigating losses in general capabilities. To optimize the retention of the model's inherent capabilities, we incorporate a self-distillation mechanism within the global alignment process, aligning the learnable image-text encoder with a frozen teacher model derived from an exponential moving average. Under the constraint of self-distillation, it effectively mitigates the catastrophic forgetting of pretrained knowledge during fine-tuning. To improve compositional understanding, we first leverage the in-context learning capability of Large Language Models (LLMs) to construct about 2M high-quality negative captions across five types. Subsequently, we propose the Image-Grounded Contrast (IGC) loss and Text-Grounded Contrast (TGC) loss to enhance vision-language compositionally. Experimental results across both general and compositional reasoning tasks validate the effectiveness of the DeGLA framework. Our code is released at https://github.com/xiaoxing2001/DeGLA.
Ultra-fine-grained visual classification (ultra-FGVC) targets at classifying sub-grained categories of fine-grained objects. This inevitably requires discriminative representation learning within a limited training set. Exploring intrinsic features from the object itself via contrastive learning has demonstrated great progress towards learning discriminative representation. Yet forcingly dividing highly similar categories at the representation level may over-guide the learned feature space, leading to overfitting in the ultra-FGVC tasks. To this end, this paper introduces CLA-Net, a novel contrastive Lie algebra learning framework to address this fundamental problem in ultra-FGVC. The core design is a self-supervised module that performs self-shuffling and masking and then distinguishes these altered images from other images at a second-order representation level. This drives the model to learn an optimized feature space that has a large inter-class distance while remaining tolerant to intra-class variations. By incorporating this self-supervised module, the network acquires more knowledge from the intrinsic structure of the input data, which improves the generalization ability without requiring extra manual annotations. CLA-Net demonstrates strong performance on eight publicly available datasets, demonstrating its effectiveness in the ultra-FGVC task. The code is available at: https://github.com/zichengpan/CLA-NET.
Open-ended visual storytelling presents a formidable challenge for current text-to-image models, which frequently struggle to preserve both narrative coherence and consistent character depictions across generated sequences. To address this, we introduce StoryCrafter, a multi-character diffusion model that leverages a novel instance-level cross-attention module with supervised fine-tuning to ensure precise text-character alignment and consistent multi-character interactions throughout the narrative. Further, we propose Direct-Diffusion Group Relative Policy Optimization (D2GRPO), a novel RLHF stage that optimizes denoising strategies using automated story-aligned rewards, selecting the best candidate frames from a generated group. We evaluate our approach through human assessments and vision language model (VLM) scoring, measuring text-to-image alignment, style and character consistency, and fine-grained detail quality. Experiments on three benchmarks demonstrate that StoryCrafter outperforms existing methods, achieving 7% improvements in storytelling consistency and 10% in character accuracy, while outperforming baselines in both human and VLM evaluations.
Referring video object segmentation (RVOS) extracts objects from videos based on provided text narrations. Previous approaches typically work on all video frames simultaneously through offline processing, but this is not always possible. Offline processing becomes also ineffective for long videos, as directly cutting the video into short clips to fit memory limits results in the loss of temporal consistency. To make RVOS more applicable to real-world video streams or long video scenarios, we introduce a Filtering Framework for RVOS (FF-RVOS), the first model capable of operating in online, semi-online, and offline modes with just one training session. We redesign RVOS as a stochastic optimization problem and leverage filtering to optimize object states across temporal sequences. Our method enhances temporal consistency for video streams and when a long video is processed in shorter clip sequences due to memory limitations. FF-RVOS demonstrates superior performance compared to previous state-of-the-art methods on public benchmarks with clear improvements, especially when the video is cut into short clips for processing. Our framework can also be embedded into different offline methods to boost temporal consistency. The project page is https://github.com/haliphinx/FF-RVOS.
Vision-Language Models (VLMs) such as CLIP have demonstrated outstanding performance in cross-modal tasks, but the prohibitive computational cost hinders practical deployment. Although Knowledge Distillation (KD) provides a promising compression paradigm, most existing methods rely heavily on feature imitation and contrastive relations without explicit fine-grained alignment. Additionally, they do not fully leverage the multimodal interaction knowledge from the teacher model, restricting cross-modal semantic alignment. To address these challenges, we propose KAID, a Knowledge-Aware Interactive Distillation method for VLMs. Specifically, we first pretrain a large CLIP teacher model with domain few-shot labels and store text features as category vectors. Then, an Image Feature Matching (IFM) module is introduced to calculate the feature distribution of teacher-student models with improved cosine similarity, which achieves hierarchical knowledge transfer from global to local levels and enhances the fine-grained perception of student model through joint optimization. Moreover, a Pixel-Wise Alignment (PWA) module is constructed between the teacher's text features and the student's image features, employing a cross-modal attention mechanism to establish semantic associations, while a Text-guided Pixel alignment Loss function (TPloss) is concurrently designed to enhance the student's comprehension capabilities. Ultimately, the well-trained student model is used for inference. Extensive experiments on 11 datasets validate the effectiveness of our method. Specifically, our method achieves average improvements of 2.14% and 2.40% on the base and new classes across these datasets.
The reasoning segmentation task involves segmenting objects within an image by interpreting implicit user instructions. Despite significant advancements made by existing approaches, they remain constrained by low perceptual resolution, as visual encoders are typically pre-trained at lower resolutions. Furthermore, simply interpolating the positional embeddings of visual encoders to enhance perceptual resolution yields only marginal performance improvements while incurring substantial computational costs. To address this, we propose HRSeg, an efficient model with high-resolution fine-grained perception. It features two key innovations: High-Resolution Perception (HRP) and High-Resolution Enhancement (HRE). The HRP module processes high-resolution images through cropping, integrating local and global features for multi-granularity quality. The HRE module enhances mask features by integrating fine-grained information from high-resolution images, refining their alignment with text features for precise segmentation. Extensive ablation studies validate the effectiveness of our modules, while comprehensive experiments on multiple benchmark datasets demonstrate HRSeg's superior performance. Code will be available at https://github.com/WeihuangLin/HRSeg.
The domain gap between pretraining data (e.g., ImageNet, LUPerson) and downstream ReID datasets often leads to suboptimal performance when directly fine-tuning pretrained models. While existing methods attempt to bridge this gap by incorporating additional modalities (e.g., text, 3D data) or visual cues (e.g., pose, body masks), these approaches introduce two key limitations: (1) they may distract the model with irrelevant factors like background clutter or clothing variations, and (2) they inevitably increase computational overhead during inference. To address these issues, we propose the Weak Saliency Feedback Transformer (WSFFormer), inspired by the feedback mechanisms in biological visual systems. Unlike traditional one-way feature propagation, WSFFormer employs an adaptive feedback loop during training to enhance low-response regions, enabling the model to capture richer and more discriminative features. The WSFFormer introduces three key components: (1) The Lateral Feedback Module (LFM) mimics retinal lateral inhibition by adaptively suppressing high-response regions and amplifying weak discriminative features, forcing attention on subtle details; (2) The Progressive Feedback Module (PFM) refines feedback through deep-to-shallow closed-loop propagation, blending high-level semantics with spatial details; (3) The Feedback Sensitive Entropy Loss (FSE Loss) optimizes target-domain adaptation by quantifying divergence between forward and feedback-corrected features. Experiments on holistic/occluded ReID benchmarks show WSFFormer outperforms ViT/Swin-based SOTA methods without extra inference cost.
Group re-identification (G-ReID) attempts to recognize human groups across multiple camera perspectives. It is a challenging task due to occlusion, perspective variation, and illumination change. While Person Attribute Recognition (PAR) methods have shown robustness under similar challenges, yet their potential in G-ReID remains unexplored. Though existing G-ReID datasets are well-crafted, however, they lack person-level attribute annotations. This restricts G-ReID methods to explore attribute-based matching, essentially limiting their capability of multi-modal analysis. In this work, we bridge this gap by utilizing person-level attributes for group-level re-identification. We introduce PAG-ReID (Person Attribute based Group Re-identification), a large-scale dataset constructed by combining three popular G-ReID datasets: CM-Group, Road Group, and CUHK-SYSU-Group. PAG-ReID includes 19K group images encompassing 2,148 groups and 5,504 unique person IDs. At the person level, it provides 25K individual images, each annotated with 19 diverse human attributes, resulting in nearly 475K fine-grained annotations. Next, we propose an effective baseline (CLIP-based) that transforms attribute information into natural language descriptions, enabling joint multi-modal (visual-textual) reasoning for PAR as well as G-ReID tasks. Experiments demonstrate the effectiveness of our approach, setting a new direction for person attribute-centric group re-identification. To our knowledge, this is the first work to unify G-ReID and PAR in a single multi-modal framework. PAG-ReID can be found at https://github.com/draxler1/PAG-ReID.
The burgeoning growth of open-source vision-language models (VLMs) has catalyzed a plethora of applications across diverse domains. Ensuring the transparency and interpretability of these models is critical for fostering trustworthy and responsible AI systems. In this study, our objective is to delve into the internals of VLMs to interpret the functions of individual neurons. We observe the activations of neurons with respects to the input visual tokens and text tokens, and reveal some interesting findings. Particularly, we found that there are neurons responsible for only visual or text information, or both, respectively, which we refer to them as visual neurons, text neurons, and multi-modal neurons, respectively. We build a framework that automates the explanation of neurons with the assistant of GPT-4o. Meanwhile, for visual neurons, we propose an activation simulator to assess the reliability of the explanations for visual neurons. System statistical analyses on top of one representative VLM of LLaVA, uncover the behaviors/characteristics of different categories of neurons.
Existing methods for video answer localization (VAL) in instructional video focus predominantly on coarse-grained themes, failing to address detailed step content and inter-step relation crucial for effective comprehension. Current datasets, such as MedVidQA, primarily capture video content but lack annotations for step structure and inter-step relation. To address this gap, we introduce InstructStep, a newly proposed VAL task, specifically designed for Instructional Video Step Content and Relation Localization. It extends original VAL task to step-centric content and relation. Accordingly, we create a InstructStep Dataset with fine-grained step content and relation QA pairs. To tackle the challenges of this task, we propose a Step-Centric Multi-Level Knowledge Distillation (SC-MLKD) approach that: (1) A two-stage training strategy that generates step-specific summaries in the first stage and introduces a step branch in the second stage to learn step relations. This is applied only during training, ensuring no additional inference time. (2) Multi-level knowledge distillation, including feature, relation, and response distillation, across visual, text and step branches to capture fine-grained and step-centric features. Comprehensive experiments demonstrate the efficiency of SC-MLKD, with notable gains of up to 5.83% in step content and up to 5.9% in step relation. The dataset have been made publicly available on https://github.com/hewangsh/InstructStep
Semi-supervised learning (SSL) has achieved significant progress by leveraging both labeled data and unlabeled data. Existing SSL methods overlook a common real-world scenario when labeled data is extremely scarce, potentially as limited as a single labeled sample in the dataset. General SSL approaches struggle to train effectively from scratch under such constraints, while methods utilizing pre-trained models often fail to find an optimal balance between leveraging limited labeled data and abundant unlabeled data. To address this challenge, we propose Firstly Adapt, Then catEgorize (FATE), a novel SSL framework tailored for scenarios with extremely limited labeled data. At its core, the two-stage prompt tuning paradigm FATE exploits unlabeled data to compensate for scarce supervision signals, then transfers to downstream tasks. Concretely, FATE first adapts a pre-trained model to the feature distribution of downstream data using volumes of unlabeled samples in an unsupervised manner. It then applies an SSL method specifically designed for pre-trained models to complete the final classification task. FATE is designed to be compatible with both vision and vision-language pre-trained models. Extensive experiments demonstrate that FATE effectively mitigates challenges arising from the scarcity of labeled samples in SSL, achieving an average performance improvement of 33.74% across seven benchmarks compared to state-of-the-art SSL methods. Code is available at https://github.com/ganchi-huanggua/FATE.git.
MIHBench: Benchmarking and Mitigating Multi-Image Hallucinations in Multimodal Large Language Models
PDF ↗Despite growing interest in hallucination in Multimodal Large Language Models (MLLMs), existing studies primarily focus on single-image settings, leaving hallucination in multi-image scenarios largely unexplored. To address this gap, we conduct the first systematic study of hallucinations in multi-image MLLMs and propose MIHBench, a benchmark specifically tailored for evaluating object-related hallucinations across multiple images. MIHBench comprises three core tasks-Multi-Image Object Existence Hallucination, Multi-Image Object Count Hallucination, and Object Identity Consistency Hallucination-targeting semantic understanding across object existence, quantity reasoning, and cross-view identity consistency. Through extensive evaluation, we identify key factors associated with the occurrence of multi-image hallucinations, including: (1) a progressive relationship between the number of image inputs and the likelihood of hallucination occurrences; (2) a strong correlation between single-image hallucination tendencies and those observed in multi-image contexts; and (3) the influence of same object image ratios and the positional placement of negative samples within image sequences on the occurrence of object identity consistency hallucination. To address these challenges, we propose a Dynamic Attention Balancing (DAB) mechanism that adjusts inter-image attention distributions while preserving the overall visual attention proportion. Experiments across multiple state-of-the-art MLLMs demonstrate that our method effectively reduces hallucination occurrences and enhances semantic integration and reasoning stability in multi-image scenarios.
Existing medical vision-language contrastive pretraining methods aim to bring the paired image-report embeddings close together while pushing the unpaired ones apart. However, medical images often exhibit high inter-class visual similarity with only subtle differences, leading to the presence of hard negative samples that are semantically distinct from the anchor but incorrectly close to it in the embedding space, making it challenging to distinguish semantically dissimilar samples. Previous methods consider only the embedding similarity between samples to identify hard negatives, often wrongly treating false negatives as hard negatives. To address this issue, we design a simple yet effective approach called Semantic-Aware Hard Negative mining (SAHN), distinguishing hard negatives from false negatives and encouraging the model to pay greater attention to hard negatives. Specifically, hard negatives are identified as samples with high embedding similarity but low semantic similarity to the anchor and assigned greater importance weights. By integrating these importance weights into the InfoNCE loss, SAHN enhances the model's ability to separate semantically dissimilar samples while clustering semantically similar ones. We further conduct a gradient-based theoretical analysis to validate the effectiveness of SAHN. Extensive experimental results on four downstream medical tasks covering image classification, object detection, semantic segmentation, and cross-modal retrieval demonstrate the superiority of our approach.
Dense video captioning aims to generate descriptive sentences for each temporally localized event in a video. This task comprises two subtasks: event detection and event captioning. Existing methods commonly adopt a DETR-like (Detection Transformer) architecture to perform both subtasks in parallel. These methods assume that both subtasks require the same visual information and thus extract a single event representation for each event using a shared query. We observe that event detection and event captioning emphasize different regions of a video. In particular, compared to event captioning, event detection tends to focus more on the boundary regions of event proposals. Therefore, relying on shared queries may hinder the ability of the model to meet the specific needs of each subtask, leading to suboptimal performance. In this paper, we propose decoupling the two subtasks by assigning distinct queries to each, enabling more accurate capture of task-specific features. Specifically, we introduce a task-specific query transformation module. This module utilizes two sets of task-specific prompts to transform shared queries into queries tailored for each subtask. These task-specific queries enable each subtask to attend to the video regions that are most beneficial to its respective objectives. By integrating our method into several state-of-the-art frameworks, we achieve superior performance on both event detection and event captioning.
Over the past few decades, Internet video streaming has seen explosive growth, pushing network resources to their limits. Video Super-Resolution (VSR) technology, which enhances video quality while reducing bandwidth usage, offers a promising solution to replace traditional video delivery frameworks. However, existing real-world VSR approaches often struggle when faced with inevitable packet loss during network transmission, especially in bandwidth-constrained and low-latency environments. This packet loss introduces amplified noise and artifacts, significantly degrading visual quality. In this work, we decouple the various degrees of degradation caused by packet loss and comprehensively analyze the impact of different types of packet loss. To address these challenges, we propose ReinVSR, an efficient countermeasure strategy that mitigates the detrimental effects of packet loss without introducing additional network overhead. ReinVSR employs a two-pronged approach: a pre-restore module to mitigate missing pixel information and a Local Hidden State Attention module to rectify semantic distortions at the feature level by replacing corrupted hidden states with more accurate representations. Specifically, we leverage neighboring frames to generate a pool of hidden features, which are then refined using a novel spatial attention mechanism to aggregate more authentic and accurate hidden states. Extensive experiments demonstrate that ReinVSR outperforms state-of-the-art methods, achieving significant improvements in visual quality. It offers a robust and effective solution for high-quality video streaming in bandwidth-limited environments.
In few-shot medical image segmentation, most existing methods focus heavily on learning explicit correlations between support and query sets, often overlooking the core demands of the segmentation task itself. In this work, we identify three overlooked yet critical issues that limit current performance: the diversity of background distributions, the degradation of support prototypes, and the over-activation of irrelevant regions. To address these challenges, we propose a novel framework with three lightweight and adaptive modules. First, a background self-distillation module acts as a self-attention-driven agent to cluster and aggregate diverse background features, generating multiple sub-prototypes that enhance foreground-background separation. Second, we introduce a prototype self-anchoring mechanism that leverages a dual-branch correlation mapping and reverse supervision to stabilize support prototype learning and prevent feature degradation. Third, an activation self-calibration module identifies over-activated residuals and applies test-time channel manipulation to suppress noisy activations without additional training. Extensive experiments on standard few-shot medical segmentation benchmarks demonstrate the superiority of our approach over state-of-the-art methods. Our findings suggest that performance gains come not only from better support-query alignment, but also from rethinking and addressing the often neglected aspects of few-shot segmentation.
3D Visual Grounding (3DVG) aims to localize objects in 3D scenes using natural language descriptions. Although supervised methods achieve higher accuracy in constrained settings, zero-shot 3DVG holds greater promise for real-world applications since eliminating scene-specific training requirements. However, existing zero-shot methods face challenges of spatial-limited reasoning due to reliance on single-view localization, and contextual omissions or detail degradation. To address these issues, we propose SeqVLM, a novel zero-shot 3DVG framework that leverages multi-view real-world scene images with spatial information for target object reasoning. Specifically, SeqVLM first generates 3D instance proposals via a 3D semantic segmentation network and refines them through semantic filtering, retaining only semantic-relevant candidates. A proposal-guided multi-view projection strategy then projects these candidate proposals onto real scene image sequences, preserving spatial relationships and contextual details in the conversion process of 3D point cloud to images. Furthermore, to mitigate VLM computational overload, we implement a dynamic scheduling mechanism that iteratively processes sequances-query prompts, leveraging VLM's cross-modal reasoning capabilities to identify textually specified objects. Experiments on the ScanRefer and Nr3D benchmarks demonstrate state-of-the-art performance, achieving Acc@0.25 scores of 55.6% and 53.2%, surpassing previous zero-shot methods by 4.0% and 5.2%, respectively, which advance 3DVG toward greater generalization and real-world applicability.
Low-resolution (LR) depth maps captured by depth sensors often suffer from structural distortions, noise, and blurring, limiting their practical usability. While most existing depth super-resolution (DSR) methods rely on synthetic datasets, they fail to accurately model real-world degradations, leading to poor performance on real-world data. To address this limitation, we identify two key challenges in the real-world DSR task: structural contour inconsistency and regional degradation inconsistency. The former arises from structural distortions in LR depth maps, while the latter stems from varying degradation levels in smooth regions. Upon this, we propose a Semantics-Driven Contrastive Learning (SDCL) pipeline for real-world DSR, leveraging semantic priors from the SAM model to enhance structural contour reconstruction and region-wise degradation handling. We introduce two novel contrastive loss functions: Structural Contour Alignment (SCA) loss, which aligns depth contours with semantic boundaries, and Regional Degradation Discrimination (RDD) loss, which optimizes smooth region restoration through region-level contrastive learning. Our approach is model-agnostic and can be seamlessly integrated into existing DSR frameworks. Experiments demonstrate that our method significantly enhances DSR performance on real-world DSR datasets.