Large Video Models (LVMs) build on the semantic capabilities of Large Language Models (LLMs) and vision modules by integrating temporal information to better understand dynamic video content. Despite their progress, LVMs are prone to hallucinations-producing inaccurate or irrelevant descriptions. Current benchmarks for video hallucination depend heavily on manual categorization of video content, neglecting the perception-based processes through which humans naturally interpret videos. We introduce MESH, a benchmark designed to evaluate hallucinations in LVMs systematically. MESH uses a Question-Answering framework with binary and multi-choice formats incorporating target and trap instances. It follows a bottom-up approach, evaluating basic objects, coarse-to-fine subject features, and subject-action pairs, aligning with human video understanding. We demonstrate that MESH offers an effective and comprehensive approach for identifying hallucinations in video understanding. Our evaluations show that while LVMs excel at recognizing basic objects and features, their susceptibility to hallucinations increases markedly when handling fine details or aligning multiple actions involving various subjects in longer videos. The benchmark is available at MESH-Benchmark.
论文检索
输入标题、作者或关键词，从 1,620 篇学术成果中精准定位
With the rapid advancement of Multi-modal Large Language Models (MLLMs), their capability in understanding both images and text has greatly improved. However, their potential for leveraging multi-modal contextual information in Retrieval-Augmented Generation (RAG) remains largely underexplored. To address this gap, this paper introduces Multi-Modal Retrieval-Augmented Generation (M2RAG), a benchmark designed to evaluate the effectiveness of Multi-modal Large Language Models in leveraging knowledge from multi-modal retrieval documents. The benchmark comprises four tasks: image captioning, multi-modal question answering, multi-modal fact verification, and image reranking. All tasks are set in an open-domain setting, requiring RAG models to retrieve query-relevant information from a multi-modal document collection and use it as contextual input for RAG modeling. To enhance the context utilization capabilities of MLLMs, we also introduce Multi-Modal Retrieval-Augmented Instruction Tuning (MM-RAIT), an instruction tuning method that optimizes MLLMs within multi-modal contexts. Our experiments demonstrate the effectiveness of MM-RAIT by significantly improving the quality of responses generated by different RAG models, outperforming MiniCPM-V 2.6 and Qwen2-VL with 34% and 33% gains, respectively. All data and code are available at https://github.com/NEUIR/M2RAG.
Defending pre-trained vision-language models (VLMs), such as CLIP, against adversarial attacks is crucial, as these models are widely used in diverse zero-shot tasks, including image classification. However, existing adversarial training (AT) methods for robust fine-tuning largely overlook the role of language in enhancing visual robustness. Specifically, (1) supervised AT methods rely on short texts (e.g., class labels) to generate adversarial perturbations, leading to overfitting to object classes in the training data, and (2) unsupervised AT avoids this overfitting but remains suboptimal against practical text-guided adversarial attacks due to its lack of semantic guidance. To address these limitations, we propose Quality Text-guided Adversarial Fine-Tuning (QT-AFT), which leverages high-quality captions during training to guide adversarial examples away from diverse semantics present in images. This enables the visual encoder to robustly recognize a broader range of image features even under adversarial noise, thereby enhancing robustness across diverse downstream tasks. QT-AFT overcomes the key weaknesses of prior methods---overfitting in supervised AT and lack of semantic awareness in unsupervised AT---achieving state-of-the-art zero-shot adversarial robustness and clean accuracy, evaluated across 16 zero-shot datasets. Furthermore, our comprehensive study uncovers several key insights into the role of language in enhancing vision robustness; for example, describing object properties in addition to object names further enhances zero-shot robustness. Our findings point to an urgent direction for future work---centering high-quality linguistic supervision in robust visual representation learning.
Face deepfake detection is a critical technology for verifying the authenticity of facial media content and has long been a focal point in multimedia forensics. However, existing methods face significant challenges, primarily due to their limited ability to generalize across domains. Consequently, the growing variety of forgery techniques, combined with the degradation of visual quality in forged images, makes reliable detection even more difficult. To address these challenges, we propose WKD, a proactive deepfake detection framework based on Watermarking and Knowledge Distillation. The key insights of WKD are twofold: First, we embed watermark information into the Fractional-order Quaternion Radial Harmonic Fourier Moments (FrQRHFMs) space of the host image, achieving a robust balance between imperceptibility and robustness. Second, we design a dual-task learning framework consisting of a watermark extractor and a forgery discriminator, where learnable Low-Rank Adaptation (LoRA) layers are used to transfer knowledge from the extractor to the discriminator, thereby providing additional clues for deepfake detection. Specifically, the integrity of the watermark is compromised only when the host image undergoes a deepfake forgery, while it remains unaffected by conventional attacks. Experimental results on benchmark datasets demonstrate that WKD achieves state-of-the-art performance in both intra-domain and cross-domain deepfake detection, particularly when images are subjected to various conventional attacks.
Vision-language models (VLMs) exhibit remarkable zero-shot capabilities but struggle with distribution shifts in downstream tasks when labeled data is unavailable, which has motivated the development of Test-Time Adaptation (TTA) to improve VLMs' performance during inference without annotations. Among various TTA approaches, cache-based methods show promise by preserving historical knowledge from low-entropy samples in a dynamic cache and fostering efficient adaptation. However, these methods face two critical reliability challenges: (1) entropy often becomes unreliable under distribution shifts, causing error accumulation in the cache and degradation in adaptation performance; (2) the final predictions may be unreliable due to inflexible decision boundaries that fail to accommodate large downstream shifts. To address these challenges, we propose a Reliable Test-time Adaptation (ReTA) method that integrates two complementary strategies to enhance reliability from two perspectives. First, to mitigate the unreliability of entropy as a sample selection criterion for cache construction, we introduce Consistency-aware Entropy Reweighting (CER), which incorporates consistency constraints to weight entropy during cache updating. While conventional approaches rely solely on low entropy for cache prioritization and risk introducing noise, our method leverages predictive consistency to maintain a high-quality cache and facilitate more robust adaptation. Second, we present Diversity-driven Distribution Calibration (DDC), which models class-wise text embeddings as multivariate Gaussian distributions, enabling adaptive decision boundaries for more accurate predictions across visually diverse content. Extensive experiments demonstrate that ReTA consistently outperforms state-of-the-art methods, particularly under real-world distribution shifts.
Large Vision-Language Models (LVLMs) usually generate texts which satisfy context coherence but don't match the visual input. Such a hallucination issue hinders LVLMs' applicability in the real world. The key to solving hallucination in LVLM is to make the text generation rely more on the visual content. Most previous works choose to enhance/adjust the features/output of a specific modality (i.e., visual or textual) to alleviate hallucinations in LVLM, which do not explicitly or systematically enhance the visual reliance. In this paper, we comprehensively investigate the factors that may degenerate the visual reliance in text generation of LVLM from a Bayesian perspective. We propose to mitigate hallucination in LVLM from three aspects. Firstly, we observe that not all visual tokens are informative in generating meaningful texts. We propose to evaluate and remove redundant visual tokens to avoid their disturbance. Secondly, LVLM may encode inappropriate prior information, making it lean toward generating unexpected words. We propose a simple, yet effective way to rectify the prior from a Bayesian perspective. Thirdly, we observe that starting from certain steps, the posterior of next-token prediction conditioned on visual tokens may collapse to a prior distribution which does not depend on any informative visual tokens at all. Thus, we propose to stop further text generation to avoid hallucination. Extensive experiments on three benchmarks, including POPE, CHAIR, and MME, demonstrate that our method can consistently mitigate the hallucination issue of LVLM and performs favorably against previous state-of-the-arts. Codes are available at https://github.com/NeilHnxTcc/EVRB.
The rapid evolution of deepfake generation technologies necessitates the development of robust face forgery detection algorithms. Recent studies have demonstrated that wavelet analysis can enhance the generalization abilities of forgery detectors. Wavelets effectively capture key facial contours, often slender, fine-grained, and globally distributed, that may conceal subtle forgery artifacts imperceptible in the spatial domain. However, current wavelet-based approaches fail to fully exploit the distinctive properties of wavelet data, resulting in sub-optimal feature extraction and limited performance gains. To address this challenge, we introduce WMamba, a novel wavelet-based feature extractor built upon the Mamba architecture. WMamba maximizes the utility of wavelet information through two key innovations. First, we propose Dynamic Contour Convolution (DCConv), which employs specially crafted deformable kernels to adaptively model slender facial contours. Second, by leveraging the Mamba architecture, our method captures long-range spatial relationships with linear complexity. This efficiency allows for the extraction of fine-grained, globally distributed forgery artifacts from small image patches. Extensive experiments show that WMamba achieves state-of-the-art (SOTA) performance, highlighting its effectiveness in face forgery detection.
Multi-task learning (MTL) for dense prediction has shown promising results but still faces challenges in balancing shared representations with task-specific specialization. In this paper, we introduce a novel Fine-Grained Mixture of Experts (FGMoE) architecture that explores MoE-based MTL models through a combination of three key innovations and fine-tuning. First, we propose intra-task experts that partition along intermediate hidden dimensions of MLPs, enabling finer decomposition of task information while maintaining parameter efficiency. Second, we introduce shared experts that consolidate common information across different contexts of the same task, reducing redundancy, and allowing routing experts to focus on unique aspects. Third, we design a global expert that facilitates adaptive knowledge transfer across tasks based on both input feature and task requirements, promoting beneficial information sharing while preventing harmful interference. In addition, we use the fine-tuning approach to improve parameter efficiency only by training the parameters of the decoder. Extensive experimental results show that the proposed FGMoE uses fewer parameters and significantly outperforms current MoE-based competitive MTL models on two dense prediction datasets (i.e., NYUD-v2, PASCAL-Context) in various metrics.
Cross-modal retrieval refers to identifying semantically relevant data across different modalities. However, annotation errors or inherent ambiguity can cause semantic inconsistency in sample pairs, degrading retrieval performance. Prior efforts either relied heavily on the quality of explicitly dividing clean and noisy subsets, or solely leveraged carefully selected single anchor information, neglecting relationships among diverse neighbors. In this paper, we propose a novel Graph-based Label Propagation (GLP) framework that learns pseudo-labels via label propagation on a sparse graph, enabling self-correction of noisy labels. Specifically, each modality's instances are treated as nodes, connected via k-nearest neighbor (kNN) search to form a sparse graph. Pseudo-label vectors are generated for all nodes within one modality to capture the matching degree of inter-modal nodes. Through iterative label propagation, the stabilized pseudo-labels implicitly exploit both intra- and inter-modal relationships to derive a reliable matching degree. A dynamic queue further enhances graph quality by updating high-quality nodes. Experiments on Flickr30K, MSCOCO, and CC120K show that our method outperforms state-of-the-art approaches, especially under high noise. Code is available at https://github.com/njustkmg/MM25-GLP.
Accurate 3D human mesh recovery from point clouds remains challenging. Most existing methods depend on full 3D supervision or complete input data, both of which are difficult to obtain in practice. %cr update This calls for robust solutions capable of handling partial point clouds in a self-supervised manner. However, the incompleteness of point clouds and the absence of supervision signals pose dual challenges. To tackle these challenges, This calls for robust solutions to handle partial point clouds in a self-supervised manner. To tackle the dual challenges of point cloud incompleteness and the absence of supervision signals, we propose a novel method named SS-HMR, which offers three key insights. First, we estimate point-wise semantics in a self-supervised manner to match partial inputs with a canonical template. The resulting correspondences serve as supervision signals for the regression network in human mesh recovery. Second, we incorporate regression-based and optimization-based paradigms into a self-improving loop: the regression network provides strong initialization for optimization, while the optimization routine generates pseudo-labels that, in turn, enhance the regression network. This mutual feedback enables more accurate and stable mesh recovery over time. Third, generating multiple initializations and selecting the best result mitigates the optimization routine's sensitivity to initialization, improving robustness to sparse and noisy data. %cr update Third, to mitigate sensitivity to initialization in the optimization routine, we generate diverse initialization candidates and transform the challenge of escaping local optima into a controllable selection task, improving robustness against sparse and noisy data. Extensive experiments are conducted on three public datasets and results demonstrate that SS-HMR outperforms existing methods. Notably, SS-HMR performs excellently on different test data, whether from original point clouds captured by depth cameras or LiDAR devices, or from noise-added ones. This shows that SS-HMR has strong generalization ability and robustness across different data sources. Codes are available at https://github.com/suchang-99/SS-HMR.
Deep learning-based medical image segmentation, with its precise lesion localization capabilities, serves as a core component in multimedia medical applications and intelligent diagnostic assistance systems. Recent innovations in network architectures significantly improve segmentation performance. However, the Non-Effective Samples (NES) on model optimization receive little attention. These samples are characterized by minimal gradient variations in loss during training and exhibit a slight contribution to model optimization. They encompass well-segmented samples with near-zero loss values and challenging samples with consistently high loss values. Especially, when NES accumulate, the model will fall into an optimization trap, causing the optimization to stagnate. To address this issue, we propose a lightweight plug-and-play Gradient-Aware Sample Selection and Reactivation Strategy (GA-SRS) that efficiently identifies and revitalizes the training potential of NES. Firstly, GA-SRS filters NES out based on the historical training information of samples and the variations in the loss gradient during the training process. Then, GA-SRS revitalizes the training values of these samples through strong data augmentation. Extensive experiments on four public datasets and three general models demonstrate the effectiveness of GA-SRS. For example, GA-SRS helps improve the IoU metric of U-KAN from 67.20% to 71.24% on the BUSI and from 81.20% to 82.51% on the ISIC dataset, achieving state-of-the-art experimental results.
Object state recognition aims to identify the specific condition of objects, such as their positional states (e.g., open or closed) and functional states (e.g., on or off). While recent Vision-Language Models (VLMs) are capable of performing a variety of multimodal tasks, it remains unclear how precisely they can identify object states. To alleviate this issue, we introduce the STAte and Transition UnderStanding Benchmark (STATUS Bench), the first benchmark for rigorously evaluating the ability of VLMs to understand subtle variations in object states in diverse situations. Specifically, STATUS Bench introduces a novel evaluation scheme that requires VLMs to perform three tasks simultaneously: object state identification (OSI), image retrieval (IR), and state change identification (SCI). These tasks are defined over our fully hand-crafted dataset involving image pairs, their corresponding object state descriptions and state change descriptions. Furthermore, we introduce a large-scale training dataset, namely STATUS Train, which consists of 13 million semi-automatically created descriptions. This dataset serves as the largest resource to facilitate further research in this area. In our experiments, we demonstrate that STATUS Bench enables rigorous consistency evaluation and reveal that current state-of-the-art VLMs still significantly struggle to capture subtle object state distinctions. Surprisingly, under the proposed rigorous evaluation scheme, most open-weight VLMs exhibited chance-level zero-shot performance. After fine-tuning on STATUS Train, Qwen2.5-VL achieved performance comparable to Gemini 2.0 Flash. These findings underscore the necessity of STATUS Bench and Train for advancing object state recognition in VLM research.
Open-Vocabulary Multi-Label Recognition (OV-MLR) aims to identify multiple seen and unseen object categories within an image, requiring both precise intra-class localization to pinpoint objects and effective inter-class reasoning to model complex category dependencies. While Vision-Language Pre-training (VLP) models offer a strong open-vocabulary foundation, they often struggle with fine-grained localization under weak supervision and typically fail to explicitly leverage structured relational knowledge beyond basic semantics, limiting performance especially for unseen classes. To overcome these limitations, we propose the Dual Adaptive Refinement Transfer (DART) framework. DART enhances a frozen VLP backbone via two synergistic adaptive modules. For intra-class refinement, an Adaptive Refinement Module (ARM) refines patch features adaptively, coupled with a novel Weakly Supervised Patch Selecting (WPS) loss that enables discriminative localization using only image-level labels. Concurrently, for inter-class transfer, an Adaptive Transfer Module (ATM) leverages a Class Relationship Graph (CRG), constructed using structured knowledge mined from a Large Language Model (LLM), and employs graph attention network to adaptively transfer relational information between class representations. DART is the first framework, to our knowledge, to explicitly integrate external LLM-derived relational knowledge for adaptive inter-class transfer while simultaneously performing adaptive intra-class refinement under weak supervision for OV-MLR. Extensive experiments on challenging benchmarks demonstrate that our DART achieves new state-of-the-art performance, validating its effectiveness.
EventFormer: A Node-graph Hierarchical Attention Transformer for Action-centric Video Event Prediction
PDF ↗Script event induction, which aims to predict the subsequent event based on the context, is a challenging task in NLP, achieving remarkable success in practical applications. However, human events are mostly recorded and presented in the form of videos rather than scripts, yet there is a lack of related research in the realm of vision. To address this problem, we introduce AVEP (Action-centric Video Event Prediction), a task that distinguishes itself from existing video prediction tasks through its incorporation of more complex logic and richer semantic information. We present a large structured dataset, which consists of about 35K annotated videos and more than 178K video clips of event, built upon existing video event datasets to support this task. The dataset offers more fine-grained annotations, where the atomic unit is represented as a multimodal event argument node, providing better structured representations of video events. Due to the complexity of event structures, traditional visual models that take patches or frames as input are not well-suited for AVEP. We propose EventFormer, a node-graph hierarchical attention based video event prediction model, which can capture both the relationships between events and their arguments and the coreferencial relationships between arguments. We conducted experiments using several SOTA video prediction models as well as LVLMs on AVEP, demonstrating both the complexity of the task and the value of the dataset. Our approach outperforms all these video prediction models. We will release the dataset and code for replicating the experiments and annotations.
Class incremental medical segmentation (CIMS) aims to sequentially learn new classes while preserving knowledge of previously learned categories in the absence of old-class labels. Current methods suffer from performance degradation under class imbalance and require additional segmentation heads to accommodate new categories. Inspired by recent prototype learning that leverages prototypes to achieve robust recognition of new categories under limited-data regimes, we introduce a Prototype-dRIven class increMEntal (PRIME) method. PRIME replaces the incremental segmentation heads with prototypes to mitigate class imbalance, allowing new class learning with the simple addition of new prototypes. Based on prototype learning, PRIME further involves three tailored techniques. First, prototype structure alignment imposes structural constraints on inter-prototype relations to maintain consistent relative distances in the feature space, improving the model's ability to distinguish distinct classes. Second, pixel-wise contrastive loss term groups embeddings of similar samples while separating those of different classes, enhancing segmentation accuracy across all categories. Finally, the consensus-based prototype update mechanism refines the old prototypes during the learning of new classes, preventing performance degradation on the old classes. Extensive experiments on two public multi-organ segmentation datasets demonstrate that our approach significantly outperforms state-of-the-art methods, validating the effectiveness of the proposed PRIME.
Advances in automatic graphical user interface (GUI) agents have brought significant privacy and security challenges, especially cloud-based solutions that may leak sensitive data and be vulnerable to man-in-the-middle attacks. To address these issues, we propose MMPro, a novel GUI agent framework that adopts a separated perception-thinking-execution architecture. The perception module and the execution module process inputs locally to generate outputs to ensure security, and the thinking module operates on abstract representations to ensure the effectiveness of the GUI agent.By modularizing each stage while introducing a hybrid description language (HDL), MMPro transforms screen images into abstract structured representations, minimizing the risk of sensitive information leakage. Experimental results on the OSWorld benchmark show that MMPro outperforms existing GUI agent methods while ensuring strong privacy protection and real-time interaction capabilities. This work presents a pioneering approach to developing efficient, privacy-conscious, and explainable automated GUI agents.
Recently, LiDAR-based fully sparse 3D object detection has gained great attention, which utilizes point clouds to boost efficiency. Nevertheless, the relationship between well-studied dense representation and fully sparse representation is under-explored in existing studies, which focuses solely on building sparse representation by feature diffusion to solve the notorious center point missing problem. To this end, we propose a dense-guided fully sparse detection scheme, named DGFSD, to bridge the gap between dense and sparse features by dense-guided diffusion. Different from prior studies, we propose DgD (Dense-guided Diffusion) to overcome the center feature missing problem by dense knowledge transferring. Specifically, DgD transfers high-quality central point features from dense representations to endow sparse representations with dense knowledge. Moreover, we customize DFW (Dense Feature Weighting) to express uninformative representation and lift foreground representation. It makes high-quality dense feature contribute more to arcuate regression. To the best of our knowledge, we are the first to explore dense knowledge's impact on fully sparse framework. Extensive experiments conducted on nuScenes and Argoverse2 benchmark demonstrate the effectiveness of the proposed method. Specifically, DGFSD achieves 71.6% NDS and 67.3% mAP on the nuScenes test benchmark. On Argoverse2, DGFSD achieves 40.6% mAP, outperforming previous best hybrid and fully sparse methods. The code is available at https://github.com/Raiden-cn/DGFSD.
Event-driven eye-based emotion recognition has attracted increasing attention due to the high temporal resolution and dynamic range inherent to event cameras. The intrinsic spatial sparsity of event data, combined with the eye-based emotion recognition task's reliance on localized features such as eyebrows and eyelids, makes it intuitive and efficient to discard less informative regions. However, integrating such sparsification into CNNs remains challenging due to their reliance on dense grid-based operations. In this paper, we propose an efficient vision transformer framework for eye-based emotion recognition with event cameras. Specifically, we present window selection and token selection schemes tailored for event data and eye-based emotion recognition, which can diminish computing demands while enhancing performance. Firstly, we estimate the importance of all local windows and discard those with limited information, reducing computational cost while emphasizing attention on the periocular region. Secondly, we further introduce an adaptive token pruning mechanism that jointly evaluates the input event data and tokens to predict a binary decision mask, identifying and discarding uninformative tokens. Extensive experiments validate that the proposed approach outperforms existing state-of-the-art methods in accuracy by a significant margin.
Test-time adaptation (TTA) has emerged as a promising solution to improve the robustness of deep learning models under domain shifts, particularly in real-world scenarios where source data or true labels are unavailable. In this paper, we propose a novel TTA framework tailored for medical vision-language models (VLMs) that leverages a Mixture-of-Experts (MoE) mechanism. Building upon a frozen, pre-trained BiomedCLIP backbone, our method integrates parallel MoE adapters of different medical imaging modalities in each vision MLP block, enabling expert-specific adaptation without disrupting the core model representation. At inference time, only the MoE router is optimized through an entropy-regularized objective, which is further augmented by pseudo-label guidance and adaptive scaling strategies. Additionally, we propose an entropy-aware MoE scaling policy that dynamically adjusts expert influence based on prediction uncertainty, improving model adaptability. Extensive experiments on multiple medical imaging benchmarks demonstrate that our approach achieves substantial performance improvements over existing TTA baselines, while maintaining high efficiency and parameter sparsity. Our results highlight the potential of MoE-enhanced TTA to achieve robust and generalizable medical VLMs in unseen domains without access to source data. Code is available at https://openi.pcl.ac.cn/OpenMedIA/MoME-TTA.git
Long-form video understanding (LVU) addresses the challenge of answering complex questions over extended video length, where informative cues are sparse and easily overwhelmed by redundant content. To tackle this, it requires selecting a small set of question-relevant keyframes and reasoning over long-range, temporally dispersed visual evidence. However, current methods typically extract frame-level features with limited temporal context and store them in sequential memory structures. As a result, they struggle to capture the evolving relations among entities and fail to maintain identity consistency when entities temporarily leave and later reappear in the video. These limitations prevent accurate keyframe localization and coherent reasoning. In this paper, we propose GraphVideoAgent, a novel agent-based LVU framework that integrates a dynamic entity relation graph with a large language model (LLM)-based multi-round reasoning. Our framework emulates human cognitive strategies by iteratively retrieving keyframes and explicitly tracking both temporal and semantic interactions among entities. Our GraphVideoAgent iteratively reflects on question cues and visual observations, while the graph memory maintains a structured representation of evolving entity states and their causal relations. This design enables accurate keyframe selection, effective reasoning over sparse visual evidence, and interpretable prediction. Extensive experiments on two LVU benchmarks, EgoSchema and NExT-QA, demonstrate that GraphVideoAgent achieves state-of-the-art performance while using only 8.2 and 8.1 frames on average, significantly improving both accuracy and efficiency.