Multi-modal knowledge graph reasoning (MKGR) seeks to conjecture plausible facts in MKGs by learning effective representations from various modalities (e.g., structure, text, and image). However, due to holistic redundancy (i.e., each modality carries task-irrelevant redundancy) and modality conflict (i.e., different modalities contain contradictory information), the reasoning performance of current methods is substantially impaired. In this paper, we propose a novel Consistency Discovery-guided Information Bottleneck (CDIB) framework to address the aforementioned challenges. Specifically, a modality compression module is first designed to learn modality-private entity representations of alleviating redundant information. Then, a consistency discovery module is developed to discover cross-modal consistency during multi-modal fusion to learn the comprehensive entity representations. To retain task-relevant information, an information preservation module is devised to further enrich the comprehensive entity representations to be predictive for MKGR. Extensive experiments indicate that CDIB achieves state-of-the-art reasoning ability on two benchmark datasets over current MKGR baselines, and also exhibits promising robustness against noise.
论文检索
输入标题、作者或关键词，从 12,319 篇学术成果中精准定位
Due to the frequent occurrence of missing views in real-world multi-view data, incomplete multi-view clustering (IMVC) has attracted significant attention. However, most existing IMVC methods overlook the fact that incomplete data in practical applications often exhibits varying missing rates across different views, rendering their mechanisms ineffective under such conditions. Although several works based on conventional learning methods have been proposed to solve unbalanced incomplete multi-view clustering (UIMVC), their performance is limited by their shallow feature representation and over-sophisticated optimization procedure. In this paper, we propose Deep Unbalanced Incomplete Multi-view Clustering via Graph Constrained Imputation and Contrastive Learning (DUIMC) to address UIMVC with deep learning paradigm. Specifically, DUIMC introduces a novel differentiable imputation layer for dynamically handling unbalanced incompleteness and integrates it with multi-view contrastive clustering into a unified deep representation learning framework. Furthermore, bi-level graph constraints are imposed on imputation and representation learning to preserve local consistency at both the feature and instance levels. In addition, we develop adaptive fusion mechanisms to adaptively restrain the impact aroused by information unbalance among views. Extensive experimental results on five benchmark datasets demonstrate DUIMC's superior clustering performance over several traditional state-of-the-art approaches.
Existing LiDAR-based 3D object detectors typically rely on manually annotated labels for training to achieve good performance. However, obtaining high-quality 3D labels is time-consuming and labor-intensive. To address this issue, recent works explore unsupervised 3D object detection by introducing RGB images as an auxiliary modal to assist pseudo-box generation. However, these methods simply integrate pseudo-boxes generated by LiDAR point clouds and RGB images. Yet, such a label-level fusion strategy brings limited improvements to the quality of pseudo-boxes, as it overlooks the complementary nature in terms of LiDAR and RGB image data. To overcome the above limitations, we propose a novel data-level fusion framework that integrates RGB images and LiDAR data at an early stage. Specifically, we utilize vision foundation models for instance segmentation and depth estimation on images and introduce a bi-directional fusion method, where real points acquire category labels from the 2D space, while 2D pixels are projected onto 3D to enhance real point density. To mitigate noise from depth and segmentation estimations, we propose a local and global filtering method, which applies local radius filtering to suppress depth estimation errors and global statistical filtering to remove segmentation-induced outliers. Furthermore, we propose a data-level fusion based dynamic self-evolution strategy, which iteratively refines pseudo-boxes under a dense representation, significantly improving localization accuracy. Extensive experiments on the nuScenes dataset demonstrate that the detector trained by our method significantly outperforms that trained by previous state-of-the-art methods with 28.4% mAP on the nuScenes validation benchmark.
Technical Element Score (TES) and Program Component Score (PCS) evaluations in figure skating demand precise assessment of athletic actions and artistic interpretation, respectively. Existing methods face three major challenges. Firstly, video and audio cues are regarded as common features for both TES and PCS predictions in previous works without considering the prior evaluation criterion of figure skating. Secondly, action elements in competitions are separated in time, TES should be derived from each element's score, but existing methods try to give an overall TES prediction without evaluating each action element. Thirdly, lengthy competition videos make it difficult and inefficient to handle long-range contexts. To address these challenges, we propose a two-stream Mamba pyramid network that aligns with actual judging criteria to predict TES and PCS by separating visual-feature based TES evaluation stream from audio-visual-feature based PCS evaluation stream. In the PCS evaluation stream, we introduce a multi-level fusion mechanism to guarantee that video-based features remain unaffected when assessing TES, and enhance PCS estimation by fusing visual and auditory cues across each contextual level of the pyramid. In the TES evaluation stream, the multi-scale Mamba pyramid and TES head we proposed effectively address the challenges of localizing and evaluating action elements with various temporal scales and give score predictions. With Mamba's superior ability to capture long-range dependencies and its linear computational complexity, our method is ideal for handling lengthy figure skating videos. Comprehensive experimentation demonstrates that our framework attains state-of-the-art performance on the FineFS benchmark. Furthermore, it yields competitive outcomes on two additional datasets without further training. Our source code is available at https://github.com/ycwfs/Figure-Skating-Action-Quality-Assessment.
Generating accurate descriptions of human actions in videos remains a challenging task for video captioning models. Existing approaches often struggle to capture fine-grained motion details, resulting in vague or semantically inconsistent captions. In this work, we introduce the Motion-Augmented Caption Model (M-ACM), a novel generative framework that enhances caption quality by incorporating motion-aware decoding. At its core, M-ACM leverages motion representations derived from human mesh recovery to explicitly highlight human body dynamics, thereby reducing hallucinations and improving both semantic fidelity and spatial alignment in the generated captions. To support research in this area, we present the Human Motion Insight (HMI) Dataset, comprising 115K video-description pairs focused on human movement, along with HMI-Bench, a dedicated benchmark for evaluating motion-focused video captioning. Experimental results demonstrate that M-ACM significantly outperforms previous methods in accurately describing complex human motions and subtle temporal variations, setting a new standard for motion-centric video captioning.
The spread of tampered text poses a critical challenge to information security. Previous methods for tampered text detection (TTD) primarily relied on visual artifacts as clues, while overlooking potential semantic inconsistencies introduced during text manipulation. To address this limitation, we propose TVSIP (Tampered text Visual-Semantic InterPreter), a novel framework leveraging Multimodal Large Language Models (MLLMs) to integrate both visual and semantic clues for comprehensive tampered text analysis and verification. TVSIP consists of a Locator and an Interpreter. The Locator combines the visual detection ability of existing expert models with the semantic comprehension capabilities of MLLMs to create precise tampering masks. Subsequently, the Interpreter provides comprehensive descriptions and explanations based on identified tampered regions. To train and evaluate TVSIP, we construct the TextDDLE benchmark using GPT-4o. Extensive experiments demonstrate that TVSIP outperforms expert models in pixel-level localization and advanced MLLMs in interpretability. Furthermore, it maintains robustness against image degradation and exhibits strong generalization ability on out-of-domain datasets. Our work highlights the crucial role of semantic inconsistencies in TTD and establishes a more reliable verification system for ensuring document authenticity in the digital age.
The current one-stream tracking framework has received far-reaching attention for its significant improvement in tracking performance, yet it is essentially an extension of Siamese trackers. However, the one-stream framework of discriminative trackers has not been effectively exploited, still using separate feature extraction and model prediction. Therefore, this article aims to implement a one-stream learning strategy for feature extraction and model prediction under the discriminative tracking framework. To this end, we have leveraged the prevailing Vision Transformer and Vision Mamba backbones to achieve our motivation. Moreover, we innovatively combine templates with discriminative tracking methods to enhance the ability of target-aware feature learning, and further propose the attention fusion module to implement spatiotemporal template fusion, which can enhance the adaptability of the tracking model to dynamic changes of targets. The experiments on multiple popular tracking benchmarks have demonstrated that our proposed tracking architecture has superior tracking performance. Concisely, our tracker obtains an AUC of 73.3% on LaSOT dataset, and an AO of 78.2% on GOT-10k dataset. The code, raw results, and trained models are available at https://github.com/hexdjx/VisTrack.
Remote sensing change detection is essential for monitoring urban expansion, disaster assessment, and resource management, offering timely, accurate, and large-scale insights into dynamic landscape transformations. While deep learning has revolutionized change detection, the increasing complexity and computational demands of modern models have not necessarily translated into significant accuracy gains. Instead of following this trend, this study explores a more efficient approach, focusing on lightweight models that maintain high accuracy while minimizing resource consumption, which is an essential requirement for on-satellite processing. To this end, we propose FlickCD, which means quick flick then get great results, pushing the boundaries of the performance-resource trade-off. FlickCD introduces an Enhanced Difference Module (EDM) to amplify critical feature differences between temporal phases while suppressing irrelevant variations such as lighting and weather changes, thereby reducing computational costs in the subsequent change decoder. Additionally, the FlickCD decoder incorporates Local-Global Fusion Blocks, leveraging Shifted Window Self-Attention (SWSA) and Efficient Global Self-Attention (EGSA) to effectively capture semantic information at multiple scales, preserving both coarse- and fine-grained changes. Extensive experiments on four benchmark datasets demonstrate that FlickCD reduces computational and storage overheads by more than an order of magnitude while achieving state-of-the-art (SOTA) performance or incurring only a minor (<1% F1) accuracy trade-off. The implementation code is publicly available at https://github.com/xulsh8/FlickCD.
Auditory attention detection (AAD) aims to identify the attended speaker in multi-talker environments by analyzing brain activity recorded through neural monitoring techniques. Recent AAD approaches have achieved great progress in improving detection accuracy. However, they still face challenges in capturing complex spatio-temporal dependencies and high-order nonlinear relationships across brain regions. To address these challenges, this paper proposes DHGCN, a dual hypergraph convolutional network that integrates a hypergraph modeling module, a dual-branch hypergraph learning (DHGL) module, and a feature fusion module. Specifically, the hypergraph modeling module constructs spatial and temporal hypergraphs from EEG signals, enabling the representation of high-order relationships among channels and time points. The DHGL module comprises two parallel branches: a spatial branch that learns high-order spatial dependencies across EEG channels, and a temporal branch that captures complex temporal dependencies. Each branch uses its corresponding hypergraph structure, which is established during the modeling phase. The feature fusion module then aggregates spatial and temporal representations from both branches to support robust auditory attention classification. Extensive experiments on multiple benchmark datasets demonstrate that DHGCN consistently outperforms state-of-the-art AAD models. It achieves superior classification performance while reducing the trainable parameters count by over 50% compared to the state-of-the-art models. Code is available at: https://github.com/nobody1219/DHGCN.git.
Event-based object detection plays a crucial role in scenarios involving high-speed motion, extreme lighting conditions, and high-frequency detection. However, existing methods fail to address the challenges posed by small objects, including discriminative feature deficiency, the loss of critical information, and the inherent sparsity of event data. Moreover, the lack of benchmark datasets has significantly hindered progress in this field. To tackle these issues, we propose the Fully Deformable Detection Network (FDDNet), a lightweight framework that dynamically adapts to extract key features. First, we introduce a Long-Term Deformable Temporal Receptive Module (LDTR), which aligns critical features across consecutive event streams and leverages a State Space Model for long-range temporal modeling, enhancing the detection of high-speed small objects. Second, to address the sparsity of event data and the concentration of key features along object edges, we design a Sparse Feature Aggregation Block (SFAB) within the backbone and a coarse-to-fine deformable detection head, enabling hierarchical feature refinement from local to global, and improving the detection quality of sparse targets. Finally, to mitigate the lack of event-based small object datasets, we develop a high-quality, annotation-free data acquisition method and collect a real-world benchmark dataset for validation. Extensive experiments demonstrate that our approach achieves state-of-the-art (SOTA) performance on event-based small object detection tasks, with a mAP of 37.4% (+2.4%) on our benchmark and runs at 88 FPS, showcasing both accuracy and real-time capability. Our code and Supplement are available at https://github.com/Lqm26/ESOD.
Semi-Supervised Learning (SSL) can leverage abundant unlabeled data to boost model performance. However, the class-imbalanced data distribution in real-world scenarios poses great challenges to SSL, resulting in performance degradation. Existing class-imbalanced semi-supervised learning (CISSL) methods mainly focus on rebalancing datasets but ignore the potential of using hard examples to enhance performance, making it difficult to fully harness the power of unlabeled data even with sophisticated algorithms. To address this issue, we propose a method that enhances the performance of Imbalanced Semi-Supervised Learning by Mining Hard Examples (SeMi). This method distinguishes the entropy differences among logits of hard and easy examples, thereby identifying hard examples and increasing the utility of unlabeled data, better addressing the imbalance problem in CISSL. In addition, we maintain a class-balanced memory bank with confidence decay for storing high-confidence embeddings to enhance the pseudo-labels' reliability. Although our method is simple, it is effective and seamlessly integrates with existing approaches. We perform comprehensive experiments on standard CISSL benchmarks and experimentally demonstrate that our proposed SeMi outperforms existing state-of-the-art methods on multiple benchmarks, especially in reversed scenarios, where our best result shows approximately a 54.8% improvement over the baseline methods. Our code is available at https://github.com/pywin/SeMi.
Traditional Video Object Detection (VOD) is limited by pre-defined closed-set categories, restricting its ability to detect novel objects in real-world scenarios. To address this limitation, we make three key contributions. First, we formally define Open-Vocabulary Video Object Detection (Open-Vocabulary VOD) as the task of detecting objects in video streams from open-set categories, including novel categories unseen during training. Second, we establish an evaluation benchmark by utilizing existing datasets (LV-VIS, BURST, and TAO) to bridge the data gap for this new task. Third, we propose OV-VOD, an Open-Vocabulary VOD method that detects objects in videos beyond pre-defined training categories and addresses the shortcomings of image-level open-vocabulary detectors, which generally neglect the essential temporal and spatial information. Specifically, we design a Semantic-Presence Memory Tracking (SPMT) module that propagates object features across frames through a memory bank to leverage temporal consistency. Moreover, we propose a Spatial Object Relationship Distillation loss (L SR ) that captures inter-object spatial dependencies and enhances knowledge transfer during feature distillation. Experiments on multiple video datasets demonstrate that our OV-VOD exhibits superior zero-shot generalization capability compared to existing image-level open-vocabulary object detection methods.
Exposure correction aims to restore underexposed and overexposed images to normal exposed images in a single network. However, conventional methods primarily focus on correcting non-extreme exposure cases and struggle to accurately restore lightness and structure information in extreme exposure scenarios. Through a thorough investigation, we observe that the extreme exposure correction task is limited by the lack of high-quality benchmark datasets. To address the above challenges, in this paper, we construct the first Extreme Exposure Dataset named EED by manually collecting a large number of diverse scenes. By introducing probabilistic blur kernel, EED not only ensures the rich diversity and brightness distribution of scenes but also approaches the degradation of the real world. To achieve exposure correction in extreme conditions, we propose a novel Extreme Exposure Correction Network by leveraging the mask-aware Fourier transform prior, which decouples lightness and structure components precisely. To restore severe abnormal lightness and lost structure information in extreme exposure scenes, we introduce a well-exposed referenced image to guide the coarse restoration and employ a Timestep-guided Frequency Diffusion Module for further refinement. Extensive experiments demonstrate the superiority of our dataset and method. The dataset will be available at https://github.com/juvenoia/EED.
Underwater image restoration aims to remove geometric and color distortions due to water refraction, absorption, and scattering. Previous studies focus on restoring either color or geometry, but to our best knowledge, not both. However, in practice it may be cumbersome to address the two rectifications one by one. In this paper, we propose NeuroPump, a self-supervised method to simultaneously optimize and rectify underwater geometry and color as if water were pumped out. The key idea is to explicitly model refraction, absorption, and scattering in Neural Radiance Field (NeRF) pipeline, such that it not only performs simultaneous geometric and color rectification, but also enables to synthesize novel views and optical effects by controlling the decoupled parameters. In addition, to address the lack of real paired ground truth images, we propose an underwater 360 benchmark dataset that has real paired (i.e., with and without water) images. Our method clearly outperforms other baselines both quantitatively and qualitatively. Our code and dataset is available at https://ygswu.github.io/NeuroPump.github.io/.
Most few-shot learning methods aim to train models to learn parameters that can generalize to new categories using training sets, after which the model parameters are typically fixed. However, due to limited data, models often fail to learn generalizable parameters, as they tend to overfit source domain-specific inductive biases. This can lead to catastrophic forgetting or poor adaptation to new domains. Unlike previous methods, we propose a Text Feature guided dynamic Parameter Adjustment (TFPA) method for few-shot action recognition. Inspired by basis decomposition in vector spaces, TFPA reformulates the traditional linear layer into a set of basis mapping matrices in the parameter space. Each matrix functions analogously to a basis vector in linear algebra, and their linear combinations collectively span the parameter space. To construct a domain-adaptive parameter matrix from these combinations, we propose a Coordinate Vector Computation (CVC) module, which leverages text features as semantic guidance to adaptively estimate optimal linear combination coefficients for the basis mapping matrices. Furthermore, we propose a Centroid Exclusion Loss (CEL) and a Contrastive Clustering Loss (CCL) to enhance the distinctiveness among the basis mapping matrices. These regularization terms promote functional specialization and reduce redundancy across the basis mapping matrices, thereby enhancing performance. Experimental results on five benchmark datasets demonstrate the effectiveness and strong generalization ability of our method in few-shot action recognition. The code will be released soon at https://github.com/ReverseSuzhou/TFPA.
Traditional Camouflaged Object Detection (COD) methods heavily depend on labor-intensive annotated datasets which require extensive manual effort, resulting in limited generalization. While recent studies have combined Multimodal Large Language Models (MLLMs) and Vision Foundation Models (VFMs) to achieve zero-shot COD, their performance is hindered by modality gap between linguistic semantics and fine-grained visual cues, especially in complex camouflage scenarios. In this paper, we propose Language-to-instance generative visual Prompting (LiP), a novel framework that addresses this limitation by transforming text prompts generated by MLLMs into instance-level visual prompts through a text-to-image generative process. Specifically, we introduce a Diffusion-driven Visual Prompt Generation (DVPG) module that leverages Stable Diffusion model to synthesize visual references, enabling robust homogeneous modality matching for COD. Additionally, we introduce Instruction Contrastive Reasoning (ICR) module to enhance the semantic reliability of prompts by suppressing hallucinated concepts during MLLM inference. To the best of our knowledge, LiP is the first framework that utilize text-to-image generative model to construct instance-level visual prompts in COD task. Extensive experiments on four benchmark datasets demonstrate the effectiveness and strong generalization ability of our approach.
Table structure recognition (TSR), the task of extracting logical and physical structures from table images, is critical for document understanding. Current end-to-end image-to-text methods typically employ a top-down strategy where physical structure prediction depends on the logical decoder's output sequence. However, this process often suffers from training instability and misalignment between predicted bounding boxes and ground-truth cell positions. To address this issue, we propose G2LFormer, a novel transformer-based framework that employs a ''Global-to-Local'' query enhancement strategy. Specifically, G2LFormer introduces a Vision-guided Query Enhancer to integrate both textual and visual modalities, significantly improving the overall query representation capability and boosting prediction accuracy. Additionally, we design a Multi-scale Manhattan Vision-guider that leverages a spatial attenuation matrix to guide each query towards its corresponding cell location, effectively balancing local and global information for more precise bounding box generation. Extensive experiments on benchmark datasets demonstrate G2LFormer's superior performance, while ablation studies confirming the significant contribution of each proposed module in achieving state-of-the-art results. The source code and model have been released at: https://github.com/Hzbupahaozi/G2LFormer.
2D images and 3D point clouds are foundational data types for multimedia applications, including real-time video analysis, augmented reality (AR), and 3D scene understanding. Class-incremental semantic segmentation (CSS) requires incrementally learning new semantic categories while retaining prior knowledge. Existing methods typically rely on computationally expensive training based on stochastic gradient descent, employing complex regularization or exemplar replay. However, stochastic gradient descent-based approaches inevitably update the model's weights for past knowledge, leading to catastrophic forgetting, a problem exacerbated by pixel/point-level granularity. To address these challenges, we propose CFSSeg, a novel exemplar-free approach that leverages a closed-form solution, offering a practical and theoretically grounded solution for continual semantic segmentation tasks. This eliminates the need for iterative gradient-based optimization and storage of past data, requiring only a single pass through new samples per step. It not only enhances computational efficiency but also provides a practical solution for dynamic, privacy-sensitive multimedia environments. Extensive experiments on 2D and 3D benchmark datasets such as Pascal VOC2012, S3DIS, and ScanNet demonstrate CFSSeg's superior performance.
Recent latent diffusion models (LDMs) have been explored to generate diverse domain-specific images based on source domain data, showing promising performance in domain generalization tasks. However, although the generated images present counterfactual augmentation, such as the background and style changes, the distortion of object details disrupts the causal factors, such as texture and shape. This leads to negative outcomes when directly applying LDM to domain generalization in object detection. To address the problems mentioned above, we propose Object-Preserving Counterfactual Diffusion augmentation method (OPCD) to explore the diffusion model to generate diverse domain-specific images without disrupting the object details. First, we construct a region-aware image generation framework, which leverages labeled source domain data to guide LDM in generating region-constrained images that preserve the semantic consistency of the original source images. Second, we propose object-preserving counterfactual augmentation, which retains the object region of the generated image and fuses diversified global information. This ensures that object details are not distorted and that the generated information is maintained. Third, to reduce the resource burden of generating a large number of images in LDM, we design a random insertion strategy. It mixes generated and source domain images, turning limited diversity samples into abundant training data. Experimental results on several benchmark datasets show that OPCD outperforms existing methods in single-domain generalized object detection. Codes can be found at https://github.com/qinhongda8/OPCD.
Spatiotemporal Degradation-Aware 3D Gaussian Splatting for Realistic Underwater Scene Reconstruction
PDF ↗Reconstructing realistic underwater scenes from underwater video remains a meaningful yet challenging task in the multimedia domain. The inherent spatiotemporal degradations in underwater imaging, including caustics, flickering, attenuation, and backscattering, frequently result in inaccurate geometry and appearance in existing 3D reconstruction methods. While a few recent works have explored underwater degradation-aware reconstruction, they often address either spatial or temporal degradation alone, falling short in more real-world underwater scenarios where both types of degradation occur. We propose MartineSTD-GS, a novel 3D Gaussian Splatting-based framework that explicitly models both temporal and spatial degradations for realistic underwater scene reconstruction. Specifically, we introduce two paired Gaussian primitives: Intrinsic Gaussians represent the true scene, while Degraded Gaussians render the degraded observations. The color of each Degraded Gaussian is physically derived from its paired Intrinsic Gaussian via a Spatiotemporal Degradation Modeling (SDM) module, enabling self-supervised disentanglement of realistic appearance from degraded images. To ensure stable training and accurate geometry, we further propose a Depth-Guided Geometry Loss and a Multi-Stage Optimization strategy. We also construct a simulated benchmark with diverse spatial and temporal degradations and ground-truth appearances for comprehensive evaluation. Experiments on both simulated and real-world datasets show that MarineSTD-GS robustly handles spatiotemporal degradations and outperforms existing methods in novel view synthesis with realistic, water-free scene appearances.