NeRF-SLAM and GS-SLAM demonstrate excellent performance in high-fidelity rendering and real-time reconstruction in static scenes. However, real-world environments are often filled with dynamic objects, leading to tracking errors and mapping failures. Several dynamic SLAM approaches have been proposed, but they remain difficult to adopt due to challenges in deployment, framework compatibility, and generalization. To address these challenges, we introduce SLAM-X, the first plug-and-play module designed to universally enhance dynamic scene handling across a range of SLAM architectures. SLAM-Xleverages zero-shot segmentation and adaptively tracked sparse optical flow to generate dynamic masks, enabling tracking and mapping correction through continuous scene learning, while removing dynamic artifacts without requiring any task-specific fine-tuning. Extensive experiments on multiple real-world datasets demonstrate that SLAM-X effectively mitigates dynamic disturbances and seamlessly integrates with various NeRF-SLAM and GS-SLAM frameworks, achieving state-of-the-art performance in dynamic environments.
论文检索
输入标题、作者或关键词，从 1,620 篇学术成果中精准定位
In recent years, multi-view graph clustering (MVGC) has attracted increasing attention from researchers. However, many existing MVGC methods focus on view-level integration through strategies like assigning weights to different views, for example, ignoring cross-view interactions between nodes. In fact, cross-view interactions at node level are crucial for extraction and fusion of semantic information. Additionally, some methods separate representation learning from clustering, which results in suboptimal clustering performance. To address these problems, we propose a novel unsupervised cross-view message passing method for MVGC. The kernel of our method is the cross-view interaction mechanism, which dynamically constructs node-specific cross-view edges based on node features and structural information. The mechanism enables adaptive interactions of informative nodes from different views, which promotes the extraction and propagation of complementary information. Besides, our method unifies representation learning and hyperspherical clustering in an end-to-end framework, which projects node representations into a hypersphere space, thereby enabling direct acquisition of balanced clustering results without dependence on external clustering methods. We provide comprehensive analyses on our method, and evaluate our method on six multi-view datasets. The results show that our method consistently achieves superior performance than existing state-of-the-art multi-view clustering methods.
Audio-visual Generalized Zero-Shot Learning ((G)ZSL) has attracted significant attention for its ability to identify unseen classes in general video classification tasks. However, modality imbalance in (G)ZSL leads to over-reliance on the optimal modality, reducing discriminative capabilities for unseen classes. Though recent studies have attempted to address this issue, two challenges still remain unsolved: (a) Quality discrepancies, where modalities offer differing quantities and qualities of information for the same concept. (b) Content discrepancies, where the contributions of different samples within the same modality exhibit significant differences. To address these challenges, we propose a Discrepancy-Aware Attention Network (DAAN) for Enhanced Audio-Visual (G)ZSL. Our approach introduces a Redundant-Noise Mitigation Attention (RNMA) unit to minimize content discrepancies by mitigating redundant information in modalities and a Contrastive Sample Gradient Modulation (CSGM) mechanism to adjust gradient magnitudes and balance quality discrepancies. We quantify modality contributions by integrating optimization and convergence rate for more precise gradient modulation in CSGM. Experiments demonstrate DAAN achieves state-of-the-art performance on benchmark datasets, with ablation studies validating the effectiveness of individual modules. Code is available at https://github.com/xiaoxinning/DAAN-GZSL.
Multi-view clustering aims to effectively integrate data from multiple views to uncover the underlying clustering structure. However, existing methods typically adopt direct fusion strategies for multiview data, neglecting the issues of view gap-induced heterogeneity and the imbalance in view quality. Particularly, when there are significant differences between views, such direct fusion often leads to the loss of critical information and a decline in clustering performance. To address these challenges, we propose a novel Dynamic Progressive Fusion Multi-View Clustering (DPFMVC). DPFMVC employs a view-adaptive fusion mechanism that dynamically selects the most similar views, reducing conflicts between views while preserving complementary information. Additionally, DPFMVC introduces a dual contrastive loss module and a progressive fusion loss, which effectively align sample features with clustering centers, promoting efficient integration of multi-view information. Specifically, the dual contrastive loss compares the similarity between sample features and cluster centers, ensuring cross-view feature consistency and thus enhancing the discriminability of clustering. Meanwhile, the progressive fusion loss progressively adjusts the fusion order of views, effectively reducing the negative impact of low-quality views on the clustering results, strengthening the synergy between views, and facilitating more effective information fusion.Comprehensive experiments on multiple public benchmarks show that DPFMVC delivers superior clustering results and exhibits overall great effectiveness compared to state-of-the-art techniques.
Attribute-missing graph clustering has emerged as a significant unsupervised task, where only attribute vectors of partial nodes are available and the graph structure is intact. The related models generally follow the two-step paradigm of imputation and refinement. However, most imputation approaches fail to capture class-relevant semantic information, leading to sub-optimal imputation for clustering. Moreover, existing refinement strategies optimize the learned embedding through graph reconstruction, while neglecting the fact that some attributes are uncorrelated with the graph. To remedy the problems, we establish the Clustering-oriented Generative Imputation with reliable Refinement (CGIR) model. Concretely, the subcluster distributions are estimated to reveal the class-specific characteristics precisely, and constrain the sampling space of the generative adversarial module, such that the imputation nodes are impelled to align with the correct clusters. Afterwards, multiple subclusters are merged to guide the proposed edge attention network, which identifies the edge-wise attributes for each class, so as to avoid the redundant attributes in graph reconstruction from disturbing the refinement of overall embedding. To sum up, CGIR splits attribute-missing graph clustering into the search and mergence of subclusters, which guides to implement node imputation and refinement within a unified framework. Extensive experiments prove the advantages of CGIR over state-of-the-art competitors.
Image fusion aims to integrate complementary information across modalities to generate high-quality fused images, thereby enhancing the performance of high-level vision tasks. While global spatial modeling mechanisms show promising results, constructing long-range feature dependencies in the spatial domain incurs substantial computational costs. Additionally, the absence of ground-truth exacerbates the difficulty of capturing complementary features effectively. To tackle these challenges, we propose a Residual Prior-driven Frequency-aware Network, termed as RPFNet. Specifically, RPFNet employs a dual-branch feature extraction framework: the Residual Prior Module (RPM) extracts modality-specific difference information from residual maps, thereby providing complementary priors for fusion; the Frequency Domain Fusion Module (FDFM) achieves efficient global feature modeling and integration through frequency-domain convolution. Additionally, the Cross Promotion Module (CPM) enhances the synergistic perception of local details and global structures through bidirectional feature interaction. During training, we incorporate an auxiliary decoder and saliency structure loss to strengthen the model's sensitivity to modality-specific differences. Furthermore, a combination of adaptive weight-based frequency contrastive loss and SSIM loss effectively constrains the solution space, facilitating the joint capture of local details and global features while ensuring the retention of complementary information. Extensive experiments validate the fusion performance of RPFNet, which effectively integrates discriminative features, enhances texture details and salient objects, and can effectively facilitate the deployment of the high-level vision task. The source code can be available at https://github.com/wang-x-1997/RPFNet.
Multi-view clustering is one of the fundamental unsupervised multimedia analysis tasks. Recent studies have mainly focused on developing multi-view clustering approaches, which can achieve state-of-the-art clustering performance. However, most of the existing works just focus on multi-view clustering with fixed views, which lacks flexibility with guidance of the views dynamically generated. Besides, these works ignore to integrate generating views in a dynamic manner and learning the common representation shared by different views into a unified framework. To this end, we propose the Flexible Multi-view Clustering with Dynamic Views Generation (FMCDVG). Specifically, FMCDVG adopts the graph convolutional network and auto-encoder to dynamically generate the topological graph representation and node attribute representation as two different views, respectively. FMCDVG introduces the latent representation shared by different feature representations and integrates multiple feature representations based on node attributes and graph structure into the latent representation with reconstruction through reconstructed encoding networks (REN). FMCDVG jointly conducts generating views in a dynamic manner and learning the common representation shared by different views in a unified optimization framework. We demonstrate that FMCDVG is able to consistently achieve better clustering performance than the state-of-the-art methods through comprehensive experiments.
Multi-modal knowledge graph reasoning (MKGR) seeks to conjecture plausible facts in MKGs by learning effective representations from various modalities (e.g., structure, text, and image). However, due to holistic redundancy (i.e., each modality carries task-irrelevant redundancy) and modality conflict (i.e., different modalities contain contradictory information), the reasoning performance of current methods is substantially impaired. In this paper, we propose a novel Consistency Discovery-guided Information Bottleneck (CDIB) framework to address the aforementioned challenges. Specifically, a modality compression module is first designed to learn modality-private entity representations of alleviating redundant information. Then, a consistency discovery module is developed to discover cross-modal consistency during multi-modal fusion to learn the comprehensive entity representations. To retain task-relevant information, an information preservation module is devised to further enrich the comprehensive entity representations to be predictive for MKGR. Extensive experiments indicate that CDIB achieves state-of-the-art reasoning ability on two benchmark datasets over current MKGR baselines, and also exhibits promising robustness against noise.
Medical image synthesis is crucial in clinical workflows, enabling the generation of missing modalities from available imaging data. While recent diffusion-based models show promise in medical image synthesis, they face two key limitations: progressive distribution drift from coarse intermediate samples and structural granularity loss due to missing high-frequency constraints. To address these challenges, we propose Dual Diffusion Bridge (DualDB), a framework integrating implicit distribution alignment and explicit structural constraints within a unified diffusion bridge paradigm. First, implicit distribution alignment employs optimal transport-guided adversarial learning to minimize statistical discrepancies between intermediate and target distributions, mitigating global distribution drift. Second, explicit structural alignment applies gradient-driven constraints to preserve high-frequency anatomical features, preventing structural degradation during reverse diffusion. This complementary design ensures both global statistical consistency and local anatomical precision in the synthesized results. Extensive experiments on multi-contrast MRI and MRI-CT translation show that DualDB outperforms state-of-the-art methods in quantitative performance and visual fidelity, maintaining superior anatomical accuracy even under noisy conditions.
Open-vocabulary 3D visual grounding aims to localize target objects based on free-form language queries, which is crucial for embodied AI applications such as autonomous navigation, robotics, and augmented reality. Learning 3D language fields through neural representations enables accurate understanding of 3D scenes from limited viewpoints and facilitates the localization of target objects in complex environments. However, existing language field methods struggle to accurately localize instances using spatial relations in language queries, such as ''the book on the chair.'' This limitation mainly arises from inadequate reasoning about spatial relations in both language queries and 3D scenes. In this work, we propose SpatialReasoner, a novel neural representation-based framework with large language model (LLM)-driven spatial reasoning that constructs a visual properties-enhanced hierarchical feature field for open-vocabulary 3D visual grounding. To enable spatial reasoning in language queries, SpatialReasoner fine-tunes an LLM to capture spatial relations and explicitly infer instructions for the target, anchor, and spatial relation. To enable spatial reasoning in 3D scenes, SpatialReasoner incorporates visual properties (opacity and color) to construct a hierarchical feature field. This field represents language and instance features using distilled CLIP features and masks extracted via the Segment Anything Model (SAM). The field is then queried using the inferred instructions in a hierarchical manner to localize the target 3D instance based on the spatial relation in the language query. Notably, SpatialReasoner is not limited to a specific 3D neural representation; it serves as a framework adaptable to various representations, such as Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS). Extensive experiments show that our framework can be seamlessly integrated into different neural representations, outperforming baseline models in 3D visual grounding while empowering their spatial reasoning capability. Project Homepage:ZhenyangLiu.github.io/SpatialReasoner.
Model merging techniques aim to consolidate multiple fine-tuned models into a single unified model, reducing both storage and computational overhead while retaining task-specific performance. However, existing methods face several limitations: monotonous compression techniques that fail to account for task-specific weight distribution characteristics, weight-magnitude-based compression that fails to consider functional importance revealed by activation patterns, and non-adaptive allocation strategies that ignores task-specific layer importance. To overcome these challenges, we propose OA-Merge, a novel Outlier-Aware Model Merging framework that leverages task activation outliers to enable adaptive compression and resource allocation across tasks. OA-Merge comprises three key components: (1) dynamic hybrid decomposition technique that formulates task vectors as tailored combinations of low-rank and sparse components adapted to task-specific statistical distributions, (2) activation-informed compression methodology that incorporates task-specific activation statistics to prioritize functionally important weights, and (3) task-related allocation that optimizes the distribution of compression resources according to layer-specific importance metrics derived from activation outlier analysis. These hybrid outlier-aware strategies adapt dynamically to each task's intrinsic characteristics, avoiding the pitfalls of one-size-fits-all ways. Extensive experiments on both vision models (e.g., ViT) and language models (e.g., RoBERTa, Qwen) demonstrate that OA-Merge outperforms state-of-the-art baselines, achieving average performance gains of 3.2% on vision tasks and 2.8% on language tasks.
While current multimodal anomaly detection methods predominantly employ intermediate fusion strategies, they often suffer from inadequate cross-modal interaction and irreversible information loss during feature alignment processes. To overcome these limitations, we propose Hierarchical Geometry-Color Fusion (HGCF), a novel framework that establishes deep synergistic relationships between RGB texture features and point cloud geometric representations. Firstly, we propose a bidirectional cross-modal early fusion mechanism that enables complementary information exchange between point cloud and RGB modalities at the input level. Secondly, we introduce a local self-supervised geometric color reconstruction network with group-wise feature alignment, enhancing fine-grained feature extraction through joint color-geometry reconstruction tasks. Finally, we propose a local window spatial-consistent attention fusion, which achieves semantic consistency and spatial consistency by emphasizing local mutation features to improve the detection of subtle anomalies. Extensive experiments show our model achieves 99.1% I-AUROC on MVTec 3D-AD and 91.7% on Eyecandies, both surpassing state-of-the-art methods.
RGB-D salient object detection (SOD) aims to identify the most conspicuous objects in a scene with the incorporation of depth cues. Existing methods mainly rely on CNNs, limited by the local receptive fields, or Vision Transformers that suffer from the cost of quadratic complexity, posing a challenge in balancing performance and computational efficiency. Recently, state space models (SSM), Mamba, have shown great potential for modeling long-range dependency with linear complexity. However, directly applying SSM to RGB-D SOD may lead to deficient local semantics as well as the inadequate cross-modality fusion. To address these issues, we propose a Local Emphatic and Adaptive Fusion state space model (LEAF-Mamba) that contains two novel components: 1) a local emphatic state space module (LE-SSM) to capture multi-scale local dependencies for both modalities. 2) an SSM-based adaptive fusion module (AFM) for complementary cross-modality interaction and reliable cross-modality integration. Extensive experiments demonstrate that the LEAF-Mamba consistently outperforms 16 state-of-the-art RGB-D SOD methods in both efficacy and efficiency. Moreover, our method can achieve excellent performance on the RGB-T SOD task, proving a powerful generalization ability. Our code is publicly available at https://github.com/LanhooNg/LEAF-Mamba.
Stereo image super-resolution (SSR) aims to enhance high-resolution details by leveraging information from stereo image pairs. However, existing stereo super-resolution (SSR) upsampling methods (e.g., pixel shuffle) often overlook cross-view geometric consistency and are limited to fixed-scale upsampling. The key issue is that previous upsampling methods use convolutions to independently process deep features of different views, lacking cross-view and non-local information perception, making it difficult to select beneficial information from multi-view scenes adaptively. In this work, we propose Stereo Implicit Neural Representation (StereoINR), which innovatively models stereo image pairs as continuous implicit representations. This continuous representation breaks through the scale limitations, providing a unified solution for arbitrary-scale stereo super-resolution reconstruction of left-right views. Furthermore, by incorporating spatial warping and cross-attention mechanisms, StereoINR enables effective cross-view information fusion and achieves significant improvements in pixel-level geometric consistency. Extensive experiments on multiple datasets demonstrate that StereoINR outperforms out-of-training-distribution scale upsampling and matches state-of-the-art SSR methods within training-distribution scales.
Deep learning has shown remarkable performance in integrating multimodal data for survival prediction. However, existing multimodal methods mainly focus on single cancer types and overlook the challenge of generalization across cancers. In this work, we are the first to reveal that multimodal prognosis models often generalize worse than unimodal ones in cross-cancer scenarios, despite the critical need for such robustness in clinical practice. To address this, we propose a new task: Cross-Cancer Single Domain Generalization for Multimodal Prognosis, which evaluates whether models trained on a single cancer type can generalize to unseen cancers. We identify two key challenges: degraded features from weaker modalities and ineffective multimodal integration. To tackle these, we introduce two plug-and-play modules: Sparse Dirac Information Rebalancer (SDIR) and Cancer-aware Distribution Entanglement (CADE). SDIR mitigates the dominance of strong features by applying Bernoulli-based sparsification and Dirac-inspired stabilization to enhance weaker modality signals. CADE, designed to synthesize the target domain distribution, fuses local morphological cues and global gene expression in latent space. Experiments on a four-cancer-type benchmark demonstrate superior generalization, laying the foundation for practical, robust cross-cancer multimodal prognosis. Code is available at here.
Light Field (LF) semantic segmentation relies on leveraging redundant information across multiple views to assign a semantic label to each pixel of the central view. Recent approaches typically feed the views into a pre-trained backbone and utilize an estimated depth map to aggregate semantic representations for label prediction. However, these methods ignore the correlation between encoded structural cues in LF and semantic labels. On one hand, it is challenging to identify matching points for regions that are occluded in some views. This broken view consistency emphasizes object edge localization, facilitating more precise edge labeling. On the other hand, the depth continuity for the same object ensures semantic consistency in adjacent regions. Therefore, effectively extracting structural cues and integrating them into semantic segmentation are key points in LF semantic segmentation.In this paper, we propose an Epipolar Consistency-based network for structure-aware LF semantic segmentation, termed ECNet. First, we explore the epipolar consistency between views to characterize the edges and depth cues of the input. Based on the embedded edges information, we design an edge-semantic correlation transformer to generate fine-grained representations of object edges. Furthermore, the proposed depth-semantic correlation transformer maps semantic features of one object closer together according the depth information.Extensive experiments demonstrate that ECNet achieves state-of-the-art performance, which reduces computational cost by 33.3% (in terms of FLOPs) while maintaining high segmentation accuracy.
Domain adaptation, which bridges the domain gap between heterogeneous agents, has emerged as an effective solution to improve the perception capabilities of multi-agent systems. However, it may introduce backdoor vulnerabilities, as adversaries could exploit the collaborative process to propagate malicious features across agents, yet these threats remain largely unexplored. In this paper, we take the first step to study the backdoor attacks in this safety-critical scenario, with the 3D object detection task as the representative case. To this end, we propose BadMDA, the first backdoor attack tailored for the domain adaptation process to collapse multi-agent perception. Specifically, we first propose a gradient-suppression trigger optimization module to mitigate trigger distortion during the domain adaptation. By utilizing the optimizable additive triggers and minimizing gradient variations of triggered features induced by the domain adaptation, we reduce the transformation magnitude of triggered features, thereby maintaining the trigger effectiveness. Then, we propose a dual-gradient guided poisoning module to achieve clean-label poisoning in 3D object detection tasks. This module aligns training gradients with poisoned ones to learn malicious features, while enforcing the orthogonality between training and benign gradients. Consequently, the learned malicious features mislead the victim's finetuning updates, causing detection failures upon receiving triggered features while only slightly affecting the victim agent's model utility. Extensive experiments on various dominant domain adaptation methods show the superior attacking effectiveness and universality of BadMDA, underscoring the need for a more advanced defense.
Long video understanding, which leverages Video-LLMs to analyze and interpret extended video content to extract meaningful information, insights, or summaries, is a fundamental task in multimedia domain. Chain-of-thought (CoT) methods are widely adopted to enhance long video understanding by incorporating intermediate reasoning steps. However, the iterative nature of CoT-requiring a lengthy sequence of internal thoughts-significantly increases the latency of video object description. To address this challenge, we design Compressed Scene Graph-enabled CoT (CSGCoT), a novel approach that facilitates efficient and accurate long-video object description. Inspired by video codec principles, we propose a compressed scene graph composed of two components: Key-SG for key frames and Delta-SG for delta frames, enabling efficient encoding of scene information across video segments. Specifically, CSGCoT comprises three major modules: (1) a Key-SG Detector that identifies representative segments, (2) a Delta-SG Generator that produces compensated representations for delta segments, and (3) a SG-Query Manager that converts scene graphs into natural language prompts for video object description. Experiments show that CSGCoT achieves comparable accuracy to SOTA methods while reducing latency by over 62.6% on hour-long videos while maintaining comparable accuracy.
Multimodal emotion recognition (MER) aims to identify emotional states by integrating and analyzing information from multiple modalities. However, inherent modality heterogeneity and inconsistencies in emotional cues remain key challenges that hinder performance. To address these issues, we propose a Decoupled Representations with Knowledge Fusion (DRKF) method for MER. DRKF consists of two main modules: an Optimized Representation Learning (ORL) Module and a Knowledge Fusion (KF) Module. ORL employs a contrastive mutual information estimation method with progressive modality augmentation to decouple task-relevant shared representations and modality-specific features while mitigating modality heterogeneity. KF includes a lightweight self-attention-based Fusion Encoder (FE) that identifies the dominant modality and integrates emotional information from other modalities to enhance the fused representation. To handle potential errors from incorrect dominant modality selection under emotionally inconsistent conditions, we introduce an Emotion Discrimination Submodule (ED), which enforces the fused representation to retain discriminative cues of emotional inconsistency. This ensures that even if the FE selects an inappropriate dominant modality, the Emotion Classification Submodule (EC) can still make accurate predictions by leveraging preserved inconsistency information. Experiments show that DRKF achieves state-of-the-art (SOTA) performance on IEMOCAP, MELD, and M3ED. The source code is publicly available at https://github.com/PANPANKK/DRKF.
Reconstructing high-fidelity 3D head avatars is crucial in various applications such as virtual reality. The pioneering methods reconstruct realistic head avatars with Neural Radiance Fields (NeRF), which have been limited by training and rendering speed. Recent methods based on 3D Gaussian Splatting (3DGS) significantly improve the efficiency of training and rendering. However, the surface inconsistency of 3DGS results in subpar geometric accuracy; later, 2DGS uses 2D surfels to enhance geometric accuracy at the expense of rendering fidelity. To leverage the benefits of both 2DGS and 3DGS, we propose a novel method named MixedGaussianAvatar for realistically and geometrically accurate head avatar reconstruction. Our main idea is to utilize 2D Gaussians to reconstruct the surface of the 3D head, ensuring geometric accuracy. We attach the 2D Gaussians to the triangular mesh of the FLAME model and connect additional 3D Gaussians to those 2D Gaussians where the rendering quality of 2DGS is inadequate, creating a mixed 2D-3D Gaussian representation. These 2D-3D Gaussians can then be animated using FLAME parameters. We further introduce a progressive training strategy that first trains the 2D Gaussians and then fine-tunes the mixed 2D-3D Gaussians. We use a unified mixed Gaussian representation to integrate the two modalities of 2D image and 3D mesh. Furthermore, the comprehensive experiments demonstrate the superiority of MixedGaussianAvatar. The code will be released.