Conventional voice conversion modifies voice characteristics from a source speaker to a target speaker, relying on audio input from both sides. However, this process becomes infeasible when clean audio is unavailable, such as in silent videos or noisy environments. In this work, we focus on the task of Silent Face-based Voice Conversion (SFVC), which does voice conversion entirely from visual inputs. i.e., given images of a target speaker and a silent video of a source speaker containing lip motion, SFVC generates speech aligning the identity of the target speaker while preserving the speech content in the source silent video. As this task requires generating intelligible speech and converting identity using only visual cues, it is particularly challenging. To address this, we introduce MuteSwap, a novel framework that employs contrastive learning to align cross-modality identities and minimize mutual information to separate shared visual features. Experimental results show that MuteSwap achieves impressive performance in both speech synthesis and identity conversion, especially under noisy conditions where methods dependent on audio input fail to produce intelligible results, demonstrating both the effectiveness of our training approach and the feasibility of SFVC. Demo page is available at https://pussycat0700.github.io/MuteSwap-Demo/.
论文检索
输入标题、作者或关键词，从 1,620 篇学术成果中精准定位
Currently, fusion-based hyperspectral image super-resolution (fusion-based HSI-SR) has become an efficient technology to improve the spatial resolution of hyperspectral images. However, in real scenarios, it may not be possible to obtain high-resolution multispectral images (HR-MSI) of the same temporal and region corresponding to low-resolution hyperspectral images (LR-HSI) due to the limitations of imaging conditions and environmental changes. In view of this spatial-temporal constraint, it becomes a feasible solution to regard HR-MSI, which has similar spatial structure and semantics to LR-HSI, as a reference to assist in reconstruction. Therefore, this paper proposes a Cross-Correlation & Self-Similarity Guided Texture Transfer Network (C2S2TNet), which utilizes the texture details of HR-MSI and the self-similarity information of LR-HSI to achieve reference-based hyperspectral image super-resolution. Specifically, we design a Cross-Correlation & Self-Similarity Guided Cluster-Aware Matching (C2S2CAM) strategy, which realizes multi-correspondence texture matching and feature aggregation in non-local regions based on dynamic clustering and cluster-aware graph structure, effectively alleviating the misuse and underuse of information. In addition, we also propose a Spectral-Spatial State-Space Fusion Module (S2-SSFM) based on the state-space model to perform feature fusion and enhancement in both spatial and spectral domains to ensure that the target HR-HSI maintains the spatial-spectral structural consistency with the LR-HSI. Experimental verification shows that C2S2TNet can achieve excellent performance in cross-temporal and cross-regional scenarios, confirming the effectiveness of this method. Code can be accessed at https://github.com/Jiahuiqu/C2S2TNet.
Multimodal link prediction on multimodal knowledge graphs is an inference task aimed at finding missing triples, which seeks to improve prediction accuracy by leveraging a wide range of information. However, current multimodal knowledge graph link prediction methods are primarily designed in the spatial domain, necessitating ever-growing complexity in fusion strategies. In addition, most of them focus on only three modalities (text, image, and structure). Forward-looking approaches, however, should accommodate a broader array of modalities. Motivated by the operational simplification enabled by transforming features into the frequency (or time-frequency) domain, we propose a wavelet-transform-based multimodal link prediction method, WFF, which offers high modal extensibility and low fusion complexity. Specifically, for unimodal information, we designed a Unimodal Time-Frequency Knowledge Enhancement module, UTFKE, which extracts time-frequency features via discrete wavelet transform and enhances information quality through adaptive filtering. To address the challenge of multimodal fusion, we devised a Multimodal Time-Frequency Knowledge Fusion module MTFKF that supports high modal extensibility and enables effective, efficient integration. Extensive experiments on multiple well-known datasets demonstrate that WFF outperforms strong baselines and achieves state-of-the-art performance. In addition, WFF extends modality to audio and video, further validating the model's effectiveness. Our code is available at https://github.com/xxd12315/WFF.
Salient object detection (SOD) in light field data presents unique challenges due to dynamic semantic inconsistencies across focal slices and representation heterogeneity between focal slices and the all-focus image. Existing methods often treat focal slices uniformly or rely on simple fusion strategies, which fail to address focus-induced semantic drift and cross-modal feature misalignment. To tackle these issues, we propose LFMamba, a unified network that jointly models dynamic semantic consistency and adaptive cross-modal fusion. We design the Focal-aware State Space Module (FSSM), which generates focal-aware semantic prompts through low-rank decomposition and adaptively routes them according to focal plane indices, thereby enabling bidirectional semantic propagation across slices through non-causal state transitions. Furthermore, we introduce the Focal-guided Cross-modal Fusion Module (FCFM), which mitigates cross-modal heterogeneity by a two-stage hierarchical strategy, combining structure-aware low-level alignment and gated high-level semantic fusion. Extensive experiments on four public light field SOD benchmarks demonstrate that LFMamba achieves superior performance compared to state-of-the-art methods, with improved robustness and consistency under complex focal variation scenarios.
Effective cross-modal retrieval requires robust alignment of heterogeneous data types. Most existing methods focus on bi-modal retrieval tasks and rely on distributional alignment techniques such as Kullback-Leibler divergence, Maximum Mean Discrepancy, and correlation alignment. However, these methods often suffer from critical limitations, including numerical instability, sensitivity to hyperparameters, and their inability to capture the full structure of the underlying distributions. In this paper, we introduce the Cauchy-Schwarz (CS) divergence, a hyperparameter-free measure that improves both training stability and retrieval performance. We further propose a novel Generalized CS (GCS) divergence inspired by Holder's inequality. This extension enables direct alignment of three or more modalities within a unified mathematical framework through a bidirectional circular comparison scheme, eliminating the need for exhaustive pairwise comparisons. Extensive experiments on six benchmark datasets demonstrate the effectiveness of our method in both bi-modal and tri-modal retrieval tasks. The code of our CS/GCS divergence is publicly available at https://github.com/JiahaoZhang666/CSD.
Audio-visual speech recognition (AVSR) leverages complementary visual cues to improve speech recognition. However, in real-world scenarios, both modalities may suffer from noise or occlusion. In such scenarios, most existing fusion strategies overlook the variation in modality-specific quality under different degradation conditions. This limitation may lead to dominance of corrupted modality in the fusion process, resulting in worse AVSR performance than unimodal systems, termed as Corrupted Modality Bias (CMB) in this work. To address this, a self-supervised speech representation learning framework, called AV-RISE, is proposed to employ teacher-student self-distillation to robustly reconstruct clean speech representations from corrupted audio-visual inputs. A hierarchical fusion mechanism is designed to progressively refine audio and visual representations by integrating the Suppression and Enhancement Interaction (SEI) module into each layer of the pre-trained encoder. In the SEI module, cross-modal suppression and modality-oriented enhancement are performed to mitigate noise-induced feature inconsistencies, which strengthens the modeling of complementary semantic representations. Extensive experiments on the LRS2 and LRS3 datasets demonstrate that AV-RISE outperforms SOTA AVSR models, especially under extreme degradation. Most importantly, the hierarchical SEI-based fusion effectively enhances reliable semantic representations to mitigate CMB, by evaluating feature similarities between clean and noise samples.
Multimodal graphs, which integrate unstructured heterogeneous data with structured interconnections, offer substantial real-world utility but remain insufficiently explored in unsupervised learning. In this work, we initiate the study of multimodal graph clustering, aiming to bridge this critical gap. Through empirical analysis, we observe that real-world multimodal graphs often exhibit hybrid neighborhood patterns, combining both homophilic and heterophilic relationships. To address this challenge, we propose a novel framework---Disentangled Multimodal Graph Clustering (DMGC) ---which decomposes the original hybrid graph into two complementary views: (1) a homophily-enhanced graph that captures cross-modal class consistency, and (2) heterophily-aware graphs that preserve modality-specific inter-class distinctions. We introduce a Multimodal Dual-frequency Fusion mechanism that jointly filters these disentangled graphs through a dual-pass strategy, enabling effective multimodal integration while mitigating category confusion. Our self-supervised alignment objectives further guide the learning process without requiring labels. Extensive experiments on both multimodal and multi-relational graph datasets demonstrate that DMGC achieves state-of-the-art performance, highlighting its effectiveness and generalizability across diverse settings. Our code is available at https://github.com/Uncnbb/DMGC.
HER2 Expression Prediction with Flexible Multi-Modal Inputs via Dynamic Bidirectional Reconstruction
In the field of HER2 expression level assessment for breast cancer, clinical evaluations often rely on the synergistic analysis of both H&E and IHC stained images. However, acquiring dual-modality images for the same patient is frequently hindered by complex clinical workflows and high costs, resulting in missing modalities. To address this challenge, we propose an adaptive bimodal input prediction framework that flexibly supports both single-modality and dual-modality inputs. This framework employs a dynamic branch selection mechanism to overcome the rigid dependency of existing models on complete inputs, enabling accurate predictions using either H&E or IHC images alone, while retaining the ability for joint inference when both modalities are available. The core technical innovations include: a missing modality branch selector that dynamically activates either a modality completion process or an end-to-end dual-modality inference pipeline based on the available input; and a cross-modal generative adversarial network (CM-GAN) that facilitates context-aware reconstruction of the missing modality in the feature space. This design improves the prediction accuracy from 71.44% to 94.25% when using single-modality H&E images, significantly mitigating performance degradation caused by incomplete information. Experimental results demonstrate that the proposed framework achieves a prediction accuracy of 95.09% with full dual-modality input and maintains a high reliability of 90.28% under single-modality conditions. By adopting this ''dual-modality preferred, single-modality compatible'' flexible architecture, healthcare institutions can achieve near dual-modality accuracy without mandating synchronized acquisition of both image types. This is particularly valuable for regions with limited IHC staining infrastructure, offering a cost-effective clinical solution and substantially enhancing the accessibility of HER2 expression level assessment.
Lifelong Person Re-identification (LReID) focuses on continuously adapting to new domains over time while preserving knowledge from previously seen domains, particularly under the domain incremental learning setting. The major challenge of LReID is catastrophic forgetting, typically caused by large domain shifts during training. To address this, we propose a novel Amplitude-aware Domain Style Replay (ADSR) framework, which introduces a Fourier-based Style Transfer (FST) mechanism to generate synthetic data that reflects the style of previously encountered domains. These proxy images help retain prior knowledge without the need to store actual past data. Our method transfers stylistic information-mainly encoded in the amplitude spectrum-from old domains to new ones, creating old-stylized images that preserve the content of new domain data while adopting the visual style of earlier domains. To further boost generalization, we design a Self-Stylization Normalization (SSN) module that adapts the current domain's style distribution, making the model more robust to stylistic variations. Additionally, we introduce a Multi-Granularity Transfer (MGT) module that uses K-Means clustering to extract multiple representative style features from each domain, enabling compact yet comprehensive storage and replay of domain-specific information. Extensive experiments on multiple LReID benchmarks show that ADSR achieves superior performance over existing approaches, effectively reducing forgetting and improving cross-domain generalization. Our code is available at https://github.com/cclong8/MM2025-ADSR.
With the popularization of the Internet and the diversification of attack methods, web security has become an important part of information security. As a carrier of network behavior, traffic can reveal attack behaviors in the Web environment through malicious traffic detection. Since images can fully express spatial features and local associations, it is feasible to visualize traffic as images and retrieve key feature information to detect malicious traffic. However, existing methods are prone to redundancy during feature extraction. Secondly, a single perspective makes it difficult to learn patterns with universality from diverse feature information. In addition, the selection of segmentation thresholds in the preprocessing is closely related to the model's information retrieval effect. The commonly adopted preset thresholds are difficult to cope with the changes in traffic data, limiting the applicability of the existing methods. Therefore, this paper proposes a Multi-Module-Based Composite Robust Model for Network Attack Detection (MCNAD). The model adopts depthwise separable convolution (DSC) to reduce redundant information, proposes a multi-scale feature learning module to enhance the model characterization ability, and proposes a gray level co-occurrence matrix segmentation algorithm with adaptive threshold (GLCM-AT) to optimize data preprocessing. The results show that MCNAD improves detection performance with better detection efficiency, generalization ability, and robustness, demonstrating its wide applicability in multiple scenarios.
Structural variant (SV) calling plays a critical role in understanding genome diversity and disease mechanisms. Although deep learning techniques have been increasingly applied to SV identification, existing general-purpose models still face significant challenges, including incomplete extraction of alignment signals, limited accuracy and efficiency, and poor performance in highly polymorphic or structurally complex genomic regions. These limitations lead to suboptimal detection accuracy in current SV callers. In this work, we present MMF-SV, a multi-modal feature fusion-based model (MMF) for SV calling. MMF-SV integrates matching patterns and statistical information from CIGAR signals with textual features extracted from alignment information, enabling comprehensive representation of diverse SV signals. We trained MMF-SV using CLIP, and the trained model achieved over 96% F1 score for classifying various types of variations. We validated the stability and robustness of the MMF-SV model through 5-fold cross-validation. Compared to existing long-read SV callers, MMF-SV achieves higher accuracy and can be effectively integrated with them to significantly reduce the number of false positives in the calling results.
Multimodal Sentiment Analysis (MSA) faces two critical challenges: the lack of interpretability in the decision logic of multimodal fusion and modality imbalance caused by disparities in inter-modal information density. To address these issues, we propose KAN-MCP, a novel framework that integrates the interpretability of Kolmogorov-Arnold Networks (KAN) with the robustness of the Multimodal Clean Pareto (MCPareto) framework. First, KAN leverages its univariate function decomposition to achieve transparent analysis of cross-modal interactions. This structural design allows direct inspection of feature transformations without relying on external interpretation tools, thereby ensuring both high expressiveness and interpretability. Second, the proposed MCPareto enhances robustness by addressing modality imbalance and noise interference. Specifically, we introduce the Dimensionality Reduction and Denoising Modal Information Bottleneck (DRD-MIB) method, which jointly denoises and reduces feature dimensionality. This approach provides KAN with discriminative low-dimensional inputs to reduce the modeling complexity of KAN while preserving critical sentiment-related information. Furthermore, MCPareto dynamically balances gradient contributions across modalities using the purified features output by DRD-MIB, ensuring lossless transmission of auxiliary signals and effectively alleviating modality imbalance. This synergy of interpretability and robustness not only achieves superior performance on benchmark datasets such as CMU-MOSI, CMU-MOSEI, and CH-SIMS v2 but also offers an intuitive visualization interface through KAN's interpretable architecture. Our code is released on https://github.com/LuoMSen/KAN-MCP.
Salient object detection in optical remote sensing images (ORSI-SOD) faces unique challenges due to complex backgrounds, diverse scales, and multi-directional objects. Existing methods primarily rely on visual features, often struggling to distinguish salient objects from visually similar backgrounds. To address this limitation, we leverage large language models (LLMs) to expend existing ORSI-SOD datasets with detailed textual annotations, creating a more comprehensive benchmark for image-text ORSI-SOD. Building upon this foundation, we propose the Frequency Meets Semantics Network (FMS-Net), a novel framework that integrates text-visual fusion with directional spectral enhancement for ORSI-SOD. FMS-Net consists of two key innovations: the Hierarchical Multi-Modal Dual-Channel Fusion (HMDF) module and the Adaptive Directional Spectral Enhancement (ADSE) module. The HMDF module enables bidirectional interactions between visual and textual features via parallel global-local attention mechanisms, progressively enriching visual representations with semantic context. Meanwhile, the ADSE module enhances feature representations in the frequency domain, capturing directional patterns and boundary details critical for accurate saliency detection. Extensive experiments on two public datasets, ORSSD and EORSSD, demonstrate that FMS-Net outperforms state-of-the-art methods, particularly in complex scenes with ambiguous boundaries. Our work paves the way for integrating multi-modal and frequency-based approaches in the interpretation of optical remote sensing images (ORSI).
Spatial transcriptomics technologies enable the integration of gene expression profiles with spatial context, facilitating a deeper understanding of tissue architecture through downstream tasks such as clustering. However, existing approaches predominantly focus on highly variable genes (HVGs), while the informative structural and contextual signals embedded in low variability genes (LVGs) remain largely underutilized. To bridge this gap, we propose SALVG (Spatial Augmentation via Latent Variable Genes), a novel and plug-and-play framework that leverages LVG-derived structural priors to enhance HVG representation learning for spatial clustering. Specifically, SALVG constructs spatial, feature, and combined graphs for both HVGs and LVGs, and introduces two graph-based augmentation strategies to inject LVG information into HVG graphs. The first strategy enhances the HVG combined graph directly using the LVG combined graph, while the other individually augments HVG spatial and feature graphs with their LVG counterparts before fusing them into a new combined representation. These enhanced graph structures are subsequently employed for downstream clustering. To the best of our knowledge, SALVG is the first framework to exploit LVG signals for assisting HVG-centric spatial transcriptomics clustering, effectively capturing complementary structural and contextual cues. Experiments on multiple benchmarks demonstrate its effectiveness, robustness, and transferability. Case studies further confirm that LVG-derived structure enhances biological interpretability by revealing coherent spatial and cellular patterns.
Integrating multimodal learning (ML) with polysomnography (PSG) has emerged as a research hotspot for reliable sleep staging. However, the complexity of these signals and the discomfort associated with wearing multi-lead devices somewhat limit the feasibility of daily and ubiquitous sleep monitoring. Unfortunately, most existing ML paradigms are constrained by consistent and fixed input patterns. When the number of modalities is less than required by the ML framework, it is easy to cause inference bias, resulting in a significant performance degradation. To this end, we propose an elastic multimodal sleep staging network (ElaSleepNet), consisting of multimodal information completion (MIC) and adaptive cross-modal (ACM) interaction. Specifically, MIC maximizes the consistency of multimodal signals on intra-epoch temporal level and inter-epoch contextual level, thereby enhancing the reasoning and completion abilities of the available modalities for the unavailable modalities. Moreover, we introduce learnable parameters and design the ACM attention mechanism, which allows handling multimodal information interaction while maintaining robustness in the absence of certain modalities. Our ElaSleepNet demonstrates its state-of-the-art on three multimodal sleep datasets. Compared with previous methods, ElaSleepNet can achieve better performance with fewer testing modalities, making it flexible for daily monitoring.
Deep multi-view graph clustering seeks to integrate diverse graph feature sets and uncover consistent information across multiple views. While extensive prior research has utilized various neural network architectures to address multi-view graph clustering challenges, these approaches exhibit notable limitations: 1) The ''black-box'' nature of deep learning models, which obscures their internal mechanisms and impedes interpretability; 2) Insufficient efforts aim to capture low-dimensional representations through graphs that reflect intuitive clustering structures and reduce computational cost. To address these limitations, this paper introduces an interpretable multi-view graph clustering framework constructed with optimization-inspired modules. The proposed approach formulates low-dimensional clustering representation learning from graph matrices as an optimization problem, deriving an iterative solution rooted in this formulation. By seamlessly bridging this optimization process to a deep network architecture, the model learns a low-dimensional clustering representation for graph-structured data across multiple views while adhering to the iterative optimization principles and reducing computational costs. This transparent network design enhances the interpretability of multi-view clustering, enabling intuitive and human-understandable learning of clustering structures. Extensive experimental evaluations validate the proposed framework's superiority over state-of-the-art methods in multi-view clustering tasks while ensuring interpretability and reducing computational costs.
Developing computer-aided design (CAD) generation models has significantly enhanced design efficiency, facilitating innovation and transformation in the design industry. Existing methods typically require users to input prompts in a specific format, such as text descriptions or images, limiting their broader application in diverse scenarios. To address this limitation, we introduce FreeCAD, a user-friendly CAD generation framework that supports free-form inputs, including text descriptions and/or images, enabling users to express their design intentions more flexibly. Specifically, we propose a Large Language Models (LLMs)-based Text Translator, which effectively increases the success rate of generating CAD models by converting users' diversified requests for the same object into a unified expression. Additionally, the Multi-View Representation Fusion (MVRF) module enables the network to capture richer interaction information across views, facilitating the generation of more fine-grained CAD models. To support the training of FreeCAD, we construct a multimodal dataset RealCAD, comprising text, image, and CAD triplets, where the images are derived from the 3D printed products of CAD models. Extensive experiments demonstrate that FreeCAD consistently outperforms the existing state-of-the-art (SOTA) methods in multiple tasks.
Topological structures in image data, such as connected components and loops, play a crucial role in understanding image content (e.g., biomedical objects). Despite remarkable successes of numerous image processing methods that rely on appearance information, these methods often lack sensitivity to topological structures when used in general deep learning (DL) frameworks. In this paper, we introduce a new general approach, called TopoImages (for Topology Images), which computes a new representation of input images by encoding local topology of patches. In TopoImages, we leverage persistent homology (PH) to encode geometric and topological features inherent in image patches. Our main objective is to capture topological information in local patches of an input image into a vectorized form. Specifically, we first compute persistence diagrams (PDs) of the patches, and then vectorize and arrange these PDs into long vectors for pixels of the patches. The resulting multi-channel image-form representation is called a TopoImage. TopoImages offers a new perspective for data analysis. To garner diverse and significant topological features in image data and ensure a more comprehensive and enriched representation, we further generate multiple TopoImages of the input image using various filtration functions, which we call multi-view TopoImages. The multi-view TopoImages are fused with the input image for DL-based classification, with considerable improvement. Our TopoImages approach is highly versatile and can be seamlessly integrated into common DL frameworks. Experiments on three public medical image classification datasets demonstrate noticeably improved accuracy over state-of-the-art methods.
Test-Time Adaptation (TTA) enables pre-trained models to bridge the gap between source and target datasets using unlabeled test data, addressing domain shifts caused by corruptions like weather changes, noise, or sensor malfunctions in test time. Multi-Modal Continual Test-Time Adaptation (MM-CTTA), as an extension of standard TTA, further allows models to handle multi-modal inputs and adapt to continuously evolving target domains. However, MM-CTTA faces critical challenges such as catastrophic forgetting and reliability bias, which are rarely addressed effectively under multi-modal corruption scenarios. In this paper, we propose a novel approach, Multi-modality Dynamic Analytic Adapter (MDAA), to tackle MM-CTTA tasks. MDAA introduces analytic learning-a closed-form training technique-through Analytic Classifiers (ACs) to mitigate catastrophic forgetting. Furthermore, we design the Dynamic Late Fusion Mechanism (DLFM) to dynamically select and integrate reliable information from different modalities. Extensive experiments show that MDAA achieves state-of-the-art performance across the proposed tasks. Supplementary materials and codes are available at https://github.com/FeeFee-1/Analytic-Continual-Test-Time-Adaptation-for-Multi-Modality-Corruption this https URL.
Skeleton-based human action recognition has wide applications in video understanding and virtual reality. However, most existing methods focus excessively on spatial location and global movement, while underrepresenting subtle and local actions. To address the limitation, we innovatively propose a Kinematic Enhanced Hypergraph Convolutional Network(KEHCN) with LLM training guides. The network mainly consists of LLM Training Guides(LTG), Kinematic Hypergraph Convolution(KHC), and Kinematic Gating Module(KGM). Specifically, we use the hypergraph convolutional network to extract high-order correlated human skeleton features, the KHC to encode the kinematic features and the LTG to provide a pre-trained large language model to generate text and kinematic description features during the training phase. Based on the Mixture of Experts (MoE) framework, we simplify the gating network by introducing a kinematic feature threshold, thereby constructing a dual-branch global and local motion expert network (KGM). We integrated kinematic features into KHC, LTG and KGM to seek improvements from three perspectives, all of which have enhanced the performance. The experiments on three benchmark datasets(NTU RGB+D, NTU-RGB+D 120 and NW-UCLA), demonstrate the state-of-the-art performance compared to current open-source methods.