Small-scale object detection remains a major challenge in semi-supervised object detection (SSOD), particularly in medical image analysis. Conventional teacher models often struggle to accurately capture the features of low-contrast small lesions, leading to noisy pseudo-labels in both localization and classification, which introduces severe uncertainty and degrades detection performance. To address this issue, we propose Dual Teacher, a novel multimodal semi-supervised detection framework designed to enhance pseudo-label reliability and improve small-scale lesion detection. Specifically, we introduce two complementary teacher models: Hybrid-Scale Teacher, which exploits downsampled views to strengthen multi-scale feature learning, and Entropy-Based Multi-Modal Teacher, which leverages entropy maps to refine the quality of small-scale pseudo-labels. To effectively fuse predictions from both teachers and resolve conflicts, we propose a Dempster-Shafer-based Dual-Teacher pseudo-label fusion strategy that explicitly models uncertainty and optimizes classification confidence. Additionally, we introduce a class-adaptive threshold mechanism that dynamically adjusts pseudo-label selection based on dual-teacher predictions, further boosting the recall of small-scale lesions. Extensive experiments on the Dental Disease Dataset, ChestX-Det and M3FD demonstrate that our method consistently surpasses state-of-the-art SSOD approaches. Code is available at: https://github.com/z316910/Dual-Teacher.git.
论文检索
输入标题、作者或关键词，从 1,620 篇学术成果中精准定位
Multimodal federated learning (MFL) focuses on integrating distributed multimodal data from different clients to improve feature representation while preserving user privacy and has gained popularity in medical image analysis. Existing research idealizes communication efficiency and generalization ability as independent optimization objectives, leading to the failure in the trade-off between model generalization and client resource constraints. To address this challenge, we propose a lightweight Federated Learning with Mid-Frequency consensus-Driven (FedMFD) method, aiming to efficiently generalize multimodal medical image segmentation tasks while reducing communication costs. In the aspect of communication efficiency, client-side images are converted via Discrete Cosine Transform (DCT) from spatial domain into frequency domain coefficients. With only the mid-frequency components selected for transmission. Regarding generalization capability, we adopt the Earth Mover's Distance (EMD) to quantify the maximum similarity of mid-frequency features between clients, generating a global frequency consensus with the optimal transport plan. Guided by global frequency consensus, client-side structural and detailed representations are reconstructed to improve segmentation generalization in the presence of modality shifts and background noise. Extensive experiments across Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) modality datasets have verified the superiority of FedMFD against its competitors.
Pan-sharpening aims to improve the spatial resolution of low-resolution multispectral (LRMS) image by integrating high-frequency information from corresponding texture-rich panchromatic (PAN) image. While RWKV architecture has demonstrated remarkable global perception with linear computational efficiency in vision tasks, its inherent sequential scanning mechanism critically compromises local spatial coherence, hindering high-frequency reconstruction. To bridge this gap, we tailor Freq-RWKV, the first spatial-frequency adaptive RWKV featuring dual-domain scanning where wavelet-guided path selection dynamically modulates scanning granularity and orientation according to spectral-spatial information density distributions. Building upon this innovation, we architect the hierarchical U-shaped fusion network that strategically coordinates granularity-aware scanning across spatial and frequency domains, enabling adaptive trade-offs between performance and complexity. The U-shaped architecture implements coarse-to-fine enhancement: in the encoding stage, Coarse Structural Interaction (CSI-RWKV) module preserves geometric dependencies via window-constrained recurrent scanning while encoding structural priors into LRMS features; during decoding, the Fine-grained Frequency Interaction (FFI-RWKV) module performs edge-aware refinement through differentiable frequency-adaptive window partitioning, where multi-scale spectral wavelet attention prioritizes high-frequency PAN components extracted via discrete wavelet transform (DWT). This hybrid decomposition strategy maintains spectral integrity through approximation coefficients while detail coefficients regulate frequency-gated fusion thresholds. Extensive experiments on multiple satellite datasets validate the effectiveness of the proposed method.
Class Incremental Learning (CIL) aims to continually learn new classes from a stream of data without forgetting previously learned ones. Recent approaches have leveraged pre-trained models (PTMs) to improve performance, especially vision-language models, which offer better generalization than models trained solely on visual data. Many of these methods rely on simple language templates to generate class representations, which then serve as classifiers. However, due to differences between the pre-training data and downstream tasks, these textual features can become too similar for certain classes, leading to prediction errors. To address this issue, we propose a method that optimizes the geometric structure of both visual and textual features across different classes. Inspired by neural collapse theory, we introduce a multi-modal alignment strategy: for each class, a reference vector is chosen from a simplex Equiangular Tight Frame, and both the visual and textual features of the class are aligned with this vector. To better capture intra-class variations, we also construct multiple visual prototypes for each class. A multi-prototype supervised contrastive loss is then employed to pull an image feature toward the closest matching prototype of its true class and push it away from prototypes of other classes. We evaluate our approach on five widely used CIL benchmarks. The results show that our method achieves state-of-the-art performance, demonstrating its effectiveness in addressing the challenges of class incremental learning. Our code is available at https://github.com/qcNPU/NCSCMP.
The parcellation of Cranial Nerves (CNs) serves as a crucial quantitative methodology for evaluating the morphological characteristics and anatomical pathways of specific CNs. Multi-modal CNs parcellation networks have achieved promising segmentation performance, which combine structural Magnetic Resonance Imaging (MRI) and diffusion MRI. However, insufficient exploration of diffusion MRI information has led to low performance of existing multi-modal fusion. In this work, we propose a tractography-guided Dual-label Collaborative Learning Network (DCLNet) for multi-modal CNs parcellation. The key contribution of our DCLNet is the introduction of coarse labels of CNs obtained from fiber tractography through CN atlas, and collaborative learning with precise labels annotated by experts. Meanwhile, we introduce a Modality-adaptive Encoder Module (MEM) to achieve soft information swapping between structural MRI and diffusion MRI. Extensive experiments conducted on the publicly available Human Connectome Project (HCP) dataset demonstrate performance improvements compared to single-label network. This systematic validation underscores the effectiveness of dual-label strategies in addressing inherent ambiguities in CNs parcellation tasks.
Multimodal Anomaly Detection (MMAD) has attracted significant attention in industrial defect inspection as it can simultaneously leverage the complementary information from different modalities to achieve higher-precision detection. Among existing MMAD approaches, dual-branch reverse distillation is widely adopted because of its efficiency in avoiding large-scale data storage. However, it suffers from two key issues. First, the alignment of cross-modal features can lead to a loss of modality-specific characteristics. Second, when one modality indicates normal while another shows anomalies, anomaly detection may be misled by that modality ambiguity. To address these challenges, we propose a Frequency-Aware Multimodal Reverse Distillation (FAMRD) framework from the frequency domain perspective. Specifically, we introduce a frequency spectral feature alignment module that aligns the low- and medium-frequency components across modalities to preserve global shape consistency, while maintaining high-frequency modality-specific details. In addition, we design a frequency spectral anomaly synthesis module. It perturbs the normal feature of one modality to create modality consistent anomalies, fuses it with another modality normal feature to mimic modality ambiguous anomalies, and adds them to the reverse distillation process for decision boundary optimization. Extensive experiments on standard MMAD benchmarks demonstrate that FAMRD achieves competitive performance in both anomaly detection and localization, outperforming state-of-the-art methods.
Few-shot object detection (FSOD) is an important problem in computer vision, aiming to accurately detect objects with only a few annotated examples. Prototype learning has been widely explored in this field. Some methods extract visual prototypes from support images, but the limited sample size often leads to unrepresentative features. Others use textual prototypes generated by pre-trained vision-language models such as CLIP, which lack visual detail and may suffer from language ambiguity. As visual and textual prototypes offer complementary strengths --- detail and generalization respectively, single-modal prototypes struggle to balance both. To address this issue, we propose MP-DETR, the first multi-modal prototype guided method for FSOD. We design an adaptive multi-modal prototype fusion module to combine visual and textual prototypes from foundation models using a gating mechanism, producing multi-modal class prototypes that retain both general semantics and visual specificity. These prototypes are then deeply integrated into the DETR detection pipeline for guiding potential region selection, enhancing corresponding object queries, and constructing a prototype similarity-based classifier enhanced by contrastive learning to improve discrimination among similar classes. By incorporating multi-modal guidance into the detection process, MP-DETR achieves better performance than existing single-modal methods. Extensive experiments on MS-COCO and Pascal-VOC show that MP-DETR achieves SOTA results in various few-shot settings, confirming its effectiveness and superiority.
Low-light image enhancement aims to improve brightness, suppress noise, and recover accurate color and structure, requiring precise illumination modeling and reliable reflectance recovery. However, most Retinex-based methods adopt explicit, multi-stage pipelines prone to decomposition bias, error accumulation, and chromatic entanglement between illumination and reflectance. To tackle these issues, we propose IDAR (Implicit Decomposition, illumination Adjustment, and reflectance Restoration), a unified Retinex-inspired framework with two key innovations. First, we design an implicit decomposition strategy based on dual-branch feature learning: a low-frequency-constrained illumination branch models lighting with chromaticity awareness, while a contrast-guided reflection branch preserves details by decoupling reflectance from illumination. This implicit design avoids intermediate supervision and reduces decomposition bias. Second, we introduce the Illumination Chromaticity Expansion Module (ICEM), which employs text-guided chromaticity learning to enhance chromaticity perception. By learning a reflectance-independent spectral representation, ICEM reduces color shifts and improves fidelity under complex lighting. Experiments on multiple benchmarks validate the superior visual quality, quantitative performance, and physical interpretability of IDAR.
LIDAR: Lightweight Adaptive Cue-Aware Fusion Vision Mamba for Multimodal Segmentation of Structural Cracks
PDF ↗Achieving pixel-level segmentation with low computational cost using multimodal data remains a key challenge in crack segmentation tasks. Existing methods lack the capability for adaptive perception and efficient interactive fusion of cross-modal features. To address these challenges, we propose a Lightweight Adaptive Cue-Aware Vision Mamba network (LIDAR), which efficiently perceives and integrates morphological and textural cues from different modalities under multimodal crack scenarios, generating clear pixel-level crack segmentation maps. Specifically, LIDAR is composed of a Lightweight Adaptive Cue-Aware Visual State Space module (LacaVSS) and a Lightweight Dual Domain Dynamic Collaborative Fusion module (LD3CF). LacaVSS adaptively models crack cues through the proposed mask-guided Efficient Dynamic Guided Scanning Strategy (EDG-SS), while LD3CF leverages an Adaptive Frequency Domain Perceptron (AFDP) and a dual-pooling fusion strategy to effectively capture spatial and frequency-domain cues across modalities. Moreover, we design a Lightweight Dynamically Modulated Multi-Kernel convolution (LDMK) to perceive complex morphological structures with minimal computational overhead, replacing most convolutional operations in LIDAR. Experiments on three datasets demonstrate that our method outperforms other state-of-the-art (SOTA) methods. On the light-field depth dataset, our method achieves 0.8204 in F1 and 0.8465 in mIoU with only 5.35M parameters. Code and datasets are available at https://github.com/Karl1109/LIDAR-Mamba.
Multi-view multi-label learning (MVML) is a significant area of research in multimedia, providing a foundational framework for various real-world applications. However, the richness of its descriptive capabilities often results in high-dimensional data that contains redundant information, negatively affecting model performance. Most existing methods do not thoroughly explore the mapping of the distinctive parts while balancing common and distinctive information. Additionally, there has been limited focus on the relationships between different types of mappings and the high-order constraints among view-specific labels. In this paper, we propose a novel tensor-based method for view-specific label learning that integrates adaptive weight mechanisms into both global non-linear and local linear mappings. This method effectively captures high-order relationships among views and hybrid labels through hierarchical label correlation constraints. Central to our model is the ''Opposing yet Complementary'' procedure, which enhances feature weight representation at a finer-grained level. Extensive experiments on widely used multi-view multi-label datasets demonstrate significant performance improvements, underscoring the effectiveness of our proposed method.
Dynamic scene reconstruction from monocular video is essential for real-world applications. We introduce DGNS, a hybrid framework integrating Deformable Gaussian Splatting and Dynamic Neural Surfaces, effectively addressing dynamic novel-view synthesis and 3D geometry reconstruction simultaneously. During training, depth maps generated by the deformable Gaussian splatting module guide the ray sampling for faster processing and provide depth supervision within the dynamic neural surface module to improve geometry reconstruction. Conversely, the dynamic neural surface directs the distribution of Gaussian primitives around the surface, enhancing rendering quality. In addition, we propose a depth-filtering approach to further refine depth supervision. Extensive experiments conducted on public datasets demonstrate that DGNS achieves state-of-the-art performance in 3D reconstruction, along with competitive results in novel-view synthesis.
Generating interactive motion from texts has garnered significant attention in recent years. While text inputs offer greater flexibility, in many practical applications, there is a need to controllably impose strict constraints on the motion range or trajectory of virtual characters. However, existing trajectory-based methods are designed for single-actor scenarios and lack support for interactivity in interactive motions. Moreover, text-only methods struggle to accurately convey user-intended trajectories. The distribution shift between training and inference often leads to trajectory deviation and physical interpenetration. To address the questions mentioned, we introduce two key concepts: (1) Lead-Follow Paradigm: Inspired by role allocation in partner dancing, we decompose complex interactive motion tasks into a Lead-Follow paradigm. The leader's path is optimized first, and the follower's motion is subsequently adjusted for coherence and alignment. (2) Trajectory Guidance: We highlight the pivotal role of 3D trajectory guidance in interactive motion generation and accurately reflect user intentions. Through 3D trajectory control, we can more controllably generate the desired motion while avoiding physical interpenetration. In addition, we further investigate the refinement of motion scopes for interactive agents and propose an effective optimization strategy to enhance motion coherence and controllability. Experimental results show that the proposed approach, by more effectively using trajectory, outperforms existing methods in both realism and accuracy.
In recent years, the rampant spread of misinformation on social media has made accurate detection of multimodal fake news a critical research focus. However, previous research has not adequately understood the semantics of images, and models struggle to discern news authenticity with limited textual information. Meanwhile, treating all emotional types of news uniformly without tailored approaches further leads to performance degradation. Therefore, we propose a novel Knowledge Augmentation and Emotion Guidance Network (KEN). On the one hand, we effectively leverage LVLM's powerful semantic understanding and extensive world knowledge. For images, the generated captions provide a comprehensive understanding of image content and scenes, while for text, the retrieved evidence helps break the information silos caused by the closed and limited text and context. On the other hand, we consider inter-class differences between different emotional types of news through balanced learning, achieving fine-grained modeling of the relationship between emotional types and authenticity. Extensive experiments on two real-world datasets demonstrate the superiority of our KEN.
Multimodal large language models (MLLMs) have garnered widespread attention due to their ability to understand multimodal input. However, their large parameter sizes and substantial computational demands severely hinder their practical deployment and application. While quantization is an effective way to reduce model size and inference latency, its application to MLLMs remains underexplored. In this paper, we propose MQuant, a post-training quantization (PTQ) framework designed to tackle the unique challenges of multimodal large language models (MLLMs). Conventional quantization often struggles with MLLMs because of (a) high inference latency from large visual token counts, (b) distributional disparities between visual and textual tokens, and (c) extreme outliers introduced by Hadamard-based transformations. To address these issues, MQuant introduces: • Modality-Specific Static Quantization (MSQ), assigning distinct static scales for visual vs. textual tokens; • Attention-Invariant Flexible Switching (AIFS), reordering tokens to preserve casual attention while eliminating expensive token-wise scale computations; • Rotation Magnitude Suppression (RMS), mitigating weight outliers arising from online Hadamard rotations. On five mainstream MLLMs (including Qwen-VL, MiniCPM-V, CogVLM2), MQuant under W4A8 achieves near-floating-point accuracy (<1% degradation) while reducing inference latency by up to 30%, significantly outperforming existing PTQ baselines. Our MQuant effectively bridges the gap for efficient and accurate MLLMs inference in resource-constrained devices. Code will be released in https://github.com/StiphyJay/MQuant.
The 4D millimeter wave radar has gained increasing attention in autonomous driving due to its robustness against environmental interference compared to other perception devices such as cameras or LiDAR. However, the practical deployment of radar remains challenging due to the noise and high sparsity of radar point clouds, as well as the resource limitations of edge computing. To address these issues, we propose Radar-Mamba, a lightweight and efficient radar enhancement approach for boosting radar perception. The proposed approach contains three main components: cross-modal alignment, radar enhancement architecture based on the Mamba model, and Doppler feature fusion. Specifically, the point clouds of radar and LiDAR are first refined and aligned to build more dense and rich occupancies. The aligned 4D radar data is then enhanced by capturing both local and global spatial-temporal features, while integrating radar-specific velocity and elevation information for further denoising. Experiments on two open-source datasets demonstrate that our method achieves state-of-the-art performance and generates high-quality 4D point clouds with a density comparable to LiDAR while maintaining a low parameter count friendly for practical deployment.
Although object counting based on two-dimensional (2D) RGB images offers an effective solution in certain scenarios, it is significantly challenged by complex environments characterized by background noise, occlusion, depth variations, illumination changes, and other factors, often resulting in miscounting or missed detections. This issue is particularly pronounced in applications such as agriculture or retail, where fruits or products are frequently stacked in layers on shelves, severely compromising counting accuracy. To address these limitations, we propose TrueCount, a novel method that integrates segmented images, depth information, and other multi-modal data to enhance counting accuracy and robustness of pretrained large vision-language models (VLMs). TrueCount introduces a flexible framework capable of simultaneously processing multiple modal signals, including 2D RGB images, segmentation, and depth maps, while supporting both textual and visual prompts. During training and inference, TrueCount performs cross-attention and self-attention across all inputs and prompts. These features are then decoded to localize prompt features within the input, thereby jointly optimizing the model's counting capability. Additionally, TrueCount dynamically assesses the confidence of each modality for accurate counting in a given context, enabling effective fusion and complementary utilization of multi-modal information. Extensive experiments on multiple benchmarks including FSC-147 and CountBench demonstrate that TrueCount surpasses the previous state-of-the-art, e.g. achieving a new minimum mean average error of 4.64 on FSC-147.
Model Diagram-to-Code Generation aims to translate model diagrams from research papers into implementation code that reconstructs the model's architecture. This task plays a crucial role in accelerating scientific workflows and enhancing the efficiency of industrial model deployment. While recent studies have explored various Image-to-Code Generation tasks using Multimodal Large Language Models (MLLMs), these efforts have primarily focused on reconstructing the visual appearance depicted in input images, leaving this task largely underexplored. The complex structural elements and implicit relationships in model diagrams present greater challenges for MLLMs, particularly in terms of visual reasoning and semantic interpretation. To support this task, we introduce MDCDataset, a dataset designed to evaluate the ability of MLLMs to generate code from model diagrams. It comprises 1,008 instances spanning 16 research domains, each with a model diagram, structured textual content, and the ground-truth code implementation. Furthermore, to address the inherent challenges of this task, we propose MDCAgent, a collaborative multi-agent framework composed of Parsing, Generation, and Check Agents. These agents work in coordination to analyze, extract, and verify complex elements and implicit relationships within model diagrams, thereby enhancing the visual architecture-aware reasoning capabilities of MLLMs. Our extensive experiments confirm the effectiveness of the framework.
Large language models (LLMs) have shown great potential in the medical domain. However, existing models still fall short when faced with complex medical diagnosis task in the real world. This is mainly because they lack sufficient reasoning depth, which leads to information loss or logical jumps when processing a large amount of specialized medical data, leading to diagnostic errors. To address these challenges, we propose Tree-of-Reasoning (ToR), a novel multi-agent framework designed to handle complex scenarios. Specifically, ToR introduces a tree structure that can clearly record the reasoning path of LLMs and the corresponding clinical evidence. At the same time, we propose a cross-validation mechanism to ensure the consistency of multi-agent decision-making, thereby improving the clinical reasoning ability of multi-agents in complex medical scenarios. Experimental results on real-world medical data show that our framework can achieve better performance than existing baseline methods.
With the rapid development of short video platforms (such as Kuaishou and TikTok), these platforms have increasingly become important channels for the spread of fake news.Therefore, multi-modal fake news detection has attracted extensive attention.Existing studies mainly focus on directly integrating multi-modal information or discovering implicit clues in posts to improve detection performance.However, due to the abuse of video editing techniques, event-irrelevant segments (e.g., advertisements) are frequently mixed into videos, introducing noise information, thereby weakening models' ability to learn crucial information.Moreover, video creators often inject personal tampered information into original news content through audio modality manipulation, potentially distorting the factual. To address these challenges, we propose a novel Event Consistency-aware Robust Fake News Detection (ECR-FND) framework, comprising two key components: an Event-aware Video Denoising Learning (EVDL) and an Audio Tampering-information Capturing Module (ATCM).Specifically, the EVDL filters out the event-irrelevant segments within video modality to focus on core news events. The ATCM adaptively amplifies tampering information in audio modality, enhancing the model's capacity to detect manipulation attempts.Extensive experiments on two benchmark datasets (FakeSV and FakeTT) demonstrate ECR-FND's effectiveness.Our source code is available at https://github.com/immc-lab/ECR-FND.
Hyperspectral image super-resolution (HSI-SR) has attracted significant attention in high-resolution HSI reconstruction. Most existing fusion-based HSI-SR methods assume that multi-source images are perfectly registered, which is impractical due to varying imaging conditions. Furthermore, methods that consider the registration issue typically treat registration and fusion as two separate steps, resulting in the accumulation of registration errors during the fusion process. To address these issues, we propose a Cycle-Consistent Mamba-Based Registration-Fusion Joint Network (CCM-RFJN), which step-wise optimizes the Registration-Fusion Unified Module (RFUM) through multiple cyclic iterative SR processes. Specifically, in each SR iteration, we map the super-resolved HR-HSI obtained through the RFUM back to the unregistered LR-HSI for the next SR, with cycle-consistency constraints imposed on both LR-HSI and HR-HSI to adaptively optimize the RFUM based on the reciprocal training strategy. In RFUM, we integrate the proposed Interactive Mamba Registration Module (IMR) and Dual-attention Mamba Fusion Module (DAMF), thereby achieving registration-fusion joint optimization. Specifically, IMR is developed to incorporate the interactive Mamba encoder into a pyramid architecture to facilitate multi-level information interactions, generating the deformation field to correct non-rigid misalignments. DAMF is designed to utilize the dual-attention Mamba mechanism to highlight and aggregate key features, thereby enhancing fusion performance. Experiments on three public datasets demonstrate that CCM-RFJN achieves the state-of-the-art performance. The code is available at https://github.com/Jiahuiqu/CCM-RFJN.