We aim to develop a robust yet flexible visual foundation model for Earth observation. It should possess strong capabilities in recognizing and localizing diverse visual targets while providing compatibility with various input-output interfaces required across different task scenarios. Current systems cannot meet these requirements, as they typically utilize task-specific architecture trained on narrow data domains with limited semantic coverage. Our study addresses these limitations from two aspects: data and modeling. We first introduce an automatic data engine that enjoys significantly better scalability compared to previous human annotation or rule-based approaches. It has enabled us to create the largest dataset of its kind to date, comprising 270K image-text-mask triplets covering an unprecedented range of diverse semantic categories and attribute specifications. Based on this data foundation, we further propose a task unification paradigm that centers around referring expression segmentation. It effectively handles a wide range of vision-centric perception tasks, including classification, detection, segmentation, grounding, etc, using a single model without any task-specific heads. Combining these innovations on data and modeling, we present RemoteSAM, a foundation model that establishes new SoTA on several earth observation perception benchmarks, outperforming other foundation models such as Falcon, GeoChat, and LHRS-Bot with significantly higher efficiency. Models and data are publicly available at https://github.com/1e12Leon/RemoteSAM.
论文检索
输入标题、作者或关键词，从 5,999 篇学术成果中精准定位
Medical image restoration tasks aim to recover high-quality images from degraded observations, exhibiting emergent desires in many clinical scenarios, such as low-dose CT image denoising, MRI super-resolution, and MRI artifact removal. Despite the success achieved by existing deep learning-based restoration methods with sophisticated modules, they struggle with rendering computationally-efficient reconstruction results. Moreover, they usually ignore the reliability of the restoration results, which is much more urgent in medical systems. To alleviate these issues, we present LRformer, a Lightweight Transformer-based method via Reliability-guided learning in the frequency domain. Specifically, inspired by the uncertainty quantification in Bayesian neural networks (BNNs), we develop a Reliable Lesion-Semantic Prior Producer (RLPP). RLPP leverages Monte Carlo (MC) estimators with stochastic sampling operations to generate sufficiently-reliable priors by performing multiple inferences on the foundational medical image segmentation model, MedSAM. Additionally, instead of directly incorporating the priors in the spatial domain, we decompose the cross-attention (CA) mechanism into real symmetric and imaginary anti-symmetric parts via fast Fourier transform (FFT), resulting in the design of the Guided Frequency Cross-Attention (GFCA) solver. By leveraging the conjugated symmetric property of FFT, GFCA reduces the computational complexity of naive CA by nearly half. Extensive experimental results in various tasks demonstrate the superiority of the proposed LRformer in both effectiveness and efficiency.
Recent advances in Weakly Supervised Semantic Segmentation (WSSS) focus on generating high-quality Class Activation Maps (CAMs) using image-level labels. However, the co-occurrence of foreground-background concepts in a single image often induces semantic confusion, which degrades the quality of conventional CAM-based approaches. In this paper, we propose VLHP, a novel framework that leverages vision-language hybrid prototypes to overcome semantic confusion. Specifically, VLHP constructs hybrid prototypes through cross-modal association between textual embeddings and visual features, generating discriminative semantic representations while effectively bridging the modality gap. To further improve discriminability, we introduce two dedicated strategies: Discriminative Explicit Alignment (DEA) to explore cross-modal consistent discrimination and Confounding Background Decoupling (CBD) to model co-occurring backgrounds and decouple them. Finally, a Prototype-driven Class-aware Decoder (PCD) employs these refined prototypes as category-specific priors to generate precise segmentation masks in a single-stage framework. Extensive experiments on PASCAL VOC and MS COCO benchmarks demonstrate that VLHP outperforms state-of-the-art alternatives. The code is available at https://github.com/fjy0105/VLHP.
Open-vocabulary semantic segmentation (OVSS) conducts pixel-level classification via text-driven alignment, where the domain discrepancy between base category training and open-vocabulary inference poses challenges in discriminative modeling of latent unseen category. To address this challenge, existing vision-language model (VLM)-based approaches demonstrate commendable performance through pre-trained multi-modal representations. However, the fundamental mechanisms of latent semantic comprehension remain underexplored, making the bottleneck for OVSS. In this work, we initiate a probing experiment to explore distribution patterns and dynamics of latent semantics in VLMs under inductive learning paradigms. Building on these insights, we propose X-Agent, an innovative OVSS framework employing latent semantic-aware ''agent'' to orchestrate cross-modal attention mechanisms, simultaneously optimizing latent semantic dynamic and amplifying its perceptibility. Extensive benchmark evaluations demonstrate that X-Agent achieves state-of-the-art performance while effectively enhancing the latent semantic saliency.
Counting by Points: Density-Guided Weakly-Supervised Nuclei Segmentation in Histopathological Images
Nuclei instance segmentation in histopathological images is pivotal for cancer diagnosis, yet heavily reliant on costly pixel-level annotations. While point-supervised methods reduce annotation burdens, existing approaches struggle to reconstruct accurate nuclear boundaries and fail to leverage spatial distribution cues, particularly in dense or irregularly structured tissues. This paper presents DeNSe, a novel density-guided weakly-supervised framework that integrates nuclei counting with instance segmentation to address these limitations. Unlike conventional methods, DeNSe introduces a multi-task learning architecture, combining density regression and segmentation through two key innovations: (1) a distribution-aware alignment module based on optimal transport theory, which harmonizes feature representations between density maps and segmentation masks to enhance instance differentiation; and (2) a morphology-aware refinement strategy that dynamically adjusts pseudo-labels using density-guided confidence scores, mitigating errors from coarse annotations. By formulating counting as a density regression task, DeNSe captures global nuclei distribution, enabling robust segmentation in densely packed regions. Extensive experiments on three benchmarks demonstrate state-of-the-art performance, achieving significant improvements across various metrics over existing methods. Notably, DeNSe exhibits strong robustness to annotation offsets and generalizes across diverse tissue types, offering a cost-effective and scalable tool for clinical applications. The code is available at https://github.com/lingboboo/DENSE.
Open vocabulary image segmentation tackles the challenge of recognizing dynamically adjustable, predefined novel categories at inference time by leveraging vision-language alignment. However, existing paradigms typically perform class-agnostic region segmentation followed by category matching, which deviates from the human visual system's process of recognizing objects based on semantic concepts, leading to poor alignment between region segmentation and object concepts. To bridge this gap, we propose a novel Cognition-Inspired Framework for open vocabulary image segmentation that emulates the human visual recognition process: first forming a conceptual understanding of an object, then perceiving its spatial extent. The framework consists of three core components: (1) A Generative Vision-Language Model (G-VLM) that mimics human cognition by generating object concepts to provide semantic guidance for region segmentation. (2) A Concept-Aware Visual Enhancer module that fuses textual concept features with global visual representations, enabling adaptive visual perception based on object concepts. (3) A Cognition-Inspired Mask Decoder that integrates local instance features with G-VLM-provided semantic cues, allowing selective classification over a subset of relevant categories. Extensive experiments demonstrate that our framework achieves significant improvements, reaching 27.2 PQ, 17.0 mAP, and 35.3 mIoU on A-150. It further attains 56.2, 28.2, 15.4, 59.2, 18.7, and 95.8 mIoU on Cityscapes, Mapillary Vistas, A-847, PC-59, PC-459, and PAS-20, respectively. In addition, our framework supports vocabulary-free image segmentation, offering enhanced flexibility in recognizing unseen categories.
Despite progress in pixel-level medical image perception, existing methods remain task-specific or depend on precise prompts like bounding boxes or text. However, the need for medical knowledge limits accessibility for the general public, who are more likely to use logically reasoned oral queries than domain-specific inputs. In this paper, we introduce a novel medical vision task: Medical Reasoning Segmentation and Detection (MedSD), which aims to comprehend implicit queries about medical images and generate the corresponding segmentation mask and bounding box for the target object. To accomplish this task, we first introduce a Multi-perspective, Logic-driven Medical Reasoning Segmentation and Detection (MLMR-SD) dataset, which encompasses a substantial collection of medical entity targets along with their corresponding reasoning. Furthermore, we propose MediSee, an effective baseline model designed for MedSD. The experimental results indicate that the proposed method can effectively address MedSD with implicit colloquial queries and outperform traditional medical referring segmentation methods. The MediSee project can be found here.
Anomaly segmentation aims to identify Out-of-Distribution (OoD) anomalous objects within images. Existing pixel-wise methods typi- cally assign anomaly scores individually and employ a global thresh- olding strategy to segment anomalies. Despite their effectiveness, these approaches encounter significant challenges in real-world applications: (1) neglecting spatial correlations among pixels within the same object, resulting in fragmented segmentation; (2) variabil- ity in anomaly score distributions across image regions, causing global thresholds to either generate false positives in background areas or miss segments of anomalous objects. In this work, we intro- duce OoDDINO, a novel multi-level anomaly segmentation frame- work designed to address these limitations through a coarse-to-fine anomaly detection strategy. OoDDINO combines an uncertainty- guided anomaly detection model with a pixel-level segmentation model within a two-stage cascade architecture. Initially, we propose an Orthogonal Uncertainty-Aware Fusion Strategy (OUAFS) that sequentially integrates multiple uncertainty metrics with visual representations, employing orthogonal constraints to strengthen the detection model's capacity for localizing anomalous regions accurately. Subsequently, we develop an Adaptive Dual-Threshold Network (ADT-Net), which dynamically generates region-specific thresholds based on object-level detection outputs and pixel-wise anomaly scores. This approach allows for distinct thresholding strategies within foreground and background areas, achieving fine- grained anomaly segmentation. The proposed framework is compatible with other pixel-wise anomaly detection models, which act as a plug-in to boost the performance. Extensive experiments on two benchmark datasets validate our framework's superiority and compatibility over state-of-the-art methods. Source code is available at: https://github.com/OoDDINO/OoD-DINO.
Multimodal learning benefits from multiple modal information, and each learned modal representations can be divided into uni-modal that can be learned from uni-modal training and paired-modal features that can be learned from cross-modal interaction. Building on this perspective, we propose a partitioner-guided modal learning framework, PgM, which consists of the modal partitioner, uni-modal learner, paired-modal learner, and uni-paired modal decoder. Modal partitioner segments the learned modal representation into uni-modal and paired-modal features. Modal learner incorporates two dedicated components for uni-modal and paired-modal learning. Uni-paired modal decoder reconstructs modal representation based on uni-modal and paired-modal features. PgM offers three key benefits: 1) thorough learning of uni-modal and paired-modal features, 2) flexible distribution adjustment for uni-modal and paired-modal representations to suit diverse downstream tasks, and 3) different learning rates across modalities and partitions. Extensive experiments demonstrate the effectiveness of PgM across four multimodal tasks and further highlight its transferability to existing models. Additionally, we visualize the distribution of uni-modal and paired-modal features across modalities and tasks, offering insights into their respective contributions.
Whole-body PET tumor segmentation remains challenging due to limited training data and substantial tumor heterogeneity, which impact the segmentation accuracy and the clinical utility. Tumor distribution information is usually contained in patient medical records and routinely utilized in medical image interpretation, which can also be used to improve the segmentation accuracy. This study introduces a novel 3D PET/MR tumor segmentation framework which integrates tumor distribution priors extracted from medical records. The proposed Tumor Localization Priors(TLP) are generated based on medical records using Large Language Models (LLMs) and organ localization based on MRI. Furthermore, the Region-Aware Fusion Module (RAFM) is designed to fuse TLP and encoded PET information through attention within the Anatomically-Consistent Multitask Model(ACMM). Moreover, Anatomical Consistency Loss(AC Loss) is introduced, integrating tumor localization and its anatomical distribution to enhance segmentation performance. Our method achieves an 9.30% Dice improvement over the baseline nnU-Net v2, with particularly notable 23.06% gains in precision while maintaining high recall (+6.19%). Clinical evaluations confirm superior detection of both primary and metastatic lesions, alongside reduced physiological uptake artifacts.
Multi-modal test-time adaptation (TTA) for 3D semantic segmentation has increasingly become a research hotspot due to its ability to address label dependency and enable rapid adaptation. Existing methods rely on learnable extra components to mitigate reliability bias, however, learning-based approaches in TTA scenarios often lack sufficient training. Moreover, most existing approaches update only normalization layers in the teacher-student framework, which limits their ability to model domain shifts. To overcome these limitations, we propose PLATO-TTA, a novel multi-modal TTA method for 3D semantic segmentation leveraging the native stability in robust prototypes and adaptive tuning of critical teacher-student parameters. The approach contains three key components: Prototype-Guided Pseudo-Labeling (PGPL), Consistency Based Backtracking (CBB), and Domain Specific Updating (DSU). PGPL reduces reliability bias by constructing pseudo-source domain prototypes and computing modality fusion weights based on domain discrepancies. CBB updates all student model parameters while preventing catastrophic forgetting through a parameter backtracking mechanism. DSU selectively updates the teacher model using only domain-specific parameters from the student model, ensuring rapid adaptation and stable guidance. Extensive experiments demonstrate the effectiveness of PLATO-TTA, bringing a 6.3% gain to the SynthiatoSemanticKITTI scenario with severe reliability bias and significant domain discrepancy, and achieve state-of-the-art performance across various domain adaptation scenarios.
Diffusion models have recently shown strong capabilities in image generation. This paper investigates their potential for semantic segmentation, with a focus on RGB-D tasks that demand precise pixel-level predictions. In particular, we delve into the intermediate activations generated during the reverse Markov step of diffusion process, discovering that these activations can effectively capture the semantic information of an input image, making them outstanding representations for addressing segmentation challenges. This paper proposes Diffusion-Enhanced Multi-Modal Segmenter (DiffuSeg), which innovatively combines RGB features with those generated by an additional diffusion model, facilitating the extraction of comprehensive and nuanced semantic features. Furthermore, we propose the Cross Attention-and-Aggregation Module (CAAM), which not only fosters long-range interactions between RGB and diffusion-derived features but also recalibrates both feature sets before integration, enhancing multi-modal synergy. Additionally, our model incorporates a Dynamic Cascade Kernel (DCK) architecture that exploits local and intricate multi-scale geometric details. As a part of DCK, the Spatial Interaction Module (SIM) dynamically encodes spatial information by establishing pixel-level correlations, thereby enhancing the spatial feature representation capacity. Extensive experiments on two benchmark datasets demonstrate the strong capability of DiffuSeg in handling challenging semantic segmentation tasks.
With the popularization of the Internet and the diversification of attack methods, web security has become an important part of information security. As a carrier of network behavior, traffic can reveal attack behaviors in the Web environment through malicious traffic detection. Since images can fully express spatial features and local associations, it is feasible to visualize traffic as images and retrieve key feature information to detect malicious traffic. However, existing methods are prone to redundancy during feature extraction. Secondly, a single perspective makes it difficult to learn patterns with universality from diverse feature information. In addition, the selection of segmentation thresholds in the preprocessing is closely related to the model's information retrieval effect. The commonly adopted preset thresholds are difficult to cope with the changes in traffic data, limiting the applicability of the existing methods. Therefore, this paper proposes a Multi-Module-Based Composite Robust Model for Network Attack Detection (MCNAD). The model adopts depthwise separable convolution (DSC) to reduce redundant information, proposes a multi-scale feature learning module to enhance the model characterization ability, and proposes a gray level co-occurrence matrix segmentation algorithm with adaptive threshold (GLCM-AT) to optimize data preprocessing. The results show that MCNAD improves detection performance with better detection efficiency, generalization ability, and robustness, demonstrating its wide applicability in multiple scenarios.
Multimodal federated learning (MFL) focuses on integrating distributed multimodal data from different clients to improve feature representation while preserving user privacy and has gained popularity in medical image analysis. Existing research idealizes communication efficiency and generalization ability as independent optimization objectives, leading to the failure in the trade-off between model generalization and client resource constraints. To address this challenge, we propose a lightweight Federated Learning with Mid-Frequency consensus-Driven (FedMFD) method, aiming to efficiently generalize multimodal medical image segmentation tasks while reducing communication costs. In the aspect of communication efficiency, client-side images are converted via Discrete Cosine Transform (DCT) from spatial domain into frequency domain coefficients. With only the mid-frequency components selected for transmission. Regarding generalization capability, we adopt the Earth Mover's Distance (EMD) to quantify the maximum similarity of mid-frequency features between clients, generating a global frequency consensus with the optimal transport plan. Guided by global frequency consensus, client-side structural and detailed representations are reconstructed to improve segmentation generalization in the presence of modality shifts and background noise. Extensive experiments across Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) modality datasets have verified the superiority of FedMFD against its competitors.
The parcellation of Cranial Nerves (CNs) serves as a crucial quantitative methodology for evaluating the morphological characteristics and anatomical pathways of specific CNs. Multi-modal CNs parcellation networks have achieved promising segmentation performance, which combine structural Magnetic Resonance Imaging (MRI) and diffusion MRI. However, insufficient exploration of diffusion MRI information has led to low performance of existing multi-modal fusion. In this work, we propose a tractography-guided Dual-label Collaborative Learning Network (DCLNet) for multi-modal CNs parcellation. The key contribution of our DCLNet is the introduction of coarse labels of CNs obtained from fiber tractography through CN atlas, and collaborative learning with precise labels annotated by experts. Meanwhile, we introduce a Modality-adaptive Encoder Module (MEM) to achieve soft information swapping between structural MRI and diffusion MRI. Extensive experiments conducted on the publicly available Human Connectome Project (HCP) dataset demonstrate performance improvements compared to single-label network. This systematic validation underscores the effectiveness of dual-label strategies in addressing inherent ambiguities in CNs parcellation tasks.
LIDAR: Lightweight Adaptive Cue-Aware Fusion Vision Mamba for Multimodal Segmentation of Structural Cracks
PDF ↗Achieving pixel-level segmentation with low computational cost using multimodal data remains a key challenge in crack segmentation tasks. Existing methods lack the capability for adaptive perception and efficient interactive fusion of cross-modal features. To address these challenges, we propose a Lightweight Adaptive Cue-Aware Vision Mamba network (LIDAR), which efficiently perceives and integrates morphological and textural cues from different modalities under multimodal crack scenarios, generating clear pixel-level crack segmentation maps. Specifically, LIDAR is composed of a Lightweight Adaptive Cue-Aware Visual State Space module (LacaVSS) and a Lightweight Dual Domain Dynamic Collaborative Fusion module (LD3CF). LacaVSS adaptively models crack cues through the proposed mask-guided Efficient Dynamic Guided Scanning Strategy (EDG-SS), while LD3CF leverages an Adaptive Frequency Domain Perceptron (AFDP) and a dual-pooling fusion strategy to effectively capture spatial and frequency-domain cues across modalities. Moreover, we design a Lightweight Dynamically Modulated Multi-Kernel convolution (LDMK) to perceive complex morphological structures with minimal computational overhead, replacing most convolutional operations in LIDAR. Experiments on three datasets demonstrate that our method outperforms other state-of-the-art (SOTA) methods. On the light-field depth dataset, our method achieves 0.8204 in F1 and 0.8465 in mIoU with only 5.35M parameters. Code and datasets are available at https://github.com/Karl1109/LIDAR-Mamba.
Although object counting based on two-dimensional (2D) RGB images offers an effective solution in certain scenarios, it is significantly challenged by complex environments characterized by background noise, occlusion, depth variations, illumination changes, and other factors, often resulting in miscounting or missed detections. This issue is particularly pronounced in applications such as agriculture or retail, where fruits or products are frequently stacked in layers on shelves, severely compromising counting accuracy. To address these limitations, we propose TrueCount, a novel method that integrates segmented images, depth information, and other multi-modal data to enhance counting accuracy and robustness of pretrained large vision-language models (VLMs). TrueCount introduces a flexible framework capable of simultaneously processing multiple modal signals, including 2D RGB images, segmentation, and depth maps, while supporting both textual and visual prompts. During training and inference, TrueCount performs cross-attention and self-attention across all inputs and prompts. These features are then decoded to localize prompt features within the input, thereby jointly optimizing the model's counting capability. Additionally, TrueCount dynamically assesses the confidence of each modality for accurate counting in a given context, enabling effective fusion and complementary utilization of multi-modal information. Extensive experiments on multiple benchmarks including FSC-147 and CountBench demonstrate that TrueCount surpasses the previous state-of-the-art, e.g. achieving a new minimum mean average error of 4.64 on FSC-147.
With the rapid development of short video platforms (such as Kuaishou and TikTok), these platforms have increasingly become important channels for the spread of fake news.Therefore, multi-modal fake news detection has attracted extensive attention.Existing studies mainly focus on directly integrating multi-modal information or discovering implicit clues in posts to improve detection performance.However, due to the abuse of video editing techniques, event-irrelevant segments (e.g., advertisements) are frequently mixed into videos, introducing noise information, thereby weakening models' ability to learn crucial information.Moreover, video creators often inject personal tampered information into original news content through audio modality manipulation, potentially distorting the factual. To address these challenges, we propose a novel Event Consistency-aware Robust Fake News Detection (ECR-FND) framework, comprising two key components: an Event-aware Video Denoising Learning (EVDL) and an Audio Tampering-information Capturing Module (ATCM).Specifically, the EVDL filters out the event-irrelevant segments within video modality to focus on core news events. The ATCM adaptively amplifies tampering information in audio modality, enhancing the model's capacity to detect manipulation attempts.Extensive experiments on two benchmark datasets (FakeSV and FakeTT) demonstrate ECR-FND's effectiveness.Our source code is available at https://github.com/immc-lab/ECR-FND.
By clustering pixels with locally similar values, superpixel-based approaches have shown great potential in processing hyperspectral images (HSI) , thereby reducing the computational burden associated with large spatial dimensions. However, specific for spatial-spectral fusion (SSF), superpixel segmentation is inherently non-differentiable and irreversible; hence it is inapplicable. To address the issues, we propose a semantic transformer-based solver, namely SpecSolver, which is basically inspired by the benefits of superpixel-based approaches, yet with the inner mechanism completely improved. The core idea lies in learning the intrinsic semantic states of HSIs hidden behind discretized pixel representations. Specifically, we propose a new Semantic-Attention to adaptively split the image domain into a series of learnable slices of flexible shapes, where image pixels under similar semantic states will be ascribed to the same slice. By calculating attention to the Semantic-Superpixel tokens encoded from slices, SpecSolver can effectively capture intricate semantic correlations from the vast number of pixels, which also empowers the solver with an endogenous capacity for modeling different magnification scales and allows for efficient computation in linear complexity. On that basis, we elaborate a SpatialNet module, which extracts multiscale local spectral information, and a FreqNet module, which supplements global information, capturing subtle details and variations across different spectra. Experiments on two benchmark SSF datasets verify the state-of-the-art (SOTA) performance of the proposed method, both visually and quantitatively. Also, ablation studies validate the mentioned contributions.
The segmentation of substantial brain lesions is a significant and challenging task in the field of medical image segmentation. Substantial brain lesions in brain imaging exhibit high heterogeneity, with indistinct boundaries between lesion regions and normal brain tissue. Small lesions in single slices are difficult to identify, making the accurate and reproducible segmentation of abnormal regions, as well as their feature description, highly complex. Existing methods have the following limitations: 1) They rely solely on single-modal information for learning, neglecting the multi-modal information commonly used in diagnosis. This hampers the ability to comprehensively acquire brain lesion information from multiple perspectives and prevents the effective integration and utilization of multi-modal data inputs, thereby limiting a holistic understanding of lesions. 2) They are constrained by the amount of data available, leading to low sensitivity to small lesions and difficulty in detecting subtle pathological changes. 3) Current SAM-based models rely on external prompts, which cannot achieve automatic segmentation and, to some extent, affect diagnostic efficiency.To address these issues, we have developed a large-scale fully automated segmentation model specifically designed for brain lesion segmentation, named BrainSegDMIF. This model has the following features: 1) Dynamic Modal Interactive Fusion (DMIF) module that processes and integrates multi-modal data during the encoding process, providing the SAM encoder with more comprehensive modal information. 2) Layer-by-Layer Upsampling Decoder, enabling the model to extract rich low-level and high-level features even with limited data, thereby detecting the presence of small lesions. 3) Automatic segmentation masks, allowing the model to generate lesion masks automatically without requiring manual prompts.We tested and evaluated our model on two common brain disease segmentation benchmarks, including cases of focal cortical dysplasia and gliomas. Our model outperformed existing state-of-the-art methods across four metrics.