High dynamic range (HDR) imaging is crucial for capturing real-world lighting conditions. HDR imaging is traditionally achieved either by fusing multiple exposure frames or via inverse tone mapping from a single SDR image. However, the multi-exposure HDR method is prone to motion-induced artefacts and imposes demanding hardware requirements, limiting its practical applicability. Traditional inverse tone-mapping techniques primarily rely on pixel-wise regression methods, which ignore semantic scene contexts and thus frequently introduce halo artefacts and structural distortions. To address this limitation, this paper proposes MLLM-ITM, a novel inverse tone-mapping framework incorporating multimodal large language models (MLLMs). MLLM-ITM utilizes cross-modal features extracted from a frozen MLLM to simultaneously encode visual features and semantic understanding. These features are integrated into a downstream HDR reconstruction backbone through lightweight adapters, enabling content-aware dynamic-range expansion. Furthermore, the decoupled design between MLLM and HDR backbone avoids costly fine-tuning of MLLMs and remains model-agnostic, allowing effortless substitution with emerging multimodal architectures. Extensive experiments on public benchmarks demonstrate that the proposed MLLM-ITM achieves state-of-the-art performance compared with existing inverse tone-mapping methods, highlighting the effectiveness of cross-modal semantic priors in enhancing HDR imaging performance.
论文检索
输入标题、作者或关键词，从 3,042 篇学术成果中精准定位
Estimating object mass from visual input is challenging because mass depends jointly on geometric volume and material-dependent density, neither of which is directly observable from RGB appearance. Consequently, mass prediction from pixels is ill-posed and therefore benefits from physically meaningful representations to constrain the space of plausible solutions. We propose a physically structured framework for single-image mass estimation that addresses this ambiguity by aligning visual cues with the physical factors governing mass. From a single RGB image, we recover object-centric three-dimensional geometry via monocular depth estimation to inform volume and extract coarse material semantics using a vision-language model to guide density-related reasoning. These geometry, semantic, and appearance representations are fused through an instance-adaptive gating mechanism, and two physically guided latent factors (volume- and density-related) are predicted through separate regression heads under mass-only supervision. Experiments on image2mass and ABO-500 show that the proposed method consistently outperforms state-of-the-art methods.
Parameter-efficient adaptation methods, such as adapters and prompt learning, have become popular for transferring pre-trained vision-language models (VLMs) to downstream tasks. However, under limited supervision during adaptation, overly aggressive cross-modal alignment can distort the intrinsic structure of modality-specific representations, leading to degraded generalization. In this paper, we propose Co-adaptive Dual-path Alignment (CoDA), a new adaptation framework that explicitly disentangles and coordinates cross-modal semantic alignment and intra-modal structural consistency. In CoDA, we also propose a parent class labeling strategy that injects hierarchical semantic priors into textual prompts, further stabilizing alignment under limited supervision. Extensive experiments on eleven benchmark datasets show that CoDA outperforms state-of-the-art parameter-efficient methods, particularly under few-shot learning and distribution-shift scenarios. Our code is available at https://github.com/yizhang-ac/CoDA.
Movie dubbing is the task of synthesizing speech from scripts conditioned on video scenes, requiring accurate lip sync, faithful timbre transfer, and proper modeling of character identity and emotion. However, existing methods face two major limitations: (1) high-quality multimodal dubbing datasets are limited in scale, suffer from high word error rates, contain sparse annotations, rely on costly manual labeling, and are restricted to monologue scenes, all of which hinder effective model training; (2) existing dubbing models rely solely on the lip region to learn audio-visual alignment, which limits their applicability to complex live-action cinematic scenes, and exhibit suboptimal performance in lip sync, speech quality, and emotional expressiveness. To address these issues, we propose FunCineForge, which comprises an end-to-end production pipeline for large-scale dubbing datasets and an MLLM-based dubbing model designed for diverse cinematic scenes. The pipeline enables the construction of the first television dubbing dataset, CineDub, which serves as a high-quality foundation for training and evaluation. Building on this, our dubbing model effectively captures multimodal cues and supports complex dubbing scenarios, including monologue, narration, dialogue, and multi-speaker settings. Experiments demonstrate that our approach consistently outperforms state-of-the-art methods in audio quality, word error rate, lip sync, temporal alignment, timbre transfer, and instruction following. Code and demos are available at https://funcineforge.github.io/.
4DVarGen: A 4D Variational-Inspired Generative Model for Eddy-Resolving Surface Ocean Reconstruction
Sea surface variable reconstruction from sparse observations is a key ocean-science challenge. Traditional methods, such as the four-dimensional variational (4DVar) approach, rely on numerical models for background information, leading to high computational costs. Deep learning methods are more efficient but often fail to capture eddy dynamics, resulting in limited effective resolution. We propose 4DVarGen, a 4DVar-inspired generative framework for reconstructing sea surface variable fields at eddy-resolving scales from sparse remote-sensing observations. 4DVarGen establishes a mathematical equivalence between 4DVar and an observation-guided denoising process. Its key innovation is injecting the observation-likelihood gradient into denoising iterations, driving the generated trajectories to evolve in the direction of minimizing the 4DVar objective function toward a maximum a posteriori solution. Spatiotemporal priors learned by a diffusion model serve as background information, reducing computational costs and mitigating the adverse effects of Gaussian assumptions. Experiments show that 4DVarGen effectively leverages the temporal evolution patterns of sea surface temperature (SST) and sea surface height (SSH), as well as their dynamical mappings learned by the diffusion model, leading to improved reconstruction accuracy and effective resolution. Our model, pretrained on GLORYS12V1 reanalysis data, generates sea surface variable fields guided by real observations, achieving accuracy and effective resolution improvements of 18% and 58%, respectively, compared to GLORYS12V1. This study offers a novel framework for reconstructing Earth system states from sparse observations.
Existing degradation-resistant infrared-visible image fusion methods struggle to effectively handle composite degradations, where multiple degradation types exhibit intricate coupling and mutual interference. To address this challenge, we propose SpeciFuse, an infrared-visible image fusion network that learns degradation-specific representations. By explicitly modeling the specificity of individual degradations through a siamese architecture trained on single-degradation data, our method effectively addresses the conflicts arising from multiple degradations, thereby achieving robust fusion performance that generalizes to diverse composite degradation scenarios. In SpeciFuse, a degradation-type specificity decoupling module is designed to disentangle the feature representations by attenuating their redundant correlations in a latent space. It minimizes off-diagonal elements to enforce independence among the encodings of distinct degradation types, while preserving dominant diagonal elements to maintain shared content information. Furthermore, a degradation-aware gated fusion module leverages these decoupled features to dynamically modulate cross-modal fusion across both channel and spatial dimensions. This facilitates a degradation-conditional fusion process that adaptively suppresses degradation artifacts while preserving beneficial information across varying conditions. In extensive comparisons with state-of-the-art methods, SpeciFuse demonstrates more robust performance under diverse composite degradations. The code is available at https://github.com/xbsj-cool/SpeciFuse.
Recently, Visual State Space Models offer powerful global modeling for Dual-modal Salient Object Detection (SOD). However, they are still constrained by three inherent limitations: first, Mamba's strict reliance on sequential ordering makes it sensitive to cross-modal geometric misalignment, where spatial shifts disrupt token correspondence; second, general indiscriminate scanning treating all tokens equally may lead to signal dilution, where sparse foreground features are overwhelmed by background noise; third, conventional decoders rely on implicit upsampling, causing boundary degradation during resolution recovery. To address these challenges, we propose G-SalAlignMamba, a geometry-aware framework tailored for dual-modal SOD. We introduce Geometry-Aware Encoding with explicit alignment to correct spatial shifts, Semantics-Informed Refinement to prevent signal dilution by prioritizing foregrounds, and Structure-Preserving Decoding that integrates explicit alignment with unsupervised boundary refinement. Extensive experiments show that G-SalAlignMamba achieves state-of-the-art performance on RGB-D and RGB-T benchmarks with favorable efficiency (30.41 FPS, 83.80M parameters). The code is available at https://github.com/PC1-99/G-SalAlignMamb.git.
Unified remote sensing multimodal models exhibit a pronounced spatial reversal curse: although they can accurately recognize and describe object locations in images, they often fail to faithfully execute the same spatial relations during text-to-image generation, where such relations constitute core semantic information in remote sensing. Motivated by this observation, we propose Uni-RS, the first unified multimodal model tailored for remote sensing, to explicitly address the spatial asymmetry between understanding and generation. Specifically, we first introduce explicit Spatial-Layout Planning to transform textual instructions into spatial layout plans, decoupling geometric planning from visual synthesis. We then impose Spatial-Aware Query Supervision to bias learnable queries toward spatial relations explicitly specified in the instruction. Finally, we develop Image–Caption Spatial Layout Variation to expose the model to systematic geometry-consistent spatial transformations. Extensive experiments across multiple benchmarks show that our approach substantially improves spatial faithfulness in text-to-image generation, while maintaining strong performance on multimodal understanding tasks like image captioning, visual grounding, and VQA tasks.
Existing AI-generated image (AIGI) detectors perform well in-domain but degrade severely under distribution shift. We observe that this failure is mainly caused by content shortcuts, where detectors spuriously couple forgery artifacts with semantic content, such as object categories or demographic attributes, learning content–label correlations instead of generalizable forgery patterns. To address this issue, we propose PURE (Purging Unrelated Representations for Content-Agnostic Forgery Detection), which achieves content-agnostic detection through two complementary components: a Causal Semantic Generative (CSG) mechanism that disentangles semantic representations from forgery-irrelevant nuisance factors, and a Gaussian Mixture Model (GMM)-based prototype alignment module that suppresses category-specific content bias. Extensive experiments on CIFAKE, GenImage, and AlFace show that PURE achieves superior generalization under spurious correlation reversal. The code is available at https://github.com/wuxinyu519/PURE.
Supervised proxy-based deep cross-modal hashing has become the dominant paradigm for large-scale retrieval. However, prevalent methods model class proxies as deterministic points in the embedding space. This rigid assumption causes severe gradient conflicts in multi-label scenarios, where gradient conflicts arising from label co-occurrence lead to severe gradient contention and optimization collapse.To resolve this, we propose Kent-based Distributional Proxy Hashing (KDPH), a novel framework that shifts proxy representation from static points to flexible anisotropic Kent distributions on the hypersphere. Unlike point proxies that must shift their positions to accommodate conflicting gradients, KDPH absorbs these conflicts by dynamically adjusting its directional variance. This allows the proxy to maintain a stable semantic mean direction while stretching to cover diverse label correlations. Furthermore, to ensure stable training of these geometric parameters, we derive a tailored loss function incorporating the Cayley transform to enforce strict orthogonality. To the best of our knowledge, KDPH is the first framework to successfully introduce the Kent distributions into cross-modal hashing.Experiments on three benchmark datasets demonstrate that KDPH mitigates proxy collapse and chaotic oscillation, significantly outperforms state-of-the-art methods. Code is available at https://github.com/Senmo996/KDPH-official-code.
Source-Fully-Free Domain Adaptation (SFF-DA) has emerged as a strategic paradigm to adapt Vision-Language Models (VLMs) without any access to source data or task-specific source models. However, we identify a critical "Dual Semantic Drift" that hinders this process: static drift caused by the stagnation of frozen hand-crafted templates, and dynamic drift resulting from noisy, instance-level generated captions. These issues lead to a severe stability-plasticity dilemma, manifesting as semantic misalignment and class collapse. To address this, we propose DSSG (Dual-Stream Semantic Guidance), an end-to-end framework that reconciles fine-grained plasticity with global stability. Our core contribution is the Dual Semantic Guidance (DSG) module, which integrates a captioning stream for detailed domain nuances with a template stream to anchor global categorical consistency. Furthermore, a Dynamic Cross-Modal Knowledge Distillation (CMKD) module is introduced to adaptively optimize teacher-student alignment across modalities. Extensive experiments demonstrate that DSSG significantly outperforms current state-of-the-art methods across multiple benchmarks, providing a robust solution for the unsupervised transfer of foundation models. The code is available on https://github.com/mrmenand/DSSG.
Retinal angiography provides critical vascular information, yet optical coherence tomography angiography (OCTA) acquisition remains slower and less accessible than conventional structural OCT. A key open question is: how angiographic signals can be recovered? To address the issues of artifacts and structural blurring in existing generative methods, we propose a wavelet flow matching-based model, FlowOCT. This method operates directly on OCT wavelet coefficients, learning frequency-specific velocity fields to map them to the OCTA wavelet distribution. To ensure anatomical accuracy of the vasculature, we introduce constraints such as depth-wise contrastive alignment and two-dimensional projection consistency. Given the sparse nature of vascular signals in OCTA data, a structure-focused loss function and corresponding evaluation metrics are designed. Experiment results show that FlowOCT outperforms existing methods in terms of both image quality and perceptual similarity. Downstream diagnostic tasks further validate its superior authenticity and generalization capability. Code is available at https://github.com/cnu-medilab/FlowOCT.
Large visual-language models demonstrate exceptional transferability in biomedical image analysis, yet their robustness remains challenging in generalization scenarios such as few-shot learning. This limitation stems from an overly entangled prompt space, causing instability in the conditional distribution, coupled with saliency bias that drives attention to dominate key regions, thereby restricting the coverage of visual representations. Furthermore, cross-modal fusion often struggles to extract critical information signals from redundant alignments. To address these structural bottlenecks, we propose TriCLIP, a decomposition-based framework designed to reconstruct the entire information flow across prompt conditioning, visual evidence extraction, and cross-modal fusion. Prompt Distribution Regularization enhances robustness to linguistic variations by optimizing class-conditional random context perturbations for continuous prompts. Feedback-driven Counterfactual Masking constructs a feedback-driven reverse saliency view that suppresses dominant evidence to promote complementary cue learning and mitigate attention bias. Finally, Selective Regulation Fusion regulates fusion selectivity through paired reinforcement-suppression principles, amplifying information correspondence while suppressing redundant channels. Extensive experiments on diverse biomedical benchmarks indicate that the proposed TriCLIP consistently outperforms existing methods.
Referring Multi-Object Tracking (RMOT) faces a fundamental structural contradiction between the high-discriminability demand and the sparse semantic supervision. This mismatch is particularly acute in highly homogeneous scenarios that require fine-grained discrimination over complex compositional semantics. However, under sparse supervision, models overfit to salient yet insufficient cues, leading to shortcut learning and degraded compositional discrimination. To resolve this, we propose COAL (Counterfactual and Observation-enhanced Alignment Learning), a framework that advances RMOT beyond isolated structural optimization through knowledge regularization. First, we introduce Explicit Semantic Injection (ESI) via a VLM to densify the observation space and enhance instance discriminability. Second, leveraging LLM reasoning, we propose Counterfactual Learning (CFL) to augment supervision, enforcing strict attribute verification for robust compositional recognition. These strategies are unified within a Hierarchical Multi-Stream Integration (HMSI) architecture, which distills external knowledge into domain-specific discriminative representations. Experiments on Refer-KITTI and Refer-KITTI-V2 benchmarks validate COAL's efficacy. Notably, it surpasses the state-of-the-art by 7.28% HOTA on the highly challenging Refer-KITTI-V2. These results demonstrate the effectiveness of knowledge regularization for resolving the sparsity–discriminability paradox in RMOT.
Cross-view geo-localization (CVGL) aims at localizing a ground-level query by retrieving its corresponding match from a database of geo-tagged satellite images. Existing multi-modal CVGL methods lack a structured design in the fusion stage, limiting their ability to fully exploit the information from multiple modalities. To overcome this limitation, we propose a Structural-Analysis-Based fusion principle that guides the design of network architecture. Following this principle, we present Decoupled Query Fusion (DQF), a novel fusion module that decouples feature interactions through role-specific learnable queries. These learnable queries aggregate features into distinct slots. Specifically, modality-specific queries capture unique information from each modality, while cross-modal queries extract redundant and synergistic information between different modalities. To better discriminate positive samples at varying geographic distances, we further propose Geo-aware Circle Loss, which adaptively weights supervision signal based on the geographical distance between samples and anchors. Extensive experiments on large-scale benchmarks demonstrate that our method significantly outperforms state-of-the-art approaches. Comprehensive ablation studies further validate the effectiveness of each component. Our code and models will be made publicly available.
Large Vision-Language Models (LVLMs) often suffer from hallucinations, generating descriptions that include visual details absent from the input image. Recent preference alignment methods typically rely on supervision distilled from stronger models such as GPT. However, this offline paradigm introduces a Supervision-Perception Mismatch: the student model is forced to align with fine-grained details beyond its perceptual capacity, learning to guess rather than to see. To obtain reliable self-supervision for online learning, we identify a Generative-Discriminative Gap within LVLMs, where models exhibit higher accuracy on discriminative verification than open-ended generation. Leveraging this capability, we propose Online Self-CAlibRation (OSCAR), a framework that integrates Monte Carlo Tree Search with a Dual-Granularity Reward Mechanism to construct preference data and iteratively refines the model via Direct Preference Optimization. Extensive experiments demonstrate that OSCAR achieves state-of-the-art performance on hallucination benchmarks while preserving general multimodal capabilities.
Cross-modal hashing has been extensively adopted in cross-modal retrieval tasks owing to its high storage efficiency and fast retrieval capability. However, in practical applications, multimodal data often suffer from modality missing issues, which cause semantic incompleteness and thus severely impair both cross-modal alignment and hashing-based retrieval performance. To address these issues, this paper proposes a novel incomplete cross-modal retrieval framework, called retrieval-guided completion hashing with token–patch alignment (RGCH-TPA). Specifically, it first constructs a dynamic memory bank based on the features of complete samples and uses available modalities to retrieve and locate similar regions of incomplete samples. Thus, it achieves intra-modal reconstruction of missing data under the condition of available modalities using retrieved expert signals. Moreover, a token-patch level cross-modal alignment mechanism is introduced to model the interaction between text tokens and image patches, while global representations are jointly integrated to further enhance the discriminability and robustness of the learned cross-modal representations. Extensive evaluations on three benchmark datasets demonstrate that the proposed RGCH-TPA method consistently outperforms other state-of-the-art incomplete cross-modal hashing methods across missing-modality ratios. The source code is available at https://github.com/szq0816/RGCH-TPA.
Federated Learning (FL) faces two major challenges when deploying large vision-language models such as contrastive language-image pretraining (CLIP) model, namely high computation/communication overhead and performance degradation due to data heterogeneity. To address these, we propose FedENC, a novel federated adaptive CLIP framework. FedENC keeps the CLIP encoder frozen and introduces a lightweight communicable global expert adaptation module for learning general knowledge across clients and two local experts for client-specific adaptation to reduce computation and communication overhead. To mitigate data heterogeneity, we introduce neural collapse (NC) alignment to encourage the model to learn unified and balanced image feature representations. To enrich the performance, we propose a cosine and Kullback–Leibler (KL) divergence based alignment function to improve the prediction consistency between the global and local experts. Finally, we ensemble their outputs for robust inference. Extensive experiments on six datasets (4 natural + 2 medical) in challenging settings (e.g., domain shift) show that FedENC improves the test accuracy (e.g., by +5.37% on DomainNet compared to LoRA) with reasonable communication load (e.g., 15.2x lower than LoRA, with 3.3GB GPU memory usage). The code is available at https://github.com/AIPMLab/FedENC.
Efficient transfer learning methods for large-scale vision–language models (e.g., CLIP) enable strong few-shot transfer, yet existing adaptation methods follow a fixed fine-tuning paradigm that implicitly assumes a uniform importance of the image and text branches, which has not been systematically studied in image classification. Through extensive analysis, we reveal a Branch Bias issue in vision–language image classification: adapting the image encoder does not always improve performance under out-of-distribution settings. Motivated by this observation, we propose A₃B₂, an Adaptive Asymmetric Adapter that alleviates Branch Bias in few-shot learning. A₃B₂ introduces Uncertainty-Aware Adapter Dampening (UAAD), which automatically suppresses image-branch adaptation when prediction uncertainty is high, enabling soft and data-driven control without manual intervention. Architecturally, A₃B₂ adopts a lightweight asymmetric design inspired by mixture-of-experts with Load Balancing Regularization. Extensive experiments demonstrate that A₃B₂ consistently outperforms 11 competitive prompt- and adapter-based baselines.
Vision-Language-Action (VLA) models bridge multimodal reasoning with physical control, but adapting them to new tasks with scarce demonstrations remains unreliable. While fine-tuned VLA policies often produce semantically plausible trajectories, failures often arise from unresolved geometric ambiguities, where near-miss actions lead to divergent execution outcomes under limited supervision. We study few-shot VLA adaptation from a generation-selection perspective and propose a novel framework, VGAS (Value-Guided Action-chunk Selection). It performs inference-time best-of-N selection to identify action chunks that are both semantically faithful and geometrically precise. Specifically, VGAS employs a finetuned VLA as a high-recall proposal generator and introduces Q-Chunk-Former, a geometrically grounded Transformer critic to resolve fine-grained geometric ambiguities. In addition, we propose Explicit Geometric Regularization (EGR), which shapes a discriminative value landscape to preserve action ranking resolution among near-miss candidates while mitigating value instability under scarce supervision. Experiments and theoretical analysis demonstrate that VGAS consistently improves success rates and robustness under limited demonstrations and distribution shifts. Our code is available at https://github.com/Jyugo-15/VGAS