3D Gaussian Splatting (3DGS) has demonstrated impressive capabilities in novel view synthesis. However, rendering reflective objects remains a significant challenge, particularly in inverse rendering and relighting. We introduce RTR-GS, a novel inverse rendering framework capable of robustly rendering objects with arbitrary reflectance properties, decomposing BRDF and lighting, and delivering credible relighting results. Given a collection of multi-view images, our method effectively recovers geometric structure through a hybrid rendering model that combines forward rendering for radiance transfer with deferred rendering for reflections. This approach successfully separates high-frequency and low-frequency appearances, mitigating floating artifacts caused by spherical harmonic overfitting when handling high-frequency details. We further refine BRDF and lighting decomposition using an additional physically-based deferred rendering branch. Experimental results show that our method enhances novel view synthesis, normal estimation, decomposition, and relighting while maintaining efficient training inference process.
论文检索
输入标题、作者或关键词，从 4,294 篇学术成果中精准定位
Human motion generation has found widespread applications in AR/VR, film, sports, and medical rehabilitation, offering a cost-effective alternative to traditional motion capture systems. However, evaluating the fidelity of such generated motions is a crucial, multifaceted task. Although previous approaches have attempted at motion fidelity evaluation using human perception or physical constraints, there remains an inherent gap between human-perceived fidelity and physical feasibility. Moreover, the subjective and coarse binary labeling of human perception further undermines the development of a robust data-driven metric. We address these issues by introducing a physical labeling method. This method evaluates motion fidelity by calculating the minimum modifications needed for a motion to align with physical laws. With this approach, we are able to produce fine-grained, continuous physical alignment annotations that serve as objective ground truth. With these annotations, we propose PP-Motion, a novel data-driven metric to evaluate both physical and perceptual fidelity of human motion. To effectively capture underlying physical priors, we employ Pearson's correlation loss for the training of our metric. Additionally, by incorporating a human-based perceptual fidelity loss, our metric can capture fidelity that simultaneously considers both human perception and physical alignment. Experimental results demonstrate that our metric, PP-Motion, not only aligns with physical laws but also aligns better with human perception of motion fidelity than previous work.
Offering diverse perspectives on a museum artifact can deepen visitors' understanding and help avoid the cognitive limitations of a single narrative, ultimately enhancing their overall experience. Physical museums promote diversity through visitor interactions. However, it remains a challenge to present multiple voices appropriately while attracting and sustaining a visitor's attention in the virtual museum. Inspired by recent studies that show the effectiveness of LLM-powered multi-agents in presenting different opinions about an event, we propose SimViews, an interactive multi-agent system that simulates visitor-to-visitor conversational patterns to promote the presentation of diverse perspectives. The system employs LLM-powered multi-agents that simulate virtual visitors with different professional identities, providing diverse interpretations of artifacts. Additionally, we constructed 4 conversational patterns between users and agents to simulate visitor interactions. We conducted a within-subject study with 20 participants, comparing SimViews to a traditional single-agent condition. Our results show that SimViews effectively facilitates the presentation of diverse perspectives through conversations, enhancing participants' understanding of viewpoints and engagement within the virtual museum.
Tombstones are historically and culturally rich artifacts, encapsulating individual lives, community memory, historical narratives and artistic expression. Yet, many tombstones today face significant preservation challenges, including physical erosion, vandalism, environmental degradation, and political shifts. In this paper, we introduce a novel multi-modal framework for tombstone digitization, aiming to improve the interpretation, organization and retrieval of tombstone content. Our approach leverages vision-language models (VLMs) to translate tombstone images into structured Tombstone Meaning Representations (TMRs), capturing both image and text information. To further enrich semantic parsing, we incorporate retrieval-augmented generation (RAG) to integrate externally dependent elements such as toponyms, occupation codes, and ontological concepts. Compared to traditional OCR-based pipelines, our method improves parsing accuracy from an F1 score of 36.1 to 89.5. Furthermore, we evaluate the model's robustness across diverse linguistic and cultural inscriptions, and simulate physical degradation through image fusion to assess performance under noisy or damaged conditions. Our work represents the first attempt to formalize tombstone understanding using large vision-language models, presenting implications for heritage preservation. The code and supplementary materials are available at: https://github.com/LastDance500/Tombstone-Parsing.
Crowd Dynamics Demand Adaptivity: Self-Adaptive Physics-Informed Neural Network for Crowd Simulation
Crowd simulation is crucial for urban planning, traffic management, public safety, and immersive environments. A fundamental challenge is capturing adaptive human behaviors that evolve dynamically with social interactions and task demands. Recently, physics-informed neural networks (PINNs) seamlessly integrate interpretable physics-based models with flexible data-driven learning, significantly enhancing simulation realism. However, current PINN-based methods typically rely on rigid representations of pedestrian perceptions and static task priorities of motion planning, limiting their ability to capture real-world social complexities and behavioral adaptability. To this end, we introduce SA-PINN, a novel Self-Adaptive Physics-Informed Neural Network specifically designed for modeling adaptive crowd behaviors. SA-PINN features two innovative adaptive modules: a self-adaptive social perception module, guided by a visual-field physics model to capture context-dependent social interactions dynamically; and a self-adaptive multi-task PINN training module, automatically balancing key motion objectives such as goal-reaching, collision avoidance, and alignment with real data. By jointly enabling perception-level and task-level adaptations within a unified physics-informed framework, SA-PINN generates highly realistic and physically consistent crowd simulations across diverse environmental contexts. Comprehensive evaluations on three real-world datasets (Lane, Cross 90, and GC) reveal that SA-PINN achieves a 29.7% gain in microscopic trajectory accuracy and enhances macroscopic density similarity by 23.5% compared to the best-performing baselines.
Understanding how the brain represents visual information is a fundamental challenge in neuroscience and artificial intelligence. While AI-driven decoding of neural data has provided insights into the human visual system, integrating multimodal neuroimaging signals-such as EEG, MEG, and fMRI-remains a critical hurdle due to their inherent spatiotemporal misalignment. Current approaches often analyze these modalities in isolation, limiting a holistic view of neural representation. In this study, we introduce BrainFLORA, a unified framework for integrating cross-modal neuroimaging data to construct a shared neural representation. Our approach leverages multimodal large language models (MLLMs) augmented with modality-specific adapters and task decoders, achieving state-of-the-art performance in joint-subject visual retrieval task and has the potential to extend multitasking. Combining neuroimaging analysis methods, we further reveal how visual concept representations align across neural modalities and with real-world object perception. We demonstrate that the brain's structured visual concept representations exhibit an implicit mapping to physical-world stimuli, bridging neuroscience and machine learning from different modalities of neural imaging. Beyond methodological advancements, BrainFLORA offers novel implications for cognitive neuroscience and brain-computer interfaces (BCIs). Our code is available at https://github.com/ncclab-sustech/BrainFLORA.
Underwater images typically suffer from severe colour distortions, low visibility, and reduced structural clarity due to complex optical effects such as scattering and absorption, which greatly degrade their visual quality and limit the performance of downstream visual perception tasks. Existing enhancement methods often struggle to adaptively handle diverse degradation conditions and fail to leverage underwater-specific physical priors effectively. In this paper, we propose a degradation-aware conditional diffusion model to enhance underwater images adaptively and robustly. Given a degraded underwater image as input, we first predict its degradation level using a lightweight dual-stream convolutional network, generating a continuous degradation score as semantic guidance. Based on this score, we introduce a novel conditional diffusion-based restoration network with a Swin UNet backbone, enabling adaptive noise scheduling and hierarchical feature refinement. To incorporate underwater-specific physical priors, we further propose a degradation-guided adaptive feature fusion module and a hybrid loss function that combines perceptual consistency, histogram matching, and feature-level contrast. Comprehensive experiments on benchmark datasets demonstrate that our method effectively restores underwater images with superior colour fidelity, perceptual quality, and structural details. Compared with SOTA approaches, our framework achieves significant improvements in both quantitative metrics and qualitative visual assessments.
3D Visual Question Answering (3D VQA) is crucial for enabling models to perceive the physical world and perform spatial reasoning. In 3D VQA, the free-form nature of answers often leads to improper annotations that can confuse or mislead models when training on the entire dataset. While other text generation tasks can mitigate this issue by learning on large-scale datasets, the scarcity of 3D scene data enlarges the negative effect of misleading annotations. Although active learning strategies can select valuable instances for training, they fail to identify and resolve misleading labels, which the oracle inevitably provides in practice. To address this issue, we propose a multi-turn interactive active learning strategy. This strategy selects data based on models' semantic uncertainty to form a solid knowledge foundation more effectively and actively requests reannotation from an oracle to resolve potentially misleading labels. For uncertainty assessment, we utilize a variance-based metric that takes semantic relationships between terms into consideration, thus avoiding the uniform inter-class similarity assumption of previous assessment metrics. Extensive experiments exhibit better model performance and a substantial reduction in training costs, with a halving of training costs for achieving relatively high accuracy. The code is available at https://github.com/fz-zsl/AQuA.
Images captured in low-light nighttime scenes suffer from light effects. Existing nighttime visibility enhancement methods predominantly focus on low-light image enhancement (LLIE), neglecting light-effect suppression (LES). Current LES methods mainly rely on unsupervised or zero-shot learning due to the lack of paired nighttime light-effect datasets. We construct a large-scale nighttime dataset containing diverse light effects to enable supervised learning for joint LES and LLIE. We design a two-stage structural prior-guided diffusion model for nighttime visibility enhancement, proposing a Laplacian decomposition physical model and a dual-loop Receptance Weighted Key Value (RWKV) to separate light effects from structural features. Experimental results demonstrate that our method outperforms state-of-the-art (SOTA) methods in LLIE and LES tasks. Through supervised training on our dataset, our method achieves optimal performance in joint LES and LLIE while maintaining effectiveness across various real-world scenarios.
Monocular 3D visual grounding is a novel task that aims to locate 3D objects in RGB images using text descriptions with explicit geometry information. Despite the inclusion of geometry details in the text, we observe that the text embeddings are sensitive to the magnitude of numerical values but largely ignore the associated measurement units. For example, simply equidistant mapping the length with unit 'meters' to 'decimeters' or 'centimeters' leads to severe performance degradation, even though the physical length remains equivalent. This observation signifies the weak 3D comprehension of pre-trained language model, which generates misguiding text features to hinder 3D perception. Therefore, we propose to enhance the 3D perception of model on text embeddings and geometry features with two simple and effective methods. Firstly, we introduce a pre-processing method named 3D-text Enhancement (3DTE), which enhances the comprehension of mapping relationships between different units by augmenting the diversity of distance descriptors in text queries. Next, we propose a Text-Guided Geometry Enhancement (TGE) module to further enhance the 3D-text information by projecting the basic text features into geometrically consistent space. These 3D-enhanced text features are then leveraged to precisely guide the attention of geometry features. We evaluate the proposed method through extensive comparisons and ablation studies on the Mono3DRefer dataset. Experimental results demonstrate substantial improvements over previous methods, achieving new state-of-the-art results with a notable accuracy gain of 11.94% in the 'Far' scenario. Our code will be made publicly available.
Novel view synthesis (NVS) in low-light scenes remains a significant challenge due to degraded inputs characterized by severe noise, low dynamic range (LDR) and unreliable initialization. While recent NeRF-based approaches have shown promising results, most suffer from high computational costs, and some rely on carefully captured or pre-processed data-such as RAW sensor inputs or multi-exposure sequences-which severely limits their practicality. In contrast, 3D Gaussian Splatting (3DGS) enables real-time rendering with competitive visual fidelity; however, existing 3DGS-based methods struggle with low-light sRGB inputs, resulting in unstable Gaussian initialization and ineffective noise suppression. To address these challenges, we propose LL-Gaussian, a novel framework for 3D reconstruction and enhancement from low-light sRGB images, enabling pseudo normal-light novel view synthesis. Our method introduces three key innovations: 1) an end-to-end Low-Light Gaussian Initialization Module (LLGIM) that leverages dense priors from learning-based MVS approach to generate high-quality initial point clouds; 2) a dual-branch Gaussian decomposition model that disentangles intrinsic scene properties (reflectance and illumination) from transient interference, enabling stable and interpretable optimization; 3) an unsupervised optimization strategy guided by both physical constrains and diffusion prior to jointly steer decomposition and enhancement. Additionally, we contribute a challenging dataset collected in extreme low-light environments and demonstrate the effectiveness of LL-Gaussian. Compared to state-of-the-art NeRF-based methods, LL-Gaussian achieves up to 2,000× faster inference and reduces training time to just 2%, while delivering superior reconstruction and rendering quality.
Document Large Vision Language Models excel in document-centric tasks and have become a key focus of research. Existing frameworks embed features from a lightweight, document-specific encoder into the first layer of a general-purpose Vision Language Model (VLM). However, this introduces a feature mismatch problem. VLMs typically consist of many stacked layers, with the feature hierarchy becoming increasingly abstract at higher layers. Specifically, the first-layer feature in a VLM is token-level, whereas the feature from the encoder is task-level, resulting in a mismatch. Consequently, it is crucial to identify an optimal layer within the VLM for embedding the encoder's features. Inspired by physics, we reformulate the search for the optimal embedding as a problem of finding the shortest time curve. Leveraging the properties of the shortest time curve, we theoretically derive a task-agnostic proxy score that requires only partial training and propose our searching framework, Brac4VLM. Our theoretical derivation shows that Brac4VLM reduces search time by 97.8% compared to brute-force methods. Experimental results further demonstrate that Brac4VLM identifies embedding points that closely align with the true optima. Moreover, the DocVLM with the optimal embedding position identified achieves state-of-the-art performance across various document-centric tasks. Codes: https://github.com/MaxKinny/Brac4VLM.
In histopathology, tissue sections are typically stained using common H&E staining or special stains (MAS, PAS, PASM, etc. ) to clearly visualize specific tissue structures. The rapid advancement of deep learning offers an effective solution for generating virtually stained images, significantly reducing the time and labor costs associated with traditional histochemical staining. However, a new challenge arises in separating the fundamental visual characteristics of tissue sections from the visual differences induced by staining agents. Additionally, virtual staining often overlooks essential pathological knowledge and the physical properties of staining, resulting in only style-level transfer. To address these issues, we introduce, for the first time in virtual staining tasks, a pathological vision-language large model (VLM) as an auxiliary tool. We integrate contrastive learnable prompts, foundational concept anchors for tissue sections, and staining-specific concept anchors to leverage the extensive knowledge of the pathological VLM. This approach is designed to describe, frame, and enhance the direction of virtual staining. Furthermore, we have developed a data augmentation method based on the constraints of the VLM. This method utilizes the VLM's powerful image interpretation capabilities to further integrate image style and structural information, proving beneficial in high-precision pathological diagnostics. Extensive evaluations on publicly available multi-domain unpaired staining datasets demonstrate that our method can generate highly realistic images and enhance the accuracy of downstream tasks, such as glomerular detection and segmentation. Our code. https://github.com/CZZZZZZZZZZZZZZZZZ/VPGAN-HARBOR is available.
Spatial intelligence is fundamental to AI systems that interact with the physical world, particularly in 3D scene generation and spatial comprehension. Current layout generation in 3D scene synthesis remains highly complex, often constrained by predefined datasets and limited dynamic adaptation to changing spatial relationships. In this paper, we propose GraphCanvas3D, a flexible, query-driven framework for controllable 3D scene generation. Unlike traditional methods that require retraining and predefined input masks for modifications, GraphCanvas3D provides a training-free solution supporting the generation of diverse scenes-both indoor and outdoor-through free manipulation of objects and scene elements. Our framework employs hierarchical, graph-driven scene descriptions, representing spatial elements as graph nodes and establishing coherent relationships among objects in 3D environments. The decoupled object representation enables flexible, on-the-fly scene adjustments and dynamic, customizable scene creation. Experimental results and user studies demonstrate that GraphCanvas3D improves usability, adaptability, and generalization across various 3D scene generation tasks, offering a powerful tool for scalable and diverse scene synthesis.
Despite technological progress in underwater object detection, there are still problems such as the domain shift stemming from diverse environmental conditions and severe image degradation in underwater environments. To address these challenges, we propose a hypernetwork-powered domain generalization framework that synergizes physical priors with multi-frequency feature learning, called Hy-UOD. To combat domain shift challenges, we devise a meta-learning empowered hypernetwork architecture that synthesizes domain-generalization parameters through environment-specific physical descriptor encoding for cross-domains. To further mitigate the impact of complex degradation on object detection performance, we designed a Multi-frequency Feature Dynamic Adaptation (i.e., MFDA) module based on hypernetwork features and domain-specific information. This module implements a systematic compensation for degraded features through a multi-level dynamic adaptation mechanism: ''low-frequency correction, high-frequency refinement, and mid-frequency reconstruction''. Experiments on multiple underwater datasets demonstrate the robust detection performance and strong cross-domain generalization capability of our method. The source code will be available at https://github.com/White-cat-ed/HyUOD.
Creating digitalized hand-object interaction scenes plays a crucial role in recent advancements, enabling viewers to understand how human dexterity influences and shapes the world. In this paper, we present HandCraft, a framework designed to capture and render hand-object interactions with exceptional precision and realism. Our Gaussian models are built on the development of digital representations of hands, objects, and scenes, derived from data captured using multi-modal sensing systems. By combining motion capture with IMU-based data gloves equipped with tactile sensors, HandCraft ensures precise hand pose tracking and reliable contact fidelity. HandCraft includes a novel method that uses hand motions to solve the object occlusions, effectively reconstructing missing interaction details. For enhanced physical feasibility, HandCraft incorporates optimization techniques to resolve object penetration issues and enforce temporal consistency. Using these techniques, we introduce a high-quality dataset of hand-object interaction sequences, featuring complex and prolonged daily activities. This dataset demonstrates HandCraft's ability to capture and reproduce subtle, dynamic interactions in rich detail. HandCraft holds promises in creating realistic virtual environments and advancing world modeling in both graphics and robotics research.
Low-light image enhancement aims to improve brightness, suppress noise, and recover accurate color and structure, requiring precise illumination modeling and reliable reflectance recovery. However, most Retinex-based methods adopt explicit, multi-stage pipelines prone to decomposition bias, error accumulation, and chromatic entanglement between illumination and reflectance. To tackle these issues, we propose IDAR (Implicit Decomposition, illumination Adjustment, and reflectance Restoration), a unified Retinex-inspired framework with two key innovations. First, we design an implicit decomposition strategy based on dual-branch feature learning: a low-frequency-constrained illumination branch models lighting with chromaticity awareness, while a contrast-guided reflection branch preserves details by decoupling reflectance from illumination. This implicit design avoids intermediate supervision and reduces decomposition bias. Second, we introduce the Illumination Chromaticity Expansion Module (ICEM), which employs text-guided chromaticity learning to enhance chromaticity perception. By learning a reflectance-independent spectral representation, ICEM reduces color shifts and improves fidelity under complex lighting. Experiments on multiple benchmarks validate the superior visual quality, quantitative performance, and physical interpretability of IDAR.
Generating interactive motion from texts has garnered significant attention in recent years. While text inputs offer greater flexibility, in many practical applications, there is a need to controllably impose strict constraints on the motion range or trajectory of virtual characters. However, existing trajectory-based methods are designed for single-actor scenarios and lack support for interactivity in interactive motions. Moreover, text-only methods struggle to accurately convey user-intended trajectories. The distribution shift between training and inference often leads to trajectory deviation and physical interpenetration. To address the questions mentioned, we introduce two key concepts: (1) Lead-Follow Paradigm: Inspired by role allocation in partner dancing, we decompose complex interactive motion tasks into a Lead-Follow paradigm. The leader's path is optimized first, and the follower's motion is subsequently adjusted for coherence and alignment. (2) Trajectory Guidance: We highlight the pivotal role of 3D trajectory guidance in interactive motion generation and accurately reflect user intentions. Through 3D trajectory control, we can more controllably generate the desired motion while avoiding physical interpenetration. In addition, we further investigate the refinement of motion scopes for interactive agents and propose an effective optimization strategy to enhance motion coherence and controllability. Experimental results show that the proposed approach, by more effectively using trajectory, outperforms existing methods in both realism and accuracy.
Adversarial patches are widely used to evaluate the robustness of object detection systems in real-world scenarios. These patches were initially designed to deceive single-modal detectors (e.g., visible or infrared) and have recently been extended to target visible-infrared dual-modal detectors. However, existing dual-modal adversarial patch attacks have limited attack effectiveness across diverse physical scenarios. To address this, we propose CDUPatch, a universal cross-modal patch attack against visible-infrared object detectors across scales, views, and scenarios. Specifically, we observe that color variations lead to different levels of thermal absorption, resulting in temperature differences in infrared imaging. Leveraging this property, we propose an RGB-to-infrared adapter that maps RGB patches to infrared patches, enabling unified optimization of cross-modal patches. By learning an optimal color distribution on the adversarial patch, we can manipulate its thermal response and generate an adversarial infrared texture. Additionally, we introduce a multi-scale clipping strategy and construct a new visible-infrared dataset, MSDrone, which contains aerial vehicle images in varying scales and perspectives. These data augmentation strategies enhance the robustness of our patch in real-world conditions. Experiments on four benchmark datasets (e.g., DroneVehicle, LLVIP, VisDrone, MSDrone) show that our method outperforms existing patch attacks in the digital domain. Extensive physical tests further confirm strong transferability across scales, views, and scenarios. Attack demos are provided in the supplementary materials.
Sonar image recognition is a key technology in underwater exploration systems. Compared with natural images, sonar images have fewer texture details and are easily affected by heavy noise, making it more challenging for specialists to distinguish the subtle differences among classes. In view of this, studying fine-grained classification methods for sonar images with scarce annotations is of significant importance. To address this issue, we propose a Physics-Guided Teacher-Student (PGTS) framework to explore the unique physical information of sonar images while simultaneously mitigating the effects of limited annotations. First, PGTS reconstructs sonar signals through physical simulation and a specially designed physics-guided feature generation module, which allows it to bypass the time-consuming physical simulation during inference. Then, we design a multi-modal teacher model combines the reconstructed sonar signals and sonar images to extract discriminative features to generate robust pseudo labels for fine-grained target categories. Finally, the knowledge is transferred to a single-modal student model through consistency loss. Under the joint constraints of the teacher model and the reconstructed sonar physical signals, the student model continuously improves its performance in annotation-scarce scenarios. Notably, when merely 1% of the data is labeled, our method outperforms other state-of-the-art approaches by 12.46% in terms of accuracy.