Current reaction generation studies often assume the homogeneity of all reactor body joints in the end-to-end motion generation while neglecting the physical contact information, resulting in evident joint mismatches in both temporal and spatial dimensions. In this paper, we introduce our method, Reactffusion, which addresses the reaction joint mismatch issue by explicitly leveraging the guidance from the actor-reactor physical contacts. At the mathematical modeling level, we reformulate the contact-guided reaction generation as a multi-task problem, divided into two sub-problems: contact information learning and reaction generation with physical constraints. Specifically, given the actor motion sequence, we first introduce a Contact Prediction Module (CPM), which adopts a spatial and temporal attentive mechanism to forecast the contact map, indicating the timing and the location of the potential joint collisions. Then, we employ the contact map as an explicit guide to rectify the sampling distribution in the denoising process of the proposed diffusion network. The comprehensive evaluations prove our method can achieve state-of-the-art performance compared with other reaction generation methods across multiple public benchmarks. Furthermore, the contact map predicted by the CPM can also effectively boost other baselines as an extra plug-in.
论文检索
输入标题、作者或关键词，从 4,294 篇学术成果中精准定位
In this paper, we propose a novel task of text-controlled human-object interaction generation in 3D scenes with movable objects. Existing human-scene interaction datasets suffer from insufficient interaction categories and typically only consider interactions with static objects (do not change object positions), and the collection of such datasets with movable objects is difficult and costly. To address this problem, we construct the InteractMove dataset for Movable Human-Object Interaction in 3D Scenes by aligning existing human-object interaction data with scene contexts, featuring three key characteristics: 1) scenes containing multiple movable objects with text-controlled interaction specifications (including same-category distractors requiring spatial and 3D scene context understanding), 2) diverse object types and sizes with varied interaction patterns (one-hand, two-hand, etc.), and 3) physically plausible object manipulation trajectories. With the introduction of various movable objects, this task becomes more challenging, as the model needs to identify objects to be interacted with accurately, learn to interact with objects of different sizes and categories, and avoid collisions between movable objects and the scene. To tackle such challenges, we propose a novel pipeline solution. We first use 3D visual grounding models to identify the interaction object. Then, we propose a hand-object joint affordance learning to predict contact regions for different hand joints and object parts, enabling accurate grasping and manipulation of diverse objects. Finally, we optimize interactions with local-scene modeling and collision avoidance constraints, ensuring physically plausible motions and avoiding collisions between objects and the scene. Comprehensive experiments demonstrate our method's superiority in generating physically plausible, text-compliant interactions compared to existing approaches. The code is available at https://github.com/Cxhcmhhh/InteractMove.
Diffusion model has been used in indoor scene synthesis and has made significant progress. Current works encode an indoor scene as a top-down view of the room, a list of objects, and their world co-ordinates and orientation. In this paper, we develop a diffusion-based training and synthetic method which incorporates indoor scene ''characteristics''. Firstly, we calculate the relative transformations among objects to capture the local characteristics of the scene. We send this relative transformation into the self-attention layer of the denoising network as ''relative positional encoding''. Secondly, we use room guidance to guide the objects to fit the room's geometry. This improvement uses the room's characteristics to solve the physical collision problem occurring in former diffusion-based works, while preserving plausibilities. Experiments show that our improvements improve the scene variety and quality.
Immersive telepresence aims to authentically reproduce remote physical scenes, enabling the experience of real-world places, objects and people over large geographic distances. This requires the ability to generate realistic novel views of the scene with low latency. Existing methods either depend on depth data from specialized hardware setups or precomputed templates such as human models, which severely restrict their practicality and generalization to diverse scenes. To address these challenges, we introduce RIFTCast, a real-time template-free volumetric reconstruction framework that synthesizes high-fidelity dynamic scenes from a multi-view RGB-only capture setup. The framework is specifically targeted at the efficient reconstruction, transmission and visualization of complex scenes, including extensive human-human and human-object interactions. For this purpose, our method leverages a GPU-accelerated client-server pipeline that computes a visual hull representation to select a suitable subset of images for novel view synthesis, substantially reducing bandwidth and computation demands. This lightweight architecture enables deployment from small-scale configurations to sophisticated multi-camera capture stages, achieving low-latency telepresence even on resource-constrained devices. For evaluation, we provide a comprehensive high-quality multi-view video data benchmark as well as our reconstruction and rendering code, including tools for loading and processing a variety of data input formats, to facilitate future telepresence research.
Photo-realistic novel view synthesis from multi-view images, such as neural radiance field (NeRF) and 3D Gaussian Splatting (3DGS), has gained significant attention for its superior performance. However, most existing methods rely on low dynamic range (LDR) images, limiting their ability to capture detailed scenes in high-contrast environments. While some prior works address high dynamic range (HDR) scene reconstruction, they typically require multi-view sharp images with varying exposure times captured at fixed camera positions-a process that is time-consuming and impractical. To make data acquisition more flexible, we propose Casual3DHDR, a robust one-stage method that reconstructs 3D HDR scenes from casually-captured auto-exposure (AE) videos, even under severe motion blur and unknown, varying exposure times. Our approach integrates a continuous camera trajectory into a unified physical imaging model, jointly optimizing exposure times, camera poses, and the camera response function (CRF). Extensive experiments on synthetic and real-world datasets demonstrate that Casual3DHDR outperforms existing methods in robustness and rendering quality.
Dynamic fluid scene reconstruction remains challenging in multimedia applications and digital content creation due to complex motions and changing topology. While Neural Radiance Fields (NeRF) methods are computationally expensive and 3D Gaussian Splatting (3DGS) approaches struggle with fluid phenomena, we propose Fluid-GS, a flexible, efficient end-to-end framework for sparse-view fluid reconstruction that tightly couples density field modeling with velocity estimation via differentiable advection. Our key innovation is a hybrid Lagrangian-Eulerian Gaussian primitive representation that combines the rendering efficiency of 3DGS with physically-accurate fluid motion tracking on Eulerian grid, that enables us to formulate physics-informed constraints derived from Navier-Stokes equations, enforcing temporal coherence and fluid incompressibility. Moreover, to address the inherent challenges of sparse-view reconstruction, we introduce a fluid-specific Gaussian kernel constraint that preserves the spatial characteristics of fluid phenomena, and dynamically adjusts the anisotropic kernel of Gaussian primitives based on local velocity fields, preventing non-physical artifacts. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art methods in both reconstruction quality and computational efficiency.
Recent advances in 4D Gaussian Splatting have boosted dynamic scene reconstruction and real-time rendering. However, current methods remain retrospective, lacking the ability to forecast future states-limiting their utility in tasks like autonomous navigation and robotics. To address these limitations, we propose FutureGS, a novel Gaussian-based dynamic scene representation framework tailored for continuous 3D future scene prediction and view synthesis. FutureGS introduces a dual-domain decoupled representation, consisting of a static 3D Gaussian base to maintain spatial consistency and a dynamic deformation field to explicitly model temporal motion evolution. To capture long-range dependencies and complex motion dynamics, we design a multi-window collaborative prediction strategy that leverages a sliding temporal window and a bidirectional LSTM-based temporal encoder for robust future motion estimation. Furthermore, we propose a KNN-based local rigidity-aware fusion mechanism, which adaptively regulates the prediction consistency based on local deformation intensity, enhancing the geometric stability and physical plausibility of future scenes. Extensive experiments on standard dynamic scene benchmarks, including D-NeRF and NeRF-DS, demonstrate that FutureGS achieves superior performance in terms of visual fidelity and spatiotemporal consistency, enabling real-time and photorealistic rendering from arbitrary viewpoints at future time steps.
Mental stress assessment is crucial for mental and physical well-being. However, it faces limitations due to domain fragmentation, in which contextual variations in stress triggers and demographics hinder the generalization of assessment models across real-world scenarios. Additionally, mental stress assessment is sensitive and human-centric due to its implications for mental health interventions, emphasizing the need for model transparency and trustworthiness. To address this gap, we propose Retrieval-Augmented Reasoning, a novel framework that bridges domain gaps in mental stress assessment through transparent step-by-step reasoning and dynamic in-context example retrieval. Our framework introduces two key components: (1) a ''detect-then-assess'' reasoning chain decouples stress-relevant facial action units (AUs) from domain-specific noise by first generating textual descriptions as intermediate reasoning step (e.g., ''eyebrow: inner portions raised''). The model then reflects on and learns to refine these descriptions via Direct Preference Optimization (DPO), ensuring faithfulness and helpfulness; (2) a dual-encoder multimodal retriever dynamically selects proper in-context examples from source domain to enhance target-domain assessments, leveraging feedback from the assessment model to optimize retrieval. Experimental results demonstrate that our framework consistently outperforms large multimodal foundation models, stress assessment baselines, and domain generalization methods.
Volumetric video enables immersive experiences by capturing dynamic 3D scenes, enabling diverse applications for virtual reality, education, and telepresence. However, traditional methods struggle with fixed lighting conditions, while neural approaches face trade-offs in efficiency, quality, or adaptability for relightable scenarios. To address these limitations, we present BEAM, a novel pipeline that bridges 4D Gaussian representations with physically-based rendering (PBR) to produce high-quality, relightable volumetric videos from multi-view RGB footage. BEAM recovers detailed geometry and PBR properties via a series of available Gaussian-based techniques. It first combines Gaussian-based human performance tracking with geometry-aware rasterization in a coarse-to-fine optimization framework to recover spatially and temporally consistent geometries. We further enhance Gaussian attributes by incorporating PBR properties step by step. We generate roughness via a multi-view-conditioned diffusion model, and then derive AO and base color using a 2D-to-3D strategy, incorporating a tailored Gaussian-based ray tracer for efficient visibility computation. Once recovered, these dynamic, relightable assets integrate seamlessly into traditional CG pipelines, supporting real-time rendering with deferred shading and offline rendering with ray tracing. By offering realistic, lifelike visualizations under diverse lighting conditions, BEAM opens new possibilities for interactive entertainment, storytelling, and creative visualization.
Human-Object Interaction (HOI) detection serves a broad spectrum of applications. Despite significant progress, current approaches encounter difficulties in effectively handling Non-Contact Human-Object Interaction (NCHOI) scenarios, where humans and objects remain physically apart. To address these challenges, this paper proposes a novel approach, named Latent Interactiveness Field Modeling (LIFM), which enhances HOI detection by capturing long-range contextual dependencies. Specifically, the Latent Interactiveness Field (LIF) is introduced to define potential interactive relationships between humans and objects. To complement this, the LIF Fusion Encoder is designed to adaptively fuse visual features with LIF, resulting in more informative and discriminative feature representations. The Mobile Scanning HOI Dataset (MSHD) is introduced as a comprehensive benchmark to systematically assess the robustness of existing methods on both common HOI and NCHOI in real-world applications. Extensive experimentation indicates that the proposed approach outperforms existing state-of-the-art techniques. It offers substantial improvements, particularly in NCHOI scenarios, which highlight its effectiveness in resolving issues related to long-range interactions.
Image de-reflection is a critical task in computer vision. Existing methods for de-reflection using monocular cameras face challenges due to the lack of depth cues to separate the transmission and reflection layers, particularly under strong illumination or multi-layer reflection scenarios. Although recent advances, such as 3D Gaussian Splatting (3DGS), utilize novel view-synthesis capabilities to separate transmitted and reflected layers, they still encounter difficulties in practice with monocular images. In this paper, we simplify the de-reflection task by combining dual-pixel (DP) technology with 3DGS, forming the first unsupervised de-reflection framework. Specifically, we propose the Dual-View Coordinated Reflection Removal (DCRR) Framework, which integrates depth cues from DP sensors with the rendering capabilities of 3DGS. The DCRR utilizes a dual-view approach that estimates the image transmission layer and opacity via differentiable rasterization with 3DGS and reconstructs the reflection layer through a lightweight multi-layer perceptron. We then present the Dual-Pixel-Driven Reflection Gaussian Pruning (DPRGP) to refine the separation process. By using the physical properties of DP sensors, DCRR achieves significant accuracy improvements in complex reflection scenarios. A real-world DP-based dataset that includes paired reflection/reflection-free images has been collected. Extensive experiments demonstrate our competitive performance compared to state-of-the-art de-reflection approaches.
Video imaging is often affected by complex degradations such as blur, noise, and compression artifacts. Traditional restoration methods follow a ''single-task single-model'' paradigm, resulting in poor generalization and high computational cost, limiting their applicability in real-world scenarios with diverse degradation types. We propose UniFlowRestore, a general video restoration framework that models restoration as a time-continuous evolution under a prompt-guided and physics-informed vector field. A physics-aware backbone PhysicsUNet encodes degradation priors as potential energy, while PromptGenerator produces task-relevant prompts as momentum. These components define a Hamiltonian system whose vector field integrates inertial dynamics, decaying physical gradients, and prompt-based guidance. The system is optimized via a fixed-step ODE solver to achieve efficient and unified restoration across tasks. Experiments show that UniFlowRestore delivers state-of-the-art performance with strong generalization and efficiency. Quantitative results demonstrate that UniFlowRestore achieves state-of-the-art performance, attaining the highest PSNR (33.89 dB) and SSIM (0.97) on the video denoising task, while maintaining top or second-best scores across all evaluated tasks.
Camera-based 3D Semantic Scene Completion (SSC) is a critical task in autonomous driving systems, assessing voxel-level geometry and semantics for holistic scene perception. While existing voxel-based and plane-based SSC methods have achieved considerable progress, they struggle to capture physical regularities for realistic geometric details. On the other hand, neural reconstruction methods like NeRF and 3DGS demonstrate superior physical awareness, but suffer from high computational cost and slow convergence when handling large-scale, complex autonomous driving scenes, leading to inferior semantic accuracy. To address these issues, we propose the Semantic-PHysical Engaged REpresentation (SPHERE) for camera-based SSC, which integrates voxel and Gaussian representations for joint exploitation of semantic and physical information. First, the Semantic-guided Gaussian Initialization (SGI) module leverages dual-branch 3D scene representations to locate focal voxels as anchors to guide efficient Gaussian initialization. Then, the Physical-aware Harmonics Enhancement (PHE) module incorporates semantic spherical harmonics to model physical-aware contextual details and promote semantic-geometry consistency through focal distribution alignment, generating SSC results with realistic details. Extensive experiments and analyses on the popular SemanticKITTI and SSCBench-KITTI-360 benchmarks validate the effectiveness of SPHERE. The code is available at https://github.com/PKU-ICST-MIPL/SPHERE_ACMMM2025.
Physical adversarial attacks on person detectors reveal critical vulnerabilities in safety-critical vision systems such as autonomous driving and surveillance. While recent methods enhance attack efficacy and robustness, they often neglect realistic garment deformations and motion blur, limiting real-world performance. In this work, we propose FAB-Attack (Fabric-a ware and Blur-resistant Attack), a new adversarial attack that simulates realistic garment deformation during training and targets both person detectors and image deblurring models. To enhance attack effectiveness under varying clothing deformations, we introduce a Fabric-aware Texture Appliance (FTA) module, which applies adversarial textures to clothing regions and simulates realistic fabric dynamics via physics-inspired TPS. To better emulate real-world conditions, we develop a differentiable pipeline incorporating motion blur and deblurring processes. Moreover, we demonstrate the stability of low-frequency information during motion blur's generation and removal. Based on this insight, we design a frequency band separation mechanism that suppresses high-frequency components in adversarial patterns to enhance further robustness against motion blur. Experimental results demonstrate that our approach achieves SOTA performance, reducing AP to 25.2% on the COCO dataset and achieving a 94.4% ASR in the real world under severe motion blur.
Fake document detection is an important area in image forensics. Most of the existing techniques focus on the detection of digitally forged documents. In this paper, we look into the forgery of generalized physical occlusion, which is a simple and effective strategy to generate fake document images. We propose an Adversary Decomposition Network (ADDNet) to effectively extract generalized physical occlusion features from various types of documents, where two adversarial classifiers are designed and trained for feature decomposition. On top of the ADDNet, we further propose a lightweight Document Adapter (DA) for flexible and scalable fake document detection, which works well when we encounter a new type of document with limited samples for fine-tuning. To facilitate the research, we newly construct a dataset for physical occlusion detection on different types of documents. Various experiments are carried out to demonstrate the advantage of our proposed scheme over the existing schemes for physical occlusion detection, especially when the document is unseen or has limited samples in training.
Event camera, a novel neuromorphic vision sensor, records data with high temporal resolution and wide dynamic range, offering new possibilities for accurate visual representation in challenging scenarios. However, event data is inherently sparse and noisy, mainly reflecting brightness changes, which complicates effective feature extraction. To address this, we propose a self-supervised pre-training framework to fully reveal latent information in event data, including edge information and texture cues. Our framework consists of three stages: Difference-guided Masked Modeling, inspired by the event physical sampling process, reconstructs temporal intensity difference maps to extract enhanced information from raw event data. Backbone-fixed Feature Transition contrasts event and image features without updating the backbone to preserve representations learned from masked modeling and stabilizing their effect on contrastive learning. Focus-aimed Contrastive Learning updates the entire model to improve semantic discrimination by focusing on high-value regions. Extensive experiments show our framework is robust and consistently outperforms state-of-the-art methods on various downstream tasks, including object recognition, semantic segmentation, and optical flow estimation. The code and dataset are available at https://github.com/BIT-Vision/EventPretrain.
The outdoor vision systems are frequently degraded by snow particles, which obscure scene content and impair the performance of downstream vision tasks. While previous methods rely on physical priors, their performance often deteriorates under real-world conditions. Recently, semantic priors have proven effective in guiding image restoration, especially with the advent of the Segment Anything Model (SAM), which provides robust segmentation masks under adverse weather. However, leveraging SAM in video restoration remains underexplored due to the temporal inconsistency of inter-frame segmentation. In this work, we carefully construct the first framework to incorporate SAM-derived semantic priors into video snow removal, called SAMVSR. Specifically, to address temporal SAM label misalignment, we introduce an Entropy-wise Zone Propagation technique, which selects a reliable reference mask and semantically aligns instances across different frames via an entropy-guided label matching mechanism. Based on the aligned SAM semantic priors, we propose a Zone-Focused Mamba module, a novel Mamba-based architecture that restricts its scanning scope to semantically coherent zones, effectively mitigating irrelevant interactions and enhancing temporal-spatial consistency. Extensive experiments on both synthetic and real-world benchmarks finely validate the superiority of our proposed SAMVSR over existing state-of-the-art video desnowing techniques.
Augmented Reality (AR) enhances Human-Robot Interaction (HRI) by offering diverse interaction methods. However, existing systems often fail to resolve the conflict between a user's implicit preferences and physical ergonomics, leading to suboptimal experiences. We introduce InteractGuide, a novel framework that, for the first time, uses a Large Language Model (LLM) as a central reasoning engine to dynamically balance these competing factors. Our system translates physiological signals into a symbolic ''Preference Memory'' that the LLM reasons over, alongside real-time ergonomic and contextual data, to provide personalized interaction recommendations. A 29-participant study confirms our architecture improves efficiency and experience compared to single-factor approaches, showing the potential of LLMs as reasoning engines for complex AR-HRI. This work presents a validated end-to-end architecture for user-centric interaction adaptation, demonstrating the potential of LLMs as reasoning engines in complex AR-HRI systems.
Teleportation dominates Virtual Reality (VR) locomotion, known for its efficiency and ease of use, particularly in small physical spaces where users have limited room to move. However, is efficiency the only metric that matters? In this study, we challenge this well-established technique by comparing teleportation with an alternative approach for navigating constrained spaces: Walking-with-Portals. We conducted a user study (N = 24) comparing both techniques, collecting data on orientation, cybersickness, presence, and immersion. Our results reveal a distinct trade-off: Teleportation proved significantly faster, more efficient for pathfinding, and induced less cybersickness. Conversely, Walking-with-Portals significantly enhanced users' sense of spatial presence and perceived control over their movement. While a majority preferred Teleportation for the task's efficiency, 91.7% of participants identified Walking-with-Portals as the more immersive technique. Additionally, we establish key design guidelines for intuitive portal placement that allow a large virtual environment to fit within the small play areas (about 2.5 × 2.5 meters) common to home VR users. Our findings suggest that while teleportation remains the default solution in VR locomotion, Walking-with-Portals provides unique benefits that should not be overlooked. This work invites a reflection on locomotion in VR, arguing that the future of VR navigation may go beyond teleport.
ExplorAR: Assisting Older Adults to Learn Smartphone Apps through AR-powered Trial-and-Error with Interactive Guidance
PDF ↗Older adults tend to encounter challenges when learning to use new smartphone apps due to age-related cognitive and physical changes. Compared to traditional support methods such as video tutorials, trial-and-error allows older adults to learn to use smartphone apps by making and correcting mistakes. However, it remains unknown how trial-and-error should be designed to empower older adults to use smartphone apps and how well it would work for older adults. Informed by the guidelines derived from prior work, we designed and implemented ExplorAR, an AR-based trial-and-error system that offers real-time and situated visual guidance in the augmented space around the smartphone to empower older adults to explore and correct mistakes independently. We conducted a user study with 18 older adults to compare ExplorAR with traditional video tutorials and a simplified version of ExplorAR. Results show that the AR-supported trial-and-error method enhanced older adults' learning experience by fostering deeper cognitive engagement and improving confidence in exploring unknown operations.