Residential design is a complex and open-ended problem that requires designers to integrate diverse types of input information while adhering to stringent energy consumption standards. However, most current research in this field focuses on generating floor plans from a limited set of input types, often neglecting to incorporate energy-related physical constraints. Existing approaches are limited by: (1) the lack of multimodal datasets in this domain, (2) the absence of comprehensive residential energy consumption data, and (3) the challenges associated with effectively integrating multiple input types into a unified model. To address these challenges, we propose MRED-14, the first large-scale Multimodal Residential Energy Dataset, comprising 14 input types, including energy consumption values, vector drawings, and textual descriptions, paired with 41,280 high-quality residential floor plans that have been scored and annotated by human experts. Based on this dataset, we introduce the LER-net model, which can flexibly adapt to various input types and generate low-energy residential floor plans. Experimental results demonstrate that LER-net outperforms existing models, achieving state-of-the-art performance under the same input conditions. In addition, the energy consumption of the generated floor plans is reduced by 5.1% compared to the actual residential designs. Further expert evaluations confirm the LER-net model's feasibility for use in residential design.
论文检索
输入标题、作者或关键词，从 3,314 篇学术成果中精准定位
Vision-language models (VLMs) have achieved remarkable success in various vision-language tasks, such as image captioning and visual question answering. However, these models often lack physical common sense, frequently failing to identify visually evident violations of common physical principles. Therefore, evaluating the VLMs' understanding of physical common sense is essential, which has not yet been systematically explored in existing research. To fill this gap, we introduce PhyVIB (Physical Common Sense Violation Image Benchmark). This novel benchmark consists of 16,000 images across eight categories, aiming to systematically assess the VLMs' capability to detect violations of physical common sense in images. Our evaluations show that even the state-of-the-art VLMs perform poorly on PhyVIB, highlighting a significant area for improvement. In response, we propose PhyDetector, a two-stage fine-tuning framework to enhance the VLMs' capability to detect violations of physical common sense. The first stage involves supervised fine-tuning, which equips the VLM with essential concepts related to visual physical anomalies. The second stage utilizes group relative policy optimization to enhance the VLM's multimodal reasoning capability on physical plausibility. Experimental results show that the model fine-tuned with PhyDetector can significantly outperform the state-of-the-art VLMs in physical common sense understanding. Our artifacts are available at https://github.com/ZitongWang018/PhyVIB.
While 3D Gaussian representations (3DGS) have proven effective for modeling the geometry and appearance of objects, their potential for capturing other physical attributes-such as sound-remains largely unexplored. In this paper, we present a novel framework dubbed SonicGauss for synthesizing impact sounds from 3DGS representations by leveraging their inherent geometric and material properties. Specifically, we integrate a diffusion-based sound synthesis model with a PointTransformer-based feature extractor to infer material characteristics and spatial-acoustic correlations directly from Gaussian ellipsoids. Our approach supports spatially varying sound responses conditioned on impact locations and generalizes across a wide range of object categories. Experiments on the ObjectFolder dataset and real-world recordings demonstrate that our method produces realistic, position-aware auditory feedback. The results highlight the framework's robustness and generalization ability, offering a promising step toward bridging 3D visual representations and interactive sound synthesis.
Generating highly realistic 4D interaction in real time is significant for visual content generation. Although existing works have validated to produce impressive dynamics by employing physical simulation and learned material mainly from pre-trained video diffusion models, it is still challenging to generate real-time 4D interaction with high-quality motion due to the heavy time consumption of the simulation solver and indirect material learning strategy. This paper proposes a novel physics-based 4D generation method, Phys4DRT, for arbitrary realistic real-time interaction on 3D Gaussian Splatting (3DGS) objects with direct motion supervision in time-frequency domain. Specifically, we devise a fast and differentiable eXtended Position Based Dynamics (XPBD) simulator as the light-weight controller for efficient physical evolution on a quasi-regular tetrahedral proxy mesh, into which we immerse the static 3DGS for efficient and stable deformation simulation. In addition, to learn the heterogeneous material for realistic motion, we directly supervise the generated dynamic 3D behavior by the motion representation of the optical flow and spectral volume extracted from the generated reference video, rather than indirect supervision in the color space used in previous approaches. We thoroughly conduct experiments on the public benchmarks to demonstrate the efficiency and effectiveness of our method. Our model can accelerate real-time 4D interaction generation by approximately x20 faster than the current Material Point Method (MPM) based approaches while achieving competitive visual quality compared with the state-of-the-art baselines.
Audio-driven video generation aims to synthesize realistic videos that align with input audio recordings, akin to the human ability to visualize scenes from auditory input. However, existing approaches predominantly focus on exploring semantic information, such as the classes of sounding sources present in the audio, limiting their ability to generate videos with accurate content and spatial composition. In contrast, we humans can not only naturally identify the semantic categories of sounding sources but also determine their deeply encoded spatial attributes, including locations and movement directions. This useful information can be elucidated by considering specific spatial indicators derived from the inherent physical properties of sound, such as loudness or frequency. As prior methods largely ignore this factor, we present SpA2V, the first framework explicitly exploits these spatial auditory cues from audios to generate videos with high semantic and spatial correspondence. SpA2V decomposes the generation process into two stages: 1) Audio-guided Video Planning: We meticulously adapt a state-of-the-art MLLM for a novel task of harnessing spatial and semantic cues from input audio to construct Video Scene Layouts (VSLs). This serves as an intermediate representation to bridge the gap between the audio and video modalities. 2) Layout-grounded Video Generation: We develop an efficient and effective approach to seamlessly integrate VSLs as conditional guidance into pre-trained diffusion models, enabling VSL-grounded video generation in a training-free manner. Extensive experiments demonstrate that SpA2V excels in generating realistic videos with semantic and spatial alignment to the input audios.
4D content generation aims to create dynamically evolving 3D content that responds to specific input objects such as images or 3D representations. Current approaches typically incorporate physical priors to animate 3D representations, but these methods suffer from significant limitations: they not only require users lacking physics expertise to manually specify material properties but also struggle to effectively handle the generation of multi-material composite objects. To address these challenges, we propose Phys4DGen, a novel 4D generation framework that integrates multi-material composition perception with physical simulation. The framework achieves automated, physically plausible 4D generation through three innovative modules: first, the 3D Material Grouping module partitions heterogeneous material regions on 3D representations' surfaces via semantic segmentation; second, the Internal Physical Structure Discovery module constructs the mechanical structure of object interiors; finally, we distill physical prior knowledge from multimodal large language models to enable rapid and automatic material properties identification for both objects' surfaces and interiors. Experiments on both synthetic and real-world datasets demonstrate that Phys4DGen can generate high-fidelity 4D content with physical realism in open-world scenarios, significantly outperforming state-of-the-art methods.
Learning natural and diverse behaviors from human motion datasets remains a significant challenge in physics-based character control. Existing conditional adversarial models often suffer from tight and biased embedding distributions where embeddings from the same motion are closely grouped in a small area, and shorter motions occupy even less space. Our empirical observations indicate this limits the representational capacity and diversity under each skill. An ideal latent space should be maximally packed by all motion's embedding clusters. Although methods that employ separate embedding spaces for each motion mitigate this limitation to some extent, introducing a hybrid discrete-continuous embedding space imposes a huge exploration burden on the high-level policy. To address the above limitations, we propose a versatile skill-conditioned controller that learns diverse skills with expressive variations. Our approach leverages the Neural Collapse phenomenon, a natural outcome of the classification-based encoder, to uniformly distribute cluster centers. We additionally propose a novel Embedding Expansion technique to form stylistic embedding clusters for diverse skills that are uniformly distributed on a hypersphere, maximizing the representational area occupied by each skill and minimizing unmapped regions. This maximally packed and uniformly distributed embedding space ensures that embeddings within the same cluster generate behaviors conforming to the characteristics of the corresponding motion clips, yet exhibiting noticeable variations within each cluster. Compared to existing methods, experimental results demonstrate that our controller not only generates high-quality, diverse motions covering the entire dataset but also achieves superior controllability, motion coverage, and diversity under each skill. Both qualitative and quantitative results confirm these traits, enabling our controller to be applied to a wide range of downstream tasks and serving as a cornerstone for diverse applications.
Current reaction generation studies often assume the homogeneity of all reactor body joints in the end-to-end motion generation while neglecting the physical contact information, resulting in evident joint mismatches in both temporal and spatial dimensions. In this paper, we introduce our method, Reactffusion, which addresses the reaction joint mismatch issue by explicitly leveraging the guidance from the actor-reactor physical contacts. At the mathematical modeling level, we reformulate the contact-guided reaction generation as a multi-task problem, divided into two sub-problems: contact information learning and reaction generation with physical constraints. Specifically, given the actor motion sequence, we first introduce a Contact Prediction Module (CPM), which adopts a spatial and temporal attentive mechanism to forecast the contact map, indicating the timing and the location of the potential joint collisions. Then, we employ the contact map as an explicit guide to rectify the sampling distribution in the denoising process of the proposed diffusion network. The comprehensive evaluations prove our method can achieve state-of-the-art performance compared with other reaction generation methods across multiple public benchmarks. Furthermore, the contact map predicted by the CPM can also effectively boost other baselines as an extra plug-in.
In this paper, we propose a novel task of text-controlled human-object interaction generation in 3D scenes with movable objects. Existing human-scene interaction datasets suffer from insufficient interaction categories and typically only consider interactions with static objects (do not change object positions), and the collection of such datasets with movable objects is difficult and costly. To address this problem, we construct the InteractMove dataset for Movable Human-Object Interaction in 3D Scenes by aligning existing human-object interaction data with scene contexts, featuring three key characteristics: 1) scenes containing multiple movable objects with text-controlled interaction specifications (including same-category distractors requiring spatial and 3D scene context understanding), 2) diverse object types and sizes with varied interaction patterns (one-hand, two-hand, etc.), and 3) physically plausible object manipulation trajectories. With the introduction of various movable objects, this task becomes more challenging, as the model needs to identify objects to be interacted with accurately, learn to interact with objects of different sizes and categories, and avoid collisions between movable objects and the scene. To tackle such challenges, we propose a novel pipeline solution. We first use 3D visual grounding models to identify the interaction object. Then, we propose a hand-object joint affordance learning to predict contact regions for different hand joints and object parts, enabling accurate grasping and manipulation of diverse objects. Finally, we optimize interactions with local-scene modeling and collision avoidance constraints, ensuring physically plausible motions and avoiding collisions between objects and the scene. Comprehensive experiments demonstrate our method's superiority in generating physically plausible, text-compliant interactions compared to existing approaches. The code is available at https://github.com/Cxhcmhhh/InteractMove.
Diffusion model has been used in indoor scene synthesis and has made significant progress. Current works encode an indoor scene as a top-down view of the room, a list of objects, and their world co-ordinates and orientation. In this paper, we develop a diffusion-based training and synthetic method which incorporates indoor scene ''characteristics''. Firstly, we calculate the relative transformations among objects to capture the local characteristics of the scene. We send this relative transformation into the self-attention layer of the denoising network as ''relative positional encoding''. Secondly, we use room guidance to guide the objects to fit the room's geometry. This improvement uses the room's characteristics to solve the physical collision problem occurring in former diffusion-based works, while preserving plausibilities. Experiments show that our improvements improve the scene variety and quality.
Immersive telepresence aims to authentically reproduce remote physical scenes, enabling the experience of real-world places, objects and people over large geographic distances. This requires the ability to generate realistic novel views of the scene with low latency. Existing methods either depend on depth data from specialized hardware setups or precomputed templates such as human models, which severely restrict their practicality and generalization to diverse scenes. To address these challenges, we introduce RIFTCast, a real-time template-free volumetric reconstruction framework that synthesizes high-fidelity dynamic scenes from a multi-view RGB-only capture setup. The framework is specifically targeted at the efficient reconstruction, transmission and visualization of complex scenes, including extensive human-human and human-object interactions. For this purpose, our method leverages a GPU-accelerated client-server pipeline that computes a visual hull representation to select a suitable subset of images for novel view synthesis, substantially reducing bandwidth and computation demands. This lightweight architecture enables deployment from small-scale configurations to sophisticated multi-camera capture stages, achieving low-latency telepresence even on resource-constrained devices. For evaluation, we provide a comprehensive high-quality multi-view video data benchmark as well as our reconstruction and rendering code, including tools for loading and processing a variety of data input formats, to facilitate future telepresence research.
Photo-realistic novel view synthesis from multi-view images, such as neural radiance field (NeRF) and 3D Gaussian Splatting (3DGS), has gained significant attention for its superior performance. However, most existing methods rely on low dynamic range (LDR) images, limiting their ability to capture detailed scenes in high-contrast environments. While some prior works address high dynamic range (HDR) scene reconstruction, they typically require multi-view sharp images with varying exposure times captured at fixed camera positions-a process that is time-consuming and impractical. To make data acquisition more flexible, we propose Casual3DHDR, a robust one-stage method that reconstructs 3D HDR scenes from casually-captured auto-exposure (AE) videos, even under severe motion blur and unknown, varying exposure times. Our approach integrates a continuous camera trajectory into a unified physical imaging model, jointly optimizing exposure times, camera poses, and the camera response function (CRF). Extensive experiments on synthetic and real-world datasets demonstrate that Casual3DHDR outperforms existing methods in robustness and rendering quality.
Dynamic fluid scene reconstruction remains challenging in multimedia applications and digital content creation due to complex motions and changing topology. While Neural Radiance Fields (NeRF) methods are computationally expensive and 3D Gaussian Splatting (3DGS) approaches struggle with fluid phenomena, we propose Fluid-GS, a flexible, efficient end-to-end framework for sparse-view fluid reconstruction that tightly couples density field modeling with velocity estimation via differentiable advection. Our key innovation is a hybrid Lagrangian-Eulerian Gaussian primitive representation that combines the rendering efficiency of 3DGS with physically-accurate fluid motion tracking on Eulerian grid, that enables us to formulate physics-informed constraints derived from Navier-Stokes equations, enforcing temporal coherence and fluid incompressibility. Moreover, to address the inherent challenges of sparse-view reconstruction, we introduce a fluid-specific Gaussian kernel constraint that preserves the spatial characteristics of fluid phenomena, and dynamically adjusts the anisotropic kernel of Gaussian primitives based on local velocity fields, preventing non-physical artifacts. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art methods in both reconstruction quality and computational efficiency.
Recent advances in 4D Gaussian Splatting have boosted dynamic scene reconstruction and real-time rendering. However, current methods remain retrospective, lacking the ability to forecast future states-limiting their utility in tasks like autonomous navigation and robotics. To address these limitations, we propose FutureGS, a novel Gaussian-based dynamic scene representation framework tailored for continuous 3D future scene prediction and view synthesis. FutureGS introduces a dual-domain decoupled representation, consisting of a static 3D Gaussian base to maintain spatial consistency and a dynamic deformation field to explicitly model temporal motion evolution. To capture long-range dependencies and complex motion dynamics, we design a multi-window collaborative prediction strategy that leverages a sliding temporal window and a bidirectional LSTM-based temporal encoder for robust future motion estimation. Furthermore, we propose a KNN-based local rigidity-aware fusion mechanism, which adaptively regulates the prediction consistency based on local deformation intensity, enhancing the geometric stability and physical plausibility of future scenes. Extensive experiments on standard dynamic scene benchmarks, including D-NeRF and NeRF-DS, demonstrate that FutureGS achieves superior performance in terms of visual fidelity and spatiotemporal consistency, enabling real-time and photorealistic rendering from arbitrary viewpoints at future time steps.
Mental stress assessment is crucial for mental and physical well-being. However, it faces limitations due to domain fragmentation, in which contextual variations in stress triggers and demographics hinder the generalization of assessment models across real-world scenarios. Additionally, mental stress assessment is sensitive and human-centric due to its implications for mental health interventions, emphasizing the need for model transparency and trustworthiness. To address this gap, we propose Retrieval-Augmented Reasoning, a novel framework that bridges domain gaps in mental stress assessment through transparent step-by-step reasoning and dynamic in-context example retrieval. Our framework introduces two key components: (1) a ''detect-then-assess'' reasoning chain decouples stress-relevant facial action units (AUs) from domain-specific noise by first generating textual descriptions as intermediate reasoning step (e.g., ''eyebrow: inner portions raised''). The model then reflects on and learns to refine these descriptions via Direct Preference Optimization (DPO), ensuring faithfulness and helpfulness; (2) a dual-encoder multimodal retriever dynamically selects proper in-context examples from source domain to enhance target-domain assessments, leveraging feedback from the assessment model to optimize retrieval. Experimental results demonstrate that our framework consistently outperforms large multimodal foundation models, stress assessment baselines, and domain generalization methods.
Volumetric video enables immersive experiences by capturing dynamic 3D scenes, enabling diverse applications for virtual reality, education, and telepresence. However, traditional methods struggle with fixed lighting conditions, while neural approaches face trade-offs in efficiency, quality, or adaptability for relightable scenarios. To address these limitations, we present BEAM, a novel pipeline that bridges 4D Gaussian representations with physically-based rendering (PBR) to produce high-quality, relightable volumetric videos from multi-view RGB footage. BEAM recovers detailed geometry and PBR properties via a series of available Gaussian-based techniques. It first combines Gaussian-based human performance tracking with geometry-aware rasterization in a coarse-to-fine optimization framework to recover spatially and temporally consistent geometries. We further enhance Gaussian attributes by incorporating PBR properties step by step. We generate roughness via a multi-view-conditioned diffusion model, and then derive AO and base color using a 2D-to-3D strategy, incorporating a tailored Gaussian-based ray tracer for efficient visibility computation. Once recovered, these dynamic, relightable assets integrate seamlessly into traditional CG pipelines, supporting real-time rendering with deferred shading and offline rendering with ray tracing. By offering realistic, lifelike visualizations under diverse lighting conditions, BEAM opens new possibilities for interactive entertainment, storytelling, and creative visualization.
Human-Object Interaction (HOI) detection serves a broad spectrum of applications. Despite significant progress, current approaches encounter difficulties in effectively handling Non-Contact Human-Object Interaction (NCHOI) scenarios, where humans and objects remain physically apart. To address these challenges, this paper proposes a novel approach, named Latent Interactiveness Field Modeling (LIFM), which enhances HOI detection by capturing long-range contextual dependencies. Specifically, the Latent Interactiveness Field (LIF) is introduced to define potential interactive relationships between humans and objects. To complement this, the LIF Fusion Encoder is designed to adaptively fuse visual features with LIF, resulting in more informative and discriminative feature representations. The Mobile Scanning HOI Dataset (MSHD) is introduced as a comprehensive benchmark to systematically assess the robustness of existing methods on both common HOI and NCHOI in real-world applications. Extensive experimentation indicates that the proposed approach outperforms existing state-of-the-art techniques. It offers substantial improvements, particularly in NCHOI scenarios, which highlight its effectiveness in resolving issues related to long-range interactions.
Video imaging is often affected by complex degradations such as blur, noise, and compression artifacts. Traditional restoration methods follow a ''single-task single-model'' paradigm, resulting in poor generalization and high computational cost, limiting their applicability in real-world scenarios with diverse degradation types. We propose UniFlowRestore, a general video restoration framework that models restoration as a time-continuous evolution under a prompt-guided and physics-informed vector field. A physics-aware backbone PhysicsUNet encodes degradation priors as potential energy, while PromptGenerator produces task-relevant prompts as momentum. These components define a Hamiltonian system whose vector field integrates inertial dynamics, decaying physical gradients, and prompt-based guidance. The system is optimized via a fixed-step ODE solver to achieve efficient and unified restoration across tasks. Experiments show that UniFlowRestore delivers state-of-the-art performance with strong generalization and efficiency. Quantitative results demonstrate that UniFlowRestore achieves state-of-the-art performance, attaining the highest PSNR (33.89 dB) and SSIM (0.97) on the video denoising task, while maintaining top or second-best scores across all evaluated tasks.
Camera-based 3D Semantic Scene Completion (SSC) is a critical task in autonomous driving systems, assessing voxel-level geometry and semantics for holistic scene perception. While existing voxel-based and plane-based SSC methods have achieved considerable progress, they struggle to capture physical regularities for realistic geometric details. On the other hand, neural reconstruction methods like NeRF and 3DGS demonstrate superior physical awareness, but suffer from high computational cost and slow convergence when handling large-scale, complex autonomous driving scenes, leading to inferior semantic accuracy. To address these issues, we propose the Semantic-PHysical Engaged REpresentation (SPHERE) for camera-based SSC, which integrates voxel and Gaussian representations for joint exploitation of semantic and physical information. First, the Semantic-guided Gaussian Initialization (SGI) module leverages dual-branch 3D scene representations to locate focal voxels as anchors to guide efficient Gaussian initialization. Then, the Physical-aware Harmonics Enhancement (PHE) module incorporates semantic spherical harmonics to model physical-aware contextual details and promote semantic-geometry consistency through focal distribution alignment, generating SSC results with realistic details. Extensive experiments and analyses on the popular SemanticKITTI and SSCBench-KITTI-360 benchmarks validate the effectiveness of SPHERE. The code is available at https://github.com/PKU-ICST-MIPL/SPHERE_ACMMM2025.
Physical adversarial attacks on person detectors reveal critical vulnerabilities in safety-critical vision systems such as autonomous driving and surveillance. While recent methods enhance attack efficacy and robustness, they often neglect realistic garment deformations and motion blur, limiting real-world performance. In this work, we propose FAB-Attack (Fabric-a ware and Blur-resistant Attack), a new adversarial attack that simulates realistic garment deformation during training and targets both person detectors and image deblurring models. To enhance attack effectiveness under varying clothing deformations, we introduce a Fabric-aware Texture Appliance (FTA) module, which applies adversarial textures to clothing regions and simulates realistic fabric dynamics via physics-inspired TPS. To better emulate real-world conditions, we develop a differentiable pipeline incorporating motion blur and deblurring processes. Moreover, we demonstrate the stability of low-frequency information during motion blur's generation and removal. Based on this insight, we design a frequency band separation mechanism that suppresses high-frequency components in adversarial patterns to enhance further robustness against motion blur. Experimental results demonstrate that our approach achieves SOTA performance, reducing AP to 25.2% on the COCO dataset and achieving a 94.4% ASR in the real world under severe motion blur.