Retargeting human kinematic reference motion onto a robot's morphology remains a formidable challenge. Existing methods often produce physical inconsistencies, such as foot sliding, self-collisions, or dynamically infeasible motions, which hinder downstream imitation learning. We propose a bilevel optimization framework that jointly adapts reference motions to a robot's morphology while training a tracking policy using reinforcement learning. To make the optimization tractable, we derive an approximate gradient for the upper-level loss. Our framework requires only a sparse set of semantic rigid-body correspondences and eliminates the need for manual tuning by identifying optimal values for a parameterization expressive enough to preserve characteristic motion across different embodiments. Moreover, by integrating retargeting directly with physics simulation, we produce physically plausible motions that facilitate robust imitation learning. We validate our method in simulation and on hardware, demonstrating challenging motions for morphologies that differ significantly from a human, including retargeting onto a quadruped.
论文检索
输入标题、作者或关键词，从 4,294 篇学术成果中精准定位
We present a novel spatio-temporal reuse framework for time-resolved light transport, enabling efficient Monte Carlo rendering of time-of-flight (ToF) phenomena such as time-gated imaging and transient light capture. Existing ToF rendering methods are computationally expensive, scale poorly to complex dynamic scenes, and are therefore unsuitable for applications with strict latency constraints. To address this limitation, we draw inspiration from ReSTIR, a reuse-based technique for steady-state real-time rendering, and adapt its core principles to interactive-rate ToF simulation. However, naively applying existing ReSTIR methods to ToF rendering leads to severe inefficiency, as reused paths frequently violate optical path-length constraints and thus contribute little or no signal. We overcome this challenge by introducing a path reuse formulation that explicitly enforces physically valid optical path lengths. The key idea is path-length-aware shift mapping, a geometric transformation based on Newton's method that adjusts reused light paths to satisfy temporal gating constraints, inspired by specular manifold exploration in steady-state caustics rendering. The resulting framework substantially improves the efficiency of ToF rendering across a wide range of scenarios, including complex scenes with glossy or specular materials and dynamic motion. Our method supports both time-gated and transient rendering at interactive frame rates, enabling simulation under practical latency constraints. We demonstrate the effectiveness of our approach through two downstream applications, including shape reconstruction and navigation.
Data-driven motion priors that can guide agents toward producing naturalistic behaviors play a pivotal role in creating life-like virtual characters. Adversarial imitation learning has been a highly effective method for learning motion priors from reference motion data. However, adversarial priors, with few exceptions, need to be retrained for each new controller, thereby limiting their reusability and necessitating the retention of the reference motion data when applied to downstream tasks. In this work, we present Score-Matching Motion Priors (SMP), which leverages pre-trained motion diffusion models and score distillation sampling (SDS) to create reusable task-agnostic motion priors. SMPs can be pre-trained on a motion dataset, independent of any control policy or task. Once trained, SMPs can be kept frozen and reused as general-purpose reward functions to train new policies to produce naturalistic behaviors for downstream tasks. We show that a general motion prior trained on large-scale datasets can be repurposed into a variety of style-specific priors. Furthermore, SMP can compose different styles to synthesize new styles not present in the original dataset. Our method can create reusable and modular motion priors that produce high-quality motions comparable to state-of-the-art adversarial imitation learning methods. In our experiments, we demonstrate the effectiveness of SMP across a diverse suite of control tasks with physically simulated humanoid characters. Video available at youtu.be/jBA2tWk6vzU.
Capturing the full plenoptic light transport across spatial, angular, and temporal dimensions has long been a pursuit in computational imaging, yet it remains fundamentally constrained by the high dimensionality of the sampling space and the physical inaccessibility of scene regions due to self-occlusions. While time-resolved imaging records the temporal axis, existing methods are bottlenecked by the combinatorial complexity of the plenoptic function. This high dimensionality makes dense omni-dimensional sampling physically prohibitive. Simultaneously, tight coupling between illumination and viewpoint in current systems also precludes the full acquisition of plenoptic light transport. In this work, we present GenPIE, a Generative Plenoptic Imager designed to bridge the gap between sparse physical observations and high-dimensional light transport. We introduce a decoupled laser-detector hardware setup that enables independent control over illumination and detection, allowing for active probing of indirect light paths. To overcome the ill-posedness of sparse sampling and physical blind spots, we propose a generative inverse transient rendering framework. Our approach leverages 3D foundation models to provide strong semantic and 3D geometric priors for initialization, which are subsequently refined through a differentiable transient path tracer to ensure physically grounded adherence to the Transient Rendering Equation. We demonstrate that GenPIE supports a range of applications that are challenging for steady-state or purely neural methods, including disentangling multi-bounce light transport directly from captured transient videos, time unwarping, and time-resolved relighting. The project page is at https://wangzh1.github.io/GenPIE.
We present Mixwell, a family of sharp 2D fluid brushes and GPU-accelerated analytical methods for progressive, resolution-independent physics-based mixing. Derived from idealized potential flow around a cylindrical tine, Mixwell includes a cusped, Kelvinlet-style regularized velocity brush that parsimoniously captures cylinder-fluid interactions. Building on Maxwell's 1869 drift formulation, we develop GPU-friendly evaluation strategies for particle drift in both infinite and finite strokes. For image-based workflows, we introduce cylindrical Reverse-Drift Functions (RDFs), displacement fields that encode tine insertion, motion, and removal. Like signed distance fields in geometric modeling, RDFs compose naturally and can be chained in shaders to model complex operations while avoiding intermediate texture blur. We also propose a periodicity-exploiting composition scheme for complex RDF patterns. All Mixwell operations are evaluated independently per sample, enabling truly progressive mixing and rendering without global solves, grids, or intermediate texture resampling. We demonstrate real-time GLSL and HLSL implementations and production integrations in Houdini (OpenCL, OSL), enabling progressive, arbitrary-resolution mixing and rendering with negligible numerical dissipation.
Simulating the dynamic, multiscale interactions between granular materials and multiphase fluids remains a significant computational challenge in computer graphics, as the visual complexity of such mixtures arises from strongly coupled small-scale structures. We present a novel, physically-based simulation framework for sand-water-air mixtures that couples a Lattice Boltzmann Method (LBM) for weakly-compressible two-phase fluids with a Material Point Method (MPM) for granular sand. Our approach is built upon a unified continuum formulation that expresses the governing equations for both fluid phases (air and water) and the granular medium within a consistent framework. To accurately capture the transition of sand from a dry, friction-dominated state to a soaked, sticky medium, we introduce a water retention model that describes how liquid infiltrates and is retained within the granular structure. Furthermore, we enforce volume conservation of the fluids within the mixture, ensuring numerical stability and physical realism. Our robust coupling mechanism enables the simulation of complex phenomena such as sand mobilization, transport, settling, and erosion across a wide range of density ratios. We demonstrate the efficiency of our method through several challenging scenarios, including the breaching of sand-walled basins, sediment-laden flows, and the erosive collapse of sand structures.
Synthesizing realistic 3D indoor scenes remains challenging due to data scarcity and the difficulty of simultaneously enforcing global architectural constraints and local semantic consistency. Existing approaches often overlook structural boundaries or rely on fully connected relation graphs that introduce redundant generation errors. Inspired by human design cognition, we present CasLayout, a cascaded diffusion framework that decomposes the joint scene generation task into four conditional sub-stages with explicit physical and semantic roles: (1) predicting furniture quantity and categories, (2) refining object sizes and feature embeddings, (3) modeling spatial relationships in a latent space, and (4) generating Oriented Bounding Boxes (OBBs). This decoupled architecture reduces data requirements and enables flexible integration of Large Language Models (LLMs) and Vision Language Models (VLMs) for zero-shot tasks such as image-to-scene generation. To maintain physical validity within complex floor plans, we explicitly model building elements (e.g., walls, doors, and windows) as conditional constraints. Furthermore, to address the high entropy of dense relation graphs, we introduce a sparse relation graph formulation aligned with human spatial descriptions. By encoding these sparse graphs into a compact latent space using a bidirectional Variational Autoencoder (VAE), the proposed framework provides enhanced relational controllability, allowing generated layouts to better respect functional organization. Experiments demonstrate that CasLayout achieves state-of-the-art performance in fidelity and diversity while enabling improved controllability in practical applications.
This paper presents a highly efficient and robust differentiable flu::id framework, centered on a novel surrogate gradient method that utilizes the flow map structural advantages. Our key insight reveals a significant misalignment between computational intensity and gradient importance during the backward pass. Specifically, we identify a physical duality within the adjoint process, revealing that the cross-step connections inherent in the flow map act as dominant gradient "highways" that propagate sensitivities over long horizons with high fidelity. Leveraging these insights, we develop a surrogate gradient model that retains these critical connections while pruning redundant adjoint computations in a physics-informed manner. Integrated with tailored acceleration techniques, our framework is successfully applied to diverse, challenging optimization tasks characterized by long time horizons and rich vorticity. Results demonstrate significant speedups and memory reductions while maintaining nearly-identical gradients compared to the full-gradient baseline.
Gaussian-based representations have enabled efficient physically-based volume rendering at a fraction of the memory cost of regular, discrete, voxel-based distributions. One of the remaining advantages of classic voxel grids, however, is the ease of constructing hierarchical representations by either storing volumetric mipmaps or selectively pruning branches of an already hierarchical voxel grid. Such strategies reduce rendering time and eliminate aliasing when lower levels of detail are required. Constructing similar strategies for Gaussian-based volumes is not trivial. Straightforward solutions, such as prefiltering or computing mipmap-style representations, lead to increased memory requirements or expensive re-fitting of each level separately. Additionally, such solutions do not guarantee a smooth transition between different hierarchy levels. To address these limitations, we propose Gabor Fields, a mixture of Gabor kernels that enables continuous, orientation-selective frequency filtering at no cost. The frequency content of the asset is reduced by selectively pruning primitives, directly benefiting rendering performance, without refitting or extra storage. Beyond filtering, we demonstrate that stochastically sampling from different frequencies and orientations at each ray recursion enables masking substantial portions of the volume, accelerating ray traversal time in single- and multiple-scattering settings. Furthermore, inspired by procedural volumes, we present an application for efficient design and rendering of procedural clouds as Gabor-noise-modulated Gaussians.
Cloaking objects with metamaterials has been extensively studied to hide internal objects across various physical properties, including optical, acoustic, and thermal. We explore a new direction in mechanical cloaking: halftoning an image using a porous structure that behaves like a uniform, linear, isotropic material and visually matches a target image. For an external observer, this creates the surprising effect where the object appears mechanically isotropic and homogeneous while its porous structure resembles a target image. We introduce a parametric class of porous structures suitable for this problem, as demonstrated by numerical experiments. The structures we define offer a wide range of visual contrasts, enabling effective halftoning while maintaining near isotropic effective mechanical properties.
High-quality facial appearance capture has traditionally required costly studio recording. Recent works consider an in-the-wild smartphone-based setup; however, their model-based inverse rendering paradigm struggles with the complex disentanglement of reflectance from unknown illumination. To bridge this gap, we propose to shift the paradigm into training a powerful delighting network as a prior to constrain the optimization. We leverage the OLAT dataset and the rendered Light Stage scans for training, and propose Dataset Latent Modulation (DLM) to seamlessly integrate these heterogeneous data sources. Specifically, by conditioning the core network on learnable source-aware tokens, we decouple dataset-specific styles from physical delighting principles, enabling the emergence of a delighting prior that outperforms existing proprietary models. This powerful delighting prior enables a simple and automatic appearance capture pipeline that achieves high-quality reflectance estimation from casual video inputs, outperforming prior arts by a large margin. Furthermore, we leverage our appearance capture method to transform the multi-view NeRSemble dataset into NeRSemble-Scan, a large-scale collection of 4K-resolution relightable scans. By open-sourcing our model and the NeRSemble-Scan dataset, we democratize high-end facial capture and provide a new foundation for the research community to build photorealistic digital humans.
Holography offers unique advantages for delivering perceptual realism while preserving compact form factors in VR/AR. Its perceptual quality, however, hinges on encoding rich wavefronts of photorealistic scenes into interference patterns and then incoherently multiplexing the resulting wave fields for perception. Existing CGH paradigms decouple radiance estimation from wave propagation by pre-rendering radiance on discretized scene sectors. This separation between radiometric and wave-optical computation inherently limits the range of focus cues and visual effects that can be faithfully reproduced, including depth- and view-continuity, and physically based material behaviors such as glossy or mirror-like reflection and refraction. We present a physically accurate yet computationally efficient wave optics rendering framework leveraging path tracing to encode full 3D visual cues into phase holograms. Specifically, we employ a Monte Carlo method to solve both the rendering equation and the Rayleigh-Sommerfeld integral simultaneously. Our algorithm is fully compatible with modern graphics techniques and can generate multiple time-multiplexed random holograms with minimal additional time cost via Path Reuse. By employing a fast approximation with an ambient radiance cache, we realize an order of magnitude convergence speed improvement. The resulting coherent wave fields that inherently encode comprehensive visual effects are converted into phase-only holograms under complex-amplitude supervision. Through extensive simulations and experimental validations on a spatial light modulator-based display prototype, we demonstrate faithful holographic reconstructions of natural 3D cues and complex materials, including realistic defocus blur, view-dependent effects, as well as appearance highlights and reflections.
Realistic sound propagation is essential for immersion in a virtual scene, yet physically accurate wave-based simulations remain computationally prohibitive for real-time applications. Wave coding methods address this limitation by precomputing and compressing impulse responses of a given scene into a set of scalar acoustic parameters, which can reach unmanageable sizes in large environments with many source-receiver pairs. We introduce Reciprocal Latent Fields (RLF), a memory-efficient framework for encoding and predicting these acoustic parameters. The RLF framework employs a volumetric grid of trainable latent embeddings decoded with a symmetric function, ensuring acoustic reciprocity. We study a variety of decoders and show that leveraging Riemannian metric learning leads to a better reproduction of acoustic phenomena in complex scenes. Experimental validation demonstrates that RLF maintains replication quality while reducing the memory footprint by several orders of magnitude. Furthermore, a MUSHRA-like subjective listening test indicates that sound rendered via RLF is perceptually indistinguishable from ground-truth simulations.
The explosion of generative 3D assets has created a massive demand for animation, yet current motion capture methods remain brittle, restricted to species-specific templates (e.g., SMPL) or requiring labor-intensive manual rigging. We introduce TopoCap, the first unified framework capable of extracting motion from monocular video and retargeting it onto characters with arbitrary, unseen skeletal topologies, i.e., from bipeds to hexapods and inanimate objects, without test-time optimization. Our key insight is that while skeletal structures are combinatorial and discrete, the underlying physics of motion occupy a continuous, low-dimensional manifold. We materialize this insight via a two-stage generative pipeline. First, we learn a Universal Motion Manifold using a Graph CVAE that compresses heterogeneous kinematic chains into a shared, fixed-length latent code. By explicitly conditioning the decoder on a structural embedding of the target rig, we disentangle motion dynamics from skeletal topology. Second, we treat video-to-animation as a conditional flow matching problem, predicting these topology-agnostic codes from visual features. To learn this generalized prior, we introduce Mobjaverse, a massive-scale dataset curated from Objaverse-XL. Comprising over 5,000 unique skeletal topologies and 2 million frames, it exceeds the structural diversity of existing datasets by two orders of magnitude. Extensive experiments demonstrate that TopoCapoutperforms specialist models on human and quadruped benchmarks while enabling zero-shot retargeting for the long tail of 3D creatures. Dataset is publicly available at https://huggingface.co/datasets/duckduckplz/Mobjaverse
Video-guided 3D animation holds immense potential for content creation, offering intuitive and precise control over dynamic assets. However, practical deployment faces a critical yet frequently overlooked hurdle: the pose misalignment dilemma. In real-world scenarios, the initial pose of a user-provided static mesh rarely aligns with the starting frame of a reference video. Naively forcing a mesh to follow a mismatched trajectory inevitably leads to severe geometric distortion or animation failure. To address this, we present Rectified Dynamic Mesh (R-DMesh), a unified framework designed to generate high-fidelity 4D meshes that are “rectified” to align with video context. Unlike standard motion transfer approaches, our method introduces a novel VAE that explicitly disentangles the input into a conditional base mesh, relative motion trajectories, and a crucial rectification jump offset. This offset is learned to automatically transform the arbitrary pose of the input mesh to match the video’s initial state before animation begins. We process these components via a Triflow Attention mechanism, which leverages vertex-wise geometric features to modulate the three orthogonal flows, ensuring physical consistency and local rigidity during the rectification and animation process. For generation, we employ a Rectified Flow-based Diffusion Transformer conditioned on pre-trained video latents, effectively transferring rich spatio-temporal priors to the 3D domain. To support this task, we construct Video-RDMesh, a large-scale dataset of over 500k dynamic mesh sequences specifically curated to simulate pose misalignment. Extensive experiments demonstrate that R-DMesh not only solves the alignment problem but also enables robust downstream applications, including pose retargeting and holistic 4D generation. Code and pre-trained weights will be available at: https://github.com/Tencent-Hunyuan/R-DMesh.
Reconstructing realistic, physically plausible garments from a single image remains a fundamental challenge. Template-free methods capture surface geometry but lack explicit sewing structure for simulation; while programmatic systems are simulation-ready but constrained by predefined templates. This reveals a fundamental representation gap between geometric reconstruction and structured garment construction. We present PatternGSL, a structured garment representation in the form of a template-free and learnable specification language that encodes complete sewing patterns, including panel boundaries, parameterized seams, and explicit stitch topology, in a compact and standardized form. PatternGSL preserves the physical rigor of pattern-based models while removing template dependence, elevating sewing structure as a first-class target for generative modeling. We further propose a vision-language framework that predicts PatternGSL specifications directly from a single image and decodes them into garments using lightweight deterministic validity handling, without optimization-based refinement or manual cleanup. In addition, we introduce PatternGSLData, the first large-scale image-to-GSL paired dataset comprising 300K samples with complete sewing pattern annotations, enabling supervised VLM training for structured garment reconstruction. Experiments demonstrate improved pattern accuracy over prior baselines, explicit sewing-structure recovery, reliable cloth simulation, and pattern-level editing through the same deterministic decoding pipeline. Code and data-processing scripts will be released at https://github.com/PatternGSL/PatternGSL.
Coordinate-motion assemblies can only be disassembled by the simultaneous motion of multiple parts along distinct paths, providing high structural stability and enabling efficient robotic assembly. Existing examples are largely limited to architectural structures using joint-based connections, or puzzles created through trial-and-error, as the relationship between part geometry and coordinate motion remains poorly understood. Computationally designing such assemblies is challenging because it requires jointly achieving distributed contacts across the entire assembly and a unique coordinate motion for disassembly that rules out other feasible motions. We address this challenge by establishing a theoretical connection between part geometry and unique coordinate motion, enabling us to rigorously verify whether a given assembly admits a unique coordinate motion. Building on this theory, we introduce a two-stage algorithm that optimizes contact interfaces for a target motion and constructs physically feasible part geometries that conform to a user-specified global shape. We demonstrate our approach on models with complex geometries and topologies, including assemblies with large part counts, and validate it through physical fabrication and experiments.
We present Diceplay, a modular physical display for abstract visual composition built from a grid of identical dice. Each die has six faces with distinct geometric primitives, and images emerge through the placement and orientation of the dice. While this medium enables reusable and reconfigurable physical imagery, it poses a challenging design problem: images must be expressed through discrete, extremely low-resolution abstractions, making manual authoring difficult. To address this challenge, we introduce a computational design system that automatically generates Diceplay configurations from text prompts. Our key technical contribution is a grammar-based formulation that relaxes this discrete design space into a smooth optimization landscape, enabling gradient-based optimization using score distillation sampling. We show that our approach consistently produces meaningful abstractions for this medium, whereas state-of-the-art smoothing techniques fail in this extremely challenging regime. We demonstrate our method across a range of prompts and fabricated examples, showing how computationally generated abstractions can be realized as physical visual artifacts.
Recent radiance-field-based reconstruction methods, such as NeRF and 3DGS, achieve high visual fidelity for indoor scenes, but often break down under scene editing due to baked illumination and the lack of explicit light transport. In contrast, inverse path tracing methods based on mesh representations enforce correct light transport but require highly accurate geometry, making them difficult to apply robustly to real indoor scenes. We present Emission-Aware Gaussians and Path Tracing (EAG-PT), a method for physically based reconstruction and rendering of indoor scenes using a unified 2D Gaussian representation, targeting editable diffuse global illumination. Our approach consists of three key ideas: (1) representing indoor scenes with 2D Gaussians as a transport-friendly geometric proxy that avoids explicit mesh reconstruction; (2) explicitly separating emissive and non-emissive components during reconstruction to support editing; and (3) decoupling reconstruction from final rendering by using efficient single-bounce optimization and high-quality multi-bounce path tracing, respectively. Experiments on synthetic and real indoor scenes show that EAG-PT produces more natural and physically consistent edited renderings than radiance-field reconstructions, while preserving finer geometric detail and avoiding mesh-induced artifacts compared with mesh-based inverse path tracing. These results highlight the potential of our approach for applications such as interior design, XR content creation, and embodied AI.