World models based on video generation demonstrate remarkable potential for simulating interactive environments yet suffer from persistent difficulties in two key areas: maintaining long-term content consistency when scenes are revisited and enabling precise camera control from user-specified inputs. Existing methods based on explicit 3D reconstruction often compromise flexibility in unbounded scenarios and struggle to preserve fine-grained structures. Alternative methods rely directly on previously generated frames without establishing explicit spatial correspondence, thereby limiting controllability and consistency. To address these limitations, we present UCM, a novel framework for unified modeling of long-term memory and precise camera control via a time-aware positional encoding warping mechanism. To reduce computational overhead, we design an efficient dual-stream diffusion transformer for high-fidelity generation. Moreover, we introduce a scalable data curation strategy that utilizes point-cloud-based rendering to simulate scene revisiting, enabling training on over 500K monocular videos. Extensive experiments on real-world and synthetic benchmarks demonstrate that UCM significantly outperforms state-of-the-art methods on long-term scene consistency, while achieving precise camera controllability in high-fidelity video generation. Our code is released at https://humanaigc.github.io/ucm-webpage/.
论文检索
输入标题、作者或关键词，从 100,903 篇学术成果中精准定位
Camera redirection aims to replay a dynamic scene from a single monocular video under a user-specified camera trajectory. However, large-angle redirection is inherently ill-posed: a monocular video captures only a narrow spatio-temporal view of a dynamic 3D scene, providing severely limited observations of the underlying 4D world. The key challenge is therefore to recover a complete and coherent plenoptic representation from this limited input, with consistent geometry and coherent motion. While recent diffusion-based methods achieve impressive visual generation quality, they often break down under large-angle viewpoint changes far from the original trajectory, where missing visual grounding leads to severe geometric ambiguity and temporal inconsistency. To address this, we present FreeOrbit4D, an effective training-free framework that tackles this geometric ambiguity by recovering a foreground-complete 4D proxy as structural grounding for video generation. We obtain this proxy by decoupling foreground and background reconstructions: we unproject the monocular video into a static background and partial foreground point clouds in a unified global space, then leverage an object-centric multi-view diffusion model to synthesize multi-view images and reconstruct complete foreground point clouds in canonical object space. By aligning the canonical foreground point cloud to the global scene space via dense pixel-synchronized 3D–3D correspondences and projecting the foreground-complete 4D proxy onto target camera viewpoints, we provide geometric scaffolds (e.g., depth/visibility cues) that guide a conditional video diffusion model. Extensive experiments show that FreeOrbit4D produces more faithful and temporally coherent redirected videos under challenging large-angle trajectories, and our foreground-complete 4D proxy further opens a potential avenue for practical applications such as edit propagation and 4D data generation. Project page: https://freeorbit4d.vision.ischool.illinois.edu/
Filmmaking demands precise motion control and reference image compositing — capabilities that existing methods treat separately. Point-track-conditioned image-to-video models restrict content insertion to the first frame, while reference-to-video models lack fine-grained spatial-temporal control over how reference content integrates across frames. We present Go-with-the-Track, which unifies both capabilities by jointly conditioning on multiple reference images and reference-anchored point-tracks — extending conventional point-tracks to explicitly establish correspondences between generated frames and reference images, thus enabling precise compositing and motion control throughout the video. To achieve this, we introduce spatially-aware point-track embeddings that encode the full sequence of point-track coordinates using a coordinate-wise MLP followed by temporal pooling. This representation captures the spatial characteristics of each point-track (serving as a unique identifier), while the embedding similarity correlates directly with spatial proximity, enhancing the model’s ability to distinguish and associate point-tracks. We inject these point-track embeddings into a video diffusion transformer via a lightweight adapter, resolving the pixel-to-patch resolution mismatch while avoiding the substantial motion detail loss inherent in naive point-track subsampling. We use a hybrid training strategy to train jointly on dynamic, static, and synthetic scene video datasets to boost motion controllability. Experiments demonstrate that Go-with-the-Track achieves superior motion and reference control in a single model and enables new capabilities: multi-reference conditioned video generation with point-track driven compositing, as well as camera control for both static and dynamic scenes. Project Page: https://eyeline-labs.github.io/Go-with-the-Track/
Transformations produced by image and video generation models often evolve in a highly non-linear manner: long stretches where the content barely changes are followed by sudden, abrupt semantic jumps. To analyze and correct this behavior, we introduce a Semantic Progress Function, a one-dimensional representation that captures how the meaning of a given sequence evolves over time. For each frame, we compute distances between semantic embeddings and fit a smooth curve that reflects the cumulative semantic shift across the sequence. Departures of this curve from a straight line reveal uneven semantic pacing. Building on this insight, we propose a semantic linearization procedure that reparameterizes (or retimes) the sequence so that semantic change unfolds at a constant rate, yielding smoother and more coherent transitions. Beyond linearization, our framework provides a model-agnostic foundation for identifying temporal irregularities, comparing semantic pacing across different generators, and steering both generated and real-world video sequences toward arbitrary target pacing.
For decades, procedural worlds have been built on procedural noise functions such as Perlin noise, which are fast and infinite, yet fundamentally limited in realism and large-scale coherence. Conversely, diffusion models offer unprecedented fidelity but remain generally confined to bounded canvases. We introduce InfiniteDiffusion, a training-free algorithm that reformulates diffusion sampling for lazy and unbounded generation, bridging the fidelity of diffusion models with the properties that made procedural noise indispensable: seamless infinite extent, seed-consistency, and constant-time random access. To demonstrate the utility of this approach, we present Terrain Diffusion, a framework for learned procedural terrain generation with a procedural noise-like interface. Our framework outpaces orbital velocity by 9 times on a consumer GPU, enabling realistic terrain generation at interactive rates. We integrate a hierarchical stack of diffusion models to couple planetary context with local detail, a compact Laplacian encoding to stabilize outputs across Earth-scale dynamic ranges, and an open-source infinite-tensor framework for constant-memory manipulation of unbounded tensors. Together, these components position diffusion models as a practical foundation for the next generation of infinite virtual worlds.
Generative models for material creation are fundamentally limited by the quality and expressivity of available training data. Simple physically based rendering (PBR) materials, which combine a diffuse term with a single-lobe specular component, are commonly used for training but are insufficient to capture many important visual effects present in real materials. We present a method that enhances such simple PBR materials to more expressive ones, by augmenting the single GGX specular lobe into a layered model that captures a broader range of non-diffuse effects. Starting from a simple material, we procedurally construct a corresponding multi-lobe non-diffuse component guided by physical priors, enabling effects such as dust, clearcoat, and layered scattering. To provide a compact representation for downstream applications, we encode this non-diffuse component as a neural material with a shared 6D latent space, where each material instance is represented by two latent textures and decoded by a pretrained universal MLP. We further regularize the latent space to support material generation. The resulting neural material dataset enables training generative models for richer material creation. To demonstrate this application, we finetune a video diffusion model to produce neural latent textures that encode our multi-lobe material, and present generative results as proof of feasibility. Our procedural data enhancement approach is an important step toward improving expressivity in material generation.
Existing texture generation methods rely heavily on latent diffusion models, whose VAE-based spatial compression inherently limits fine-grained detail preservation and degrades pixel-level multi-view consistency. To address this limitation, we introduce PixTex, the first pixel-space multi-view diffusion framework for texture generation, which achieves substantially improved multi-view consistency. Operating directly in image space avoids latent compression, reduces inconsistencies introduced during latent-to-RGB upsampling, and preserves lossless pixel-level geometric guidance for accurate multi-view consistency. However, directly applying pixel-wise attention across multiple views is computationally prohibitive. To balance efficiency and fidelity, we adopt a coarse-to-fine consistency strategy: i) At a coarse patch level, we establish cross-view structural correspondence by employing 5D RoPE to correlate 2D patch coordinates with 3D world-space positions. ii) At the pixel level, a specialized 3D position-aware detailer further refines textural details based on patch features, ensuring fine-grained alignment unattainable by VAE-based methods. Additionally, we propose a novel consistency loss to explicitly guarantee multi-view coherence. Finally, we incorporate a pixel-space multi-view inpainting module to resolve self-occlusions and improve texture completeness. Extensive experiments demonstrate that our framework achieves state-of-the-art multi-view consistency, producing high-fidelity and seamless textures.
The detection of vortex structures in fluid flow is a crucial task in continuum mechanics and flow visualization. However, vortex detection is an extremely challenging task that, despite its importance, is not yet fully solved for 3D unsteady flow fields, due to the complexities introduced by the time-dependence of unsteady flow. We introduce a generic variational framework for the computation of optimal vortex cores in 3D unsteady flow that combines a geometric vortex core model with explicit reference frame optimization. Instead of focusing on a specific vortex detection criterion, we use a generically defined Lagrangian that can incorporate different vortex criteria in a unified way. A key insight of our framework is that the two-manifolds comprising 3D vortex cores in spacetime can be obtained by solving the Euler-Lagrange equations in a single time step with only one independent variable. This is enabled by a Lagrangian that is pre-integrated in time according to the pushforward of the underlying flow. The combination of temporal pre-integration and solving for the optimal two-manifold using only one spatial parameter results in an extremely efficient algorithm.
Reconstructing dynamic fluids from sparse-view videos is a severely under-constrained problem due to complex volumetric visibility and turbulent, topology-changing motion. Current dynamic deformation-based 3D Gaussian Splatting (3DGS) methods use heuristic image-space warps, which often overfit limited observations via non-physical primitive scaling or drifting, producing floaters and geometric distortions. To address these challenges, we propose a hybrid, physics-aware fluid representation that injects physical constraints directly into the 3DGS pipeline. We develop a Beer–Lambert volumetric model mapping Gaussian primitives to a continuous, physically-grounded density field, bridging optical observations with fundamental physical laws. A physically-guided optimization evolves primitives under a global, volume-preserving velocity field, avoiding arbitrary per-particle deformations and effectively preserving volume while recovering fine-scale turbulent details. This unified formulation tightly couples fluid dynamics with differentiable rendering, enabling high-quality, joint predictions of velocity and density. Experiments on synthetic and real smoke datasets demonstrate that our approach surpasses state-of-the-art methods in both visual fidelity and physical consistency, achieving stable 3D reconstructions even from sparse views.
This paper presents a modern, comprehensive system for real-time light field path tracing, addressing the prohibitive computational overhead inherent in driving high-density 3D displays. While offering immersive glasses-free 3D experiences, these displays necessitate the synthesis of massive angular information across numerous viewpoints to satisfy their optical requirements. Conventional rendering pipelines often overlook the hardware’s intrinsic spatial-angular mapping, resulting in redundant computations and extreme ray counts that preclude interactive performance. We resolve this by reformulating light field synthesis as a sparse signal reconstruction task on a high-dimensional manifold. We introduce a unified algebraic framework governed by three mathematical primitives that respectively decouple rendering logic from heterogeneous optical hardware, establish deterministic geometric connectivity across disparate viewpoints, and enable atomic access to unstructured sparse data. Leveraging this formalism, we propose a geometry-driven active sensing pipeline. By modulating a stochastic checkerboard sampling pattern with local disparity cues and coupling this with 5D temporal accumulation and unified spatio-angular integration, our method proactively allocates ray budgets to geometrically complex regions. We demonstrate a complete end-to-end system capable of driving 8K light field displays at interactive frame rates using a single consumer GPU. Our approach not only significantly outperforms existing baselines in computational efficiency and reconstruction fidelity but also naturally supports complex modern light transport effects. Project Page: https://coronaengine.github.io/ssat-page
Large-kernel filtering is a fundamental operation in image post-processing and video effects. While decomposing large kernels into multiple sparse ones is a proven strategy for acceleration on parallel architectures, existing optimization-based approximation methods are constrained to fixed-resolution processing. In this paper, we propose a novel framework based on Polyphase Filtering, which enables the differentiable optimization of sparse kernels across varying resolutions. This approach achieves the fastest approximation to date for large kernels and remains highly effective even for spatially-variant dense kernels. To further enhance efficiency, we introduce Spatial-to-Depth Transformation and Asymmetric Intensity-Decoupled Transformation. Furthermore, to address scenarios with spatially varying kernel sizes, we employ a Layer-Adaptive Filtering strategy that integrates kernels from multiple levels for rapid filtering. Extensive experiments demonstrate that our method significantly outperforms existing state-of-the-art techniques in both computational performance and visual quality.
Modern game scenes often require hundreds of lights to achieve rich visual detail, yet evaluating many lights per pixel remains a major performance bottleneck in real-time rendering, especially on low-end platforms. We present a differentiable lighting optimization pipeline that automatically reduces the number of active lights while preserving the final rendered appearance. Our method jointly optimizes light parameters and dynamically performs light removal, merging, and insertion during optimization. We introduce two loss functions to minimize redundant light overlap and prevent under-illumination, and extend our approach to support Time-of-Day lighting. Experiments on production game scenes show that our method reduces lighting cost by 18–52% while consistently outperforming artist-authored light reductions in visual fidelity.