Recent progress has shown that video diffusion models (VDMs) can be repurposed to solve various multimodal graphics tasks. However, existing approaches predominantly train separate models for each specific problem setting. This practice locks models into fixed input-output mappings, and typically ignores the joint correlations across modalities. In this paper, we present UniVidX, a unified multimodal framework designed to leverage VDM priors to enable versatile video generation. Our goal is to (i) master diverse pixel-aligned tasks by formulating them as conditional generation problems within multimodal space, (ii) adapt to modality-specific distributions without compromising the backbone's native priors, and (iii) ensure cross-modal consistency during synthesis. Concretely, we propose three key designs: 1) Stochastic Condition Masking (SCM): by randomly partitioning modalities into clean conditions and noisy targets during training, we enable the model to learn omni-directional conditional generation rather than fixed mappings. 2) Decoupled Gated LoRA (DGL): we attach per-modality LoRAs and activate them when a modality serves as a generation target, thereby preserving the VDM's strong priors. 3) Cross-Modal Self-Attention (CMSA): we explicitly share keys/values across modalities while maintaining modality-specific queries, facilitating information exchange and inter-modal alignment. We validate our framework by instantiating it in two domains: 1) UniVid-Intrinsic for RGB videos and their intrinsic maps (albedo, irradiance, normal), and 2) UniVid-Alpha for blended RGB videos and their constituent RGBA layers. Experimental results demonstrate that both models achieve performance competitive with state-of-the-art methods across distinct tasks. Notably, they exhibit robust generalization capabilities in in-the-wild scenarios, even when trained on limited datasets of fewer than 1k videos.
论文检索
输入标题、作者或关键词，从 17,376 篇学术成果中精准定位
k-nearest neighbor (k-NN) search is a fundamental primitive in geometry processing and computer graphics. While spatial partitioning structures such as kd-trees are standard, they are often manifold-blind, failing to exploit the intrinsic low-dimensional structure of points sampled from 2-manifolds. Recent advances in dynamic programming-based nearest neighbor search (DP-NNS) leverage incrementally constructed Voronoi diagrams to accelerate queries, where each site p maintains a list of successors that progressively refine its Voronoi cell. However, DP-NNS is restricted to single nearest neighbor (k = 1) searches, precluding their adoption in applications that require local neighborhood statistics. In this paper, we generalize the DP-NNS framework to support arbitrary k-NN queries for manifold-aligned data. Our approach is founded on the geometric observation that if pi is the nearest neighbor of a query q in P, then the second nearest neighbor of q must reside either within the prefix set P1:i-1 = [p1, ..., pi-1} or within pi's successor list. By recursively extending this principle, we introduce Manifold k-NN, a recursive algorithmic scheme that significantly outperforms conventional kd-trees for manifold-aligned data. Our method achieves a 1×-10× speedup in volume-to-surface query scenarios and inherently supports dynamic prefix queries—enabling k-NN searches within any subset P1:m (m ≤ n) with zero overhead. Furthermore, we extend the framework to support point deletion via local Delaunay updates, providing a complete suite of dynamic operations for point set modification. Comprehensive experiments on diverse geometric datasets demonstrate the efficiency and broad applicability of our approach for modern graphics pipelines. Source code is available at https://github.com/sssomeone/manifold-knn.
We present HumanFlow, a unified flow-matching-based framework that enables high-fidelity and controllable full-body human image generation under diverse human-centric control conditions. Despite recent progress, controllable human image generation poses a fundamental challenge in balancing high visual fidelity with strict adherence to human-centric control conditions. HumanFlow formulates human image generation as a conditional flow-matching process with deterministic generation dynamics. To incorporate such human-centric control conditions into the pretrained model, we introduce a unified control framework with Control Encoder and Token-ControlNet. A Control Encoder maps diverse conditions into a unified latent representation that is spatially aligned with the image latent space. Token-ControlNet is a lightweight control network architecturally aligned with the FLUX double-stream design. To address accurate structural control over human bodies, we further propose the Human Topology Consistency Loss (HTCL). HTCL regularizes conditional flow matching by constraining generated human configurations to a union of statistically grounded topology manifolds defined by normalized bone ratios and joint angles. To support large-scale training and systematic evaluation, we construct MiCoGen, a multi-condition human image dataset comprising over one million full-body human images with aligned text descriptions and rich human-centric control conditions. Extensive quantitative and qualitative evaluations on the MiCoGen dataset show that HumanFlow consistently achieves improved structural consistency than the existing diffusion-based and flow-matching-based methods, while maintaining high visual fidelity.
Video-guided 3D animation holds immense potential for content creation, offering intuitive and precise control over dynamic assets. However, practical deployment faces a critical yet frequently overlooked hurdle: the pose misalignment dilemma. In real-world scenarios, the initial pose of a user-provided static mesh rarely aligns with the starting frame of a reference video. Naively forcing a mesh to follow a mismatched trajectory inevitably leads to severe geometric distortion or animation failure. To address this, we present Rectified Dynamic Mesh (R-DMesh), a unified framework designed to generate high-fidelity 4D meshes that are “rectified” to align with video context. Unlike standard motion transfer approaches, our method introduces a novel VAE that explicitly disentangles the input into a conditional base mesh, relative motion trajectories, and a crucial rectification jump offset. This offset is learned to automatically transform the arbitrary pose of the input mesh to match the video’s initial state before animation begins. We process these components via a Triflow Attention mechanism, which leverages vertex-wise geometric features to modulate the three orthogonal flows, ensuring physical consistency and local rigidity during the rectification and animation process. For generation, we employ a Rectified Flow-based Diffusion Transformer conditioned on pre-trained video latents, effectively transferring rich spatio-temporal priors to the 3D domain. To support this task, we construct Video-RDMesh, a large-scale dataset of over 500k dynamic mesh sequences specifically curated to simulate pose misalignment. Extensive experiments demonstrate that R-DMesh not only solves the alignment problem but also enables robust downstream applications, including pose retargeting and holistic 4D generation. Code and pre-trained weights will be available at: https://github.com/Tencent-Hunyuan/R-DMesh.
Image customization is a fundamental image editing task that synthesizes an instruction-following image of a specific subject conditioned on reference images. However, most existing methods rely on a single reference image, which makes it difficult to preserve the subject’s underlying 3D structure and maintain multi-view consistency. We therefore study 3D-aware image customization: given an arbitrary multi-view reference set, the model generates an instruction-following image from a user-specified target camera viewpoint, while preserving both subject identity and 3D geometry. This formulation enables a practical “capture-then-customize” workflow and reduces the viewpoint drift commonly observed in 2D-conditioned editors. We propose ViewWeaver, a geometry-grounded generative rendering framework that anchors synthesis with explicit target-view rendering cues aligned with camera geometry, and aggregates multi-view evidence through a lightweight Mixture-of-Views module inside a rectified-flow DiT generator. To facilitate scalable training and systematic evaluation, we build a large-scale data engine and introduce GSO12, a comprehensive benchmark assessing visual fidelity, instruction adherence, and 3D consistency across different reference-view budgets and diverse scene contexts. Extensive experiments show that ViewWeaver significantly improves 3D consistency and viewpoint controllability while maintaining strong identity preservation, enabling accurate and flexible view-controllable customization.
Recovering high-quality video from sparse event streams is a challenging task. Regression methods often blur textures, while existing generative models struggle with long-term stability. We propose LongE2V, a novel approach that leverages pre-trained video diffusion priors to jointly handle event-based video reconstruction, prediction, and frame interpolation. By fine-tuning a foundational video model, our approach achieves high data efficiency and superior perceptual quality. We introduce Autoregressive Unrolling and Adaptive Context Switching to mitigate temporal drift in extremely long sequences. We also propose Reencoding Alignment with Cross Residual Correction to ensure precise bidirectional consistency during frame interpolation. Furthermore, Event Voxel Density Augmentation ensures robustness across varying sensor resolutions. Extensive experiments on real-world benchmarks demonstrate that LongE2V outperforms state-of-the-art methods across all three tasks, exhibiting exceptional temporal coherence and zero-shot generalization.
Vectorization of line art and sketches is an important problem in computer graphics and remains an active area of research. In this paper, we leverage several recent advances in the field to introduce a new state-of-the-art method for line art vectorization. First, we show that it is possible to go beyond heuristics to extract a meaningful set of strokes from a raster image. We leverage both depth prediction and semantic feature extraction models to propose a more principled approach to split the skeleton graph of the sketch into subgraphs to initialize a set of strokes, producing results that better align with artistic intent. Second, we model strokes as Bézier curves with both geometric and appearance components, and employ 2D Gaussian splatting for fast, differentiable rendering. This formulation enables efficient fitting of strokes to the input while jointly optimizing control points and brush texture. We further demonstrate promising results on video by incorporating temporal tracking and adaptive keyframe introduction. Our evaluation shows that we achieve state-of-the-art results for image reconstruction, while being fast and producing strokes of high quality. More importantly, the fitting is highly compatible with user input, through the correction of the splines and the selection of optimization parameters.
We present SCOPE (Scale-Consistent One-Pass Estimation of 3D Geometry), a novel approach for estimating 3D geometry from extended monocular video sequences, where existing methods struggle to maintain both geometric accuracy and temporal consistency across hundreds of frames. Our approach generates affine-invariant 3D point maps with shared parameters across entire sequences, enabling consistent scale-invariant representations. We introduce three key innovations: viewpoint-invariant geometry aligning multi-perspective points in a unified reference frame; appearance-invariant learning enforcing consistency across exponential timescales; and frequency-modulated positioning enabling extrapolation to sequences vastly exceeding training length. Experiments across diverse datasets demonstrate significant improvements, reducing relative point map error by 24.2% and temporal alignment error by 34.9% on ScanNet compared to state-of-the-art methods. Our approach handles challenging scenarios with complex camera trajectories and lighting variations while efficiently processing extended sequences in a single pass. Code and model weights: https://github.com/zhengzhang01/SCOPE.
Motion in-betweening is one of the most artistically demanding and time-consuming stages of 3D animation, where the expressivity and rhythm of motion are defined. The level of creative control it requires makes it a major production bottleneck, underscoring the need for intelligent tools that assist animators in this process. Although recent deep learning approaches have achieved strong results in motion synthesis and in-betweening, they assume data characteristics, motion styles, and problem formulations that diverge from professional animation workflows. To bridge this gap, we propose a method explicitly aligned with the constraints of motion in-betweening for keyframe-based animation in production environments. At its core, the Adaptive Interpolation–Synthesis (AIS) layer mirrors the animator’s creative process by dynamically balancing learned interpolation and direct pose synthesis. In addition, a domain-based input keypose schedule reflects the distribution of production data, improving stylistic consistency and alignment between training and real-world usage. Our method achieves state-of-the-art performance on production data; when integrated into Autodesk Maya, it enables animators to complete in-betweening tasks with a 3.5 × speedup.
We propose a robust texture optimization framework that handles inconsistent AI-generated targets by operating directly within non-differentiable production pipelines. Standard optimization approaches struggle with inconsistent targets, often producing blurry textures and ghosting artifacts. Moreover, existing inverse rendering methods rely on differentiable renderers, causing a rendering gap when assets are deployed in production engines. To address the inconsistencies, we introduce a robust formulation that jointly optimizes a deformation grid and an uncertainty map. This formulation effectively decouples geometric misalignment from semantic hallucinations. To avoid the rendering gap, we leverage finite-difference gradient estimation to operate entirely within standard rasterizers. Extensive experiments demonstrate that our approach recovers sharp, high-fidelity textures from inconsistent AI-generated targets, achieving higher visual quality than existing methods. This makes our technique a practical tool for film production, gaming, and virtual reality, where flexibility and visual quality are paramount.
Camera redirection aims to replay a dynamic scene from a single monocular video under a user-specified camera trajectory. However, large-angle redirection is inherently ill-posed: a monocular video captures only a narrow spatio-temporal view of a dynamic 3D scene, providing severely limited observations of the underlying 4D world. The key challenge is therefore to recover a complete and coherent plenoptic representation from this limited input, with consistent geometry and coherent motion. While recent diffusion-based methods achieve impressive visual generation quality, they often break down under large-angle viewpoint changes far from the original trajectory, where missing visual grounding leads to severe geometric ambiguity and temporal inconsistency. To address this, we present FreeOrbit4D, an effective training-free framework that tackles this geometric ambiguity by recovering a foreground-complete 4D proxy as structural grounding for video generation. We obtain this proxy by decoupling foreground and background reconstructions: we unproject the monocular video into a static background and partial foreground point clouds in a unified global space, then leverage an object-centric multi-view diffusion model to synthesize multi-view images and reconstruct complete foreground point clouds in canonical object space. By aligning the canonical foreground point cloud to the global scene space via dense pixel-synchronized 3D–3D correspondences and projecting the foreground-complete 4D proxy onto target camera viewpoints, we provide geometric scaffolds (e.g., depth/visibility cues) that guide a conditional video diffusion model. Extensive experiments show that FreeOrbit4D produces more faithful and temporally coherent redirected videos under challenging large-angle trajectories, and our foreground-complete 4D proxy further opens a potential avenue for practical applications such as edit propagation and 4D data generation. Project page: https://freeorbit4d.vision.ischool.illinois.edu/
Existing texture generation methods rely heavily on latent diffusion models, whose VAE-based spatial compression inherently limits fine-grained detail preservation and degrades pixel-level multi-view consistency. To address this limitation, we introduce PixTex, the first pixel-space multi-view diffusion framework for texture generation, which achieves substantially improved multi-view consistency. Operating directly in image space avoids latent compression, reduces inconsistencies introduced during latent-to-RGB upsampling, and preserves lossless pixel-level geometric guidance for accurate multi-view consistency. However, directly applying pixel-wise attention across multiple views is computationally prohibitive. To balance efficiency and fidelity, we adopt a coarse-to-fine consistency strategy: i) At a coarse patch level, we establish cross-view structural correspondence by employing 5D RoPE to correlate 2D patch coordinates with 3D world-space positions. ii) At the pixel level, a specialized 3D position-aware detailer further refines textural details based on patch features, ensuring fine-grained alignment unattainable by VAE-based methods. Additionally, we propose a novel consistency loss to explicitly guarantee multi-view coherence. Finally, we incorporate a pixel-space multi-view inpainting module to resolve self-occlusions and improve texture completeness. Extensive experiments demonstrate that our framework achieves state-of-the-art multi-view consistency, producing high-fidelity and seamless textures.
Reconstructing high-fidelity, animatable, and relightable 3D hand avatars from a single RGB image is a challenging yet critical task for immersive VR/AR applications. State-of-the-art hand reconstruction methods achieve impressive reconstruction and relighting results, but they mostly leverage dense observations, such as multi-view images or monocular video sequences and rely on per-scene optimization. Moreover, it is difficult for these methods to generate plausible appearance in occluded regions. In contrast, existing single-view hand reconstruction methods typically struggle to disentangle global illumination, resulting in textures with baked-in shading artifacts. To address these challenges, we propose FLASHand, the first feed-forward relightable and animatable 3D hand reconstruction model from a single RGB image. Our method bridges the gap between efficiency and fidelity, enabling instant creation of personalized hand avatars with disentangled appearance that can be rendered under novel lighting and poses. To ensure plausible geometry generation and achieve high-fidelity animation and relighting, we introduce a mesh-based disentangled 2D Gaussian splatting hand representation. We leverage the NIMBLE parametric prior to define a canonical attribute space, and define geometry and appearance attributes explicitly. We then propose the Feed-forward Hand Attributes Generator (FHAG) to predict these attributes from a single image. By leveraging a cross attention module to fuse canonical geometric information with input image features, FHAG effectively lifts pixel-level visual cues into the 3D hand canonical space, directly generating spatially-aligned geometry and appearance attributes. This allows us to bypass iterative optimization and instantly reconstruct a high-fidelity hand avatar from a single RGB image. Extensive experiments on both synthetic and in-the-wild datasets demonstrate that FLASHand achieves state-of-the-art performance in novel view synthesis and supports real-time animation and relighting. Code and data are available at https://github.com/IGLICT/FLASHand.
Boundary Representation (BRep) is the standard format for Computer-Aided Design (CAD), yet reconstructing high-quality BReps from single-view images remains challenging due to the complexity of topological constraints and operation sequences. We present Img2CADSeq, a multi-stage pipeline that overcomes these limitations by encoding CAD sequences into a three-level hierarchical codebook. Guided by an importance prioritization, this strategy values profiles over details, compressing long sequences into a stable discrete latent space. To bridge the modality gap, we leverage a coarse-to-fine point cloud intermediate, aligning 2D visual features with 3D CAD sequences via contrastive learning to condition a VQ-Diffusion model. Supported by newly introduced CAD-220K and PrintCAD datasets, our approach ensures robust industrial domain adaptation. Extensive experiments demonstrate that Img2CADSeq significantly outperforms state-of-the-art methods, producing standard STEP files that can be directly used in commercial CAD software. Code and data for this paper are at https://github.com/Rilpraa0110/Img2CADSeq
Generative CAD modeling has broad design and application potential. Despite significant advances in Boundary Representation (B-Rep) generation, the dominant representation in CAD, existing methods largely depend on uniformly sampled point- or grid-based geometry representations, sacrificing native surface types and parameters and thereby limiting geometric fidelity and downstream usability. We present ParaCAD, an autoregressive framework for point-cloud-conditioned B-Rep generation that directly operates on native parametric surfaces. ParaCAD introduces a surface-centric tokenization that explicitly encodes each face by its exact surface type and continuous parameters, preserving the intrinsic semantics of CAD geometry. Our model first generates parametric surfaces with constrained UV domains, and then constructs a valid B-Rep by globally intersecting these surfaces to recover edges and vertices. ParaCAD places point-cloud-conditioned generation at the core of B-Rep synthesis, making it practical for user-guided reconstruction and seamless integration into existing 3D generation pipelines. Extensive experiments demonstrate that ParaCAD produces accurate B-Reps with faithful point-cloud alignment, outperforming point-based baselines in geometric precision, robustness, watertightness and downstream usability.
Despite recent progress, text-to-image models still struggle to generate semantically diverse and compositionally accurate multi-person interaction scenes, often collapsing to repetitive layouts, stereotypical poses, and poorly grounded interactions. In this work, we bridge this gap by introducing a dual pose–image representation that brings person-centric structural priors into pretrained diffusion transformers. Our model jointly predicts a 2D pose visualization image and its corresponding RGB image, enabling structure and appearance to co-evolve during learning. At its core, a cross-modal alignment scheme binds text, pose, and image representations, ensuring consistent grounding across modalities. Furthermore, we design an iterative scene construction scheme, progressively generating complex multi-human interactions while effectively decomposing the overall generation complexity. Extensive experiments demonstrate that our method substantially improves prompt alignment and scene diversity in multi-person image generation.