Oculoplastic surgery is a critical treatment for various eye conditions, such as ptosis, which can cause both aesthetic and functional issues. Due to the anxiety about the outcome, patients are often hesitant to undergo the necessary procedures required for the surgery. Virtual oculoplastic surgery simulation technology offers a solution to alleviate these concerns by providing realistic previews of post-surgical results. In this paper, we present a novel deep learning-based virtual oculoplastic surgery simulation system that addresses the limitations of existing methods. The proposed system aims to improve the accuracy of simulations by considering the anatomical structure and characteristics of the eye. Our method utilizes a deformable parametric mesh to enhance the controllability of the image transformation process. Furthermore, the combination of a style-based generator and a neural texture has been implemented to generate high-quality results. The proposed system is expected to facilitate better communication between doctors and patients by providing anatomically inspired high-quality simulation results. The development of this advanced virtual simulation system has the potential to enhance patient experiences and improve satisfaction with outcomes in the field of oculoplastic surgery.
论文检索
输入标题、作者或关键词，从 306 篇学术成果中精准定位
Systematically simulating specular light transport requires an exhaustive search for triangle tuples containing admissible paths. Given the extreme inefficiency of enumerating all combinations, we significantly reduce the search domain by stochastically sampling such tuples. The challenge is to design proper sampling probabilities that keep the noise level controllable. Our key insight is that by bounding the irradiance contributed by each triangle tuple at a given position, we can sample a subset of triangle tuples with potentially high contributions. Although low-contribution tuples are assigned a negligible probability, the overall variance remains low. Therefore, we derive position and irradiance bounds for caustics casted by each triangle tuple, introducing a bounding property of rational functions on a Bernstein basis. When formulating position and irradiance expressions into rational functions, we handle non-rational parts through remainder variables to maintain bounding validity. Finally, we carefully design the sampling probabilities by optimizing the upper bound of the variance, expressed only using the position and irradiance bounds. The bound-driven sampling of triangle tuples is intrinsically unbiased even without defensive sampling. It can be combined with various unbiased and biased root-finding techniques within a local triangle domain. Extensive evaluations show that our method enables the fast and reliable rendering of complex caustics effects. Yet, our method is efficient for no more than two specular vertices, where complexity grows sublinearly to the number of triangles and linearly to that of emitters, and does not consider the Fresnel and visibility terms. We also rely on parameters to control subdivisions.
3D creation has always been a unique human strength, driven by our ability to deconstruct and reassemble objects using our eyes, mind and hand. However, current 3D design tools struggle to replicate this natural process, requiring considerable artistic expertise and manual labor. This paper introduces BANG, a novel generative approach that bridges 3D generation and reasoning, allowing for intuitive and flexible part-level decomposition of 3D objects. At the heart of BANG is "Generative Exploded Dynamics", which creates a smooth sequence of exploded states for an input geometry, progressively separating parts while preserving their geometric and semantic coherence. BANG utilizes a pre-trained large-scale latent diffusion model, fine-tuned for exploded dynamics with a lightweight exploded view adapter, allowing precise control over the decomposition process. It also incorporates a temporal attention module to ensure smooth transitions and consistency across time. BANG enhances control with spatial prompts, such as bounding boxes and surface regions, enabling users to specify which parts to decompose and how. This interaction can be extended with multimodal models like GPT-4, enabling 2D-to-3D manipulations for more intuitive and creative workflows. The capabilities of BANG extend to generating detailed part-level geometry, associating parts with functional descriptions, and facilitating component-aware 3D creation and manufacturing workflows. Additionally, BANG offers applications in 3D printing, where separable parts are generated for easy printing and reassembly. In essence, BANG enables seamless transformation from imaginative concepts to detailed 3D assets, offering a new perspective on creation that resonates with human intuition.
We present GenAnalysis, an implicit shape generation framework that allows joint analysis of man-made shapes, including shape matching and joint shape segmentation. The key idea is to enforce an as-affine-as-possible (AAAP) deformation between synthetic shapes of the implicit generator that are close to each other in the latent space, which we achieve by designing a regularization loss. It allows us to understand the shape variation of each shape in the context of neighboring shapes and also offers structure-preserving interpolations between the input shapes. We show how to extract these shape variations by recovering piecewise affine vector fields in the tangent space of each shape. These vector fields provide single-shape segmentation cues. We then derive shape correspondences by iteratively propagating AAAP deformations across a sequence of intermediate shapes. These correspondences are then used to aggregate single-shape segmentation cues into consistent segmentations. We conduct experiments on the ShapeNet dataset to show superior performance in shape matching and joint shape segmentation over previous methods.
In this paper, we propose a novel approach to refine 3D shape correspondences by leveraging multi-layer perceptions within the framework of functional maps. Central to our contribution is the concept of Neural Adjoint Maps, a novel neural representation that generalizes the traditional solution of functional maps for estimating correspondence between manifolds. Fostering our neural representation, we propose an iterative algorithm explicitly designed to enhance the precision and robustness of shape correspondence across diverse modalities such as meshes and point clouds. By harnessing the expressive power of non-linear solutions, our method captures intricate geometric details and feature correspondences that conventional linear approaches often overlook. Extensive evaluations on standard benchmarks and challenging datasets demonstrate that our approach achieves state-of-the-art accuracy for both isometric and non-isometric meshes and for point clouds where traditional methods frequently struggle. Moreover, we show the versatility of our method in tasks such as signal and neural field transfer, highlighting its broad applicability to domains including computer graphics, medical imaging, and other fields demanding precise transfer of information among 3D shapes. Our work sets a new standard for shape correspondence refinement, offering robust tools across various applications.
We present 3DGH, an unconditional generative model for 3D human heads with composable hair and face components. Unlike previous work that entangles the modeling of hair and face, we propose to separate them using a novel data representation with template-based 3D Gaussian Splatting, in which deformable hair geometry is introduced to capture the geometric variations across different hairstyles. Based on this data representation, we design a 3D GAN-based architecture with dual generators and employ a cross-attention mechanism to model the inherent correlation between hair and face. The model is trained on synthetic renderings using carefully designed objectives to stabilize training and facilitate hair-face separation. We conduct extensive experiments to validate the design choice of 3DGH, and evaluate it both qualitatively and quantitatively by comparing with several state-of-the-art 3D GAN methods, demonstrating its effectiveness in unconditional full-head image synthesis and composable 3D hairstyle editing. More details will be available on our project page: https://c-he.github.io/projects/3dgh/.
Existing facial appearance capture methods can reconstruct plausible facial reflectance from smartphone-recorded videos. However, the reconstruction quality is still far behind the ones based on studio recordings. This paper fills the gap by developing a novel daily-used solution with a co-located smartphone and flashlight video capture setting in a dim room. To enhance the quality, our key observation is to solve facial reflectance maps within the data distribution of studio-scanned ones. Specifically, we first learn a diffusion prior over the Light Stage scans and then steer it to produce the reflectance map that best matches the captured images. We propose to train the diffusion prior at the patch level to improve generalization ability and training stability, as current Light Stage datasets are in ultra-high resolution but limited in data size. Tailored to this prior, we propose a patch-level posterior sampling technique to sample seamless full-resolution reflectance maps from this patch-level diffusion model. Experiments demonstrate our method closes the quality gap between low-cost and studio recordings by a large margin, opening the door for everyday users to clone themselves to the digital world.
State-of-the-art neural rendering methods optimize Gaussian scene representations from a few photographs for novel-view synthesis. Building on these representations, we develop an efficient algorithm, dubbed Gaussian Wave Splatting, to turn these Gaussians into holograms. Unlike existing computergenerated holography (CGH) algorithms, Gaussian Wave Splatting supports accurate occlusions and view-dependent effects for photorealistic scenes by leveraging recent advances in neural rendering. Specifically, we derive a closed-form solution for a 2D Gaussian-to-hologram transform that supports occlusions and alpha blending. Inspired by classic computer graphics techniques, we also derive an efficient approximation of the aforementioned process in the Fourier domain that is easily parallelizable and implement it using custom CUDA kernels. By integrating emerging neural rendering pipelines with holographic display technology, our Gaussian-based CGH framework paves the way for next-generation holographic displays.
Optical heterodyne detection (OHD) employs coherent light and optical interference techniques (Fig. 1-(A)) to extract physical parameters, such as velocity or distance, which are encoded in the frequency modulation of the light. With its superior signal-to-noise ratio compared to incoherent detection methods, such as time-of-flight lidar, OHD has become integral to applications requiring high sensitivity, including autonomous navigation, atmospheric sensing, and biomedical velocimetry. However, current simulation tools for OHD focus narrowly on specific applications, relying on domain-specific settings like restricted reflection functions, scene configurations, or single-bounce assumptions, which limit their applicability. In this work, we introduce a flexible and general framework for spectral-domain simulation of OHD. We demonstrate that classical radiometry-based path integral formulation can be adapted and extended to simulate the OHD measurements in the spectral domain. This enables us to leverage the rich modeling and sampling capabilities of existing Monte Carlo path tracing techniques. Our formulation shares structural similarities with transient rendering but operates in the spectral domain and accounts for the Doppler effect (Fig. 1-(B)). While simulators for the Doppler effect in incoherent (intensity) detection methods exist, they are largely not suitable to simulate OHD. We use a microsurface interpretation to show that these two Doppler imaging techniques capture different physical quantities and thus need different simulation frameworks. We validate the correctness and predictive power of our simulation framework by qualitatively comparing the simulations with real-world captured data for three different OHD applications—FMCW lidar, blood flow velocimetry, and wind Doppler lidar (Fig. 1-(C)).
Modern nanofabrication techniques have enabled us to manipulate the wave-front of light with sub-wavelength-scale structures, offering the potential to replace bulky refractive surfaces in conventional optics with ultrathin metasurfaces. In theory, arrays of nanoposts provide unprecedented control over manipulating the wavefront in terms of phase, polarization, and amplitude at the nanometer resolution. A line of recent work successfully investigates flat computational cameras that replace compound lenses with a single metalens or an array of metasurfaces a few millimeters from the sensor. However, due to the inherent wavelength dependence of metalenses, in practice, these cameras do not match their refractive counterparts in image quality for broadband imaging, and may even suffer from hallucinations when relying on generative reconstruction methods. In this work, we investigate a collaborative array of metasurface elements that are jointly learned to perform broadband imaging. To this end, we learn a nanophotonics array with 100-million nanoposts that is end-to-end jointly optimized over the full visible spectrum—a design task that existing inverse design methods or learning approaches cannot support due to memory and compute limitations. We introduce a distributed meta-optics learning method to tackle this challenge. This allows us to optimize a large parameter array along with a learned metaatom proxy and a non-generative reconstruction method that is parallax-aware and noise-aware. The proposed camera performs favorably in simulation and in all experimental tests irrespective of the scene illumination spectrum.
In this paper, we present a holographic display using a light pipe for augmented reality, and the hologram rendering method via bandwidth-preserved kaleidoscopic guiding method. Conventional augmented reality displays typically share optical architectures where the light engine and image combiner are adjacent. Minimizing the size of both components is highly challenging, and most commercial and research prototypes of augmented reality displays are bulky, front-heavy and sight-obstructing. Here, we propose the use of light pipe to decouple and spatially reposition the light engine from the image combiner, enabling a pragmatic glasses-type design. Through total internal reflection, light pipes have an advantage in guiding the full angular bandwidth regardless of its length. By modeling such kaleidoscopic guiding of the wavefront inside the light pipe and applying it to holographic image generation, we successfully separate the light engine from the image combiner, making the front of the device clear and lightweight. We experimentally validate that the proposed light pipe system delivers virtual images with high-quality and 3D depth cues. We further present a method to simulate and compensate for light pipe misalignment, enhancing the robustness and practicality of the proposed system.
The recently developed Progressive Dynamics framework [Zhang et al. 2024] addresses the long-standing challenge in enabling rapid iterative design for high-fidelity cloth and shell animation. In this work, we identify fundamental limitations of the original method in terms of stability and temporal continuity. For robust progressive dynamics simulation we seek methods that provide: (1) stability across all levels of detail (LOD) and timesteps, (2) temporally continuous animations without jumps or jittering, and (3) user-controlled balancing between geometric consistency and enrichment at each timestep, thereby making it a practical previewing tool with high-quality results at the finest level to be used as the final output. We propose a general framework, Progressive Dynamics++, for constructing a family of progressive dynamics integration methods that advance physical simulation states forward in both time and spatial resolution, which includes Zhang et al. [2024]'s method as one member. We analyze necessary stability conditions for Progressive Dynamics integrators and introduce a novel, stable method that significantly improves temporal continuity, supported by a new quantitative measure. Additionally, we present a quantitative analysis of the trade-off between geometric consistency and enrichment, along with strategies for balancing between these aspects in transitions across resolution and time.
该来源暂未提供摘要。
Non-rigid registration of 3D shape collections using a template mesh is essential for constructing 3D datasets. Traditional non-rigid Iterative Closest Point (ICP) methods rely on manually selected template meshes, which can result in inconsistent registrations when applied to diverse shape collections. This inconsistency arises particularly when the template lacks common shape features with the input instances or when landmark annotations are sparse. To overcome this limitation, we propose a novel ICP framework that jointly optimizes a shared template shape and its instance-wise deformations. Our joint optimization framework assigns distinct roles to the shared template and instance-wise deformations: the template captures common shape features, while instance-wise deformations handle residual registration errors. We use stronger smoothness regularization on the instance-wise deformations in early iterations to prioritize the accumulation of common details on the template. Additionally, a distortion alignment energy minimizes interinstance map distortions, promoting consistent instance-wise deformations. On challenging 3D datasets with large shape variations, our method achieves state-of-the-art fitting accuracy and consistent results in shape averaging and deformation transfer. By removing the need for a carefully selected preset template, our method extends the capability of extrinsic non-rigid registration frameworks, offering a more robust and flexible solution for challenging registration scenarios.
We propose a novel method (FaCE) for normal estimation of unoriented point clouds and VR ribbon sketches that leverages a modeling of the Faraday cage effect. Input points, or a sampling of the ribbons, form a conductive cage and shield the interior from external fields. The gradient of the maximum field strength over external field scenarios is used to estimate a normal at each input point or ribbon. The electrostatic effect is modeled with a simple Poisson system, accommodating intuitive user-driven sculpting via the specification of point charges and Faraday cage points. On inputs sampled from clean, watertight meshes, our method achieves comparable normal quality to existing methods tailored for this scenario. On inputs containing interior structures and artifacts, our method produces superior surfacing output when combined with Poisson Surface Reconstruction. In the case of ribbon sketches, our method accommodates sparser ribbon input while maintaining an accurate geometry, allowing for greater flexibility in the artistic process. We demonstrate superior performance to an existing approach for surfacing ribbon sketches in this sparse setting.
Orienting point clouds is a fundamental problem in computer graphics and 3D vision, with applications in reconstruction, segmentation, and analysis. While significant progress has been made, existing approaches mainly focus on watertight, object-level 3D models. The orientation of large-scale, non-watertight 3D scenes remains an underexplored challenge. To address this gap, we propose DACPO (Divide-And-Conquer Point Orientation), a novel framework that leverages a divide-and-conquer strategy for scalable and robust point cloud orientation. Rather than attempting to orient an unbounded scene at once, DACPO segments the input point cloud into smaller, manageable blocks, processes each block independently, and integrates the results through a global optimization stage. For each block, we introduce a two-step process: estimating initial normal orientations by a randomized greedy method and refining them by an adapted iterative Poisson surface reconstruction. To achieve consistency across blocks, we model inter-block relationships using an an undirected graph, where nodes represent blocks and edges connect spatially adjacent blocks. To reliably evaluate orientation consistency between adjacent blocks, we introduce the concept of the visible connected region, which defines the region over which visibility-based assessments are performed. The global integration is then formulated as a 0-1 integer-constrained optimization problem, with block flip states as binary variables. Despite the combinatorial nature of the problem, DACPO remains scalable by limiting the number of blocks (typically a few hundred for 3D scenes) involved in the optimization. Experiments on benchmark datasets demonstrate DACPO's strong performance, particularly in challenging large-scale, non-watertight scenarios where existing methods often fail. The source code is available at https://github.com/zd-lee/DACPO.
Human-centered live events have always played a pivotal role in shaping culture and fostering social connections. Traditional 2D live transmissions fail to replicate the immersive quality of physical attendance. Addressing this gap, this paper proposes LiveSplats, a framework towards real-time, photo-realistic 3D reconstructions of live events using high-performance 3D Gaussian Splatting. Our solution capitalizes on strong geometric priors to optimize through distributed processing and load balancing, enabling interactive, freely explorable 3D experiences. By dividing scene reconstruction into actor-centric and environment-specific tasks, we employ hierarchical coarse-to-fine optimization to rapidly and accurately reconstruct human actors based on pose data, refining their geometry and appearance with photometric loss. For static environments, we focus on view-dependent appearance changes, streamlining rendering efficiency and maximizing GPU performance. To facilitate evaluation, we introduce (and distribute) a synthetic benchmark dataset of basketball games, offering high visual fidelity as ground truth. In both our synthetic benchmark and publicly available benchmarks, LiveSplats consistently outperforms existing approaches. The dataset is available at https://humansensinglab.github.io/basket-multiview.
In this paper, we propose a novel dynamic calibration method for sparse inertial motion capture systems, which is the first to break the restrictive absolute static assumption in IMU calibration, i.e., the coordinate drift RG′ G and measurement offset RBS remain constant during the entire motion, thereby significantly expanding their application scenarios. Specifically, we achieve real-time estimation of RG′ G and RBS under two relaxed assumptions: i) the matrices change negligibly in a short time window; ii) the human movements/IMU readings are diverse in such a time window. Intuitively, the first assumption reduces the number of candidate matrices, and the second assumption provides diverse constraints, which greatly reduces the solution space and allows for accurate estimation of RG′ G and RBS from a short history of IMU readings in real time. To achieve this, we created synthetic datasets of paired RG′ G, RBS matrices and IMU readings, and learned their mappings using a Transformer-based model. We also designed a calibration trigger based on the diversity of IMU readings to ensure that assumption ii) is met before applying our method. To our knowledge, we are the first to achieve implicit IMU calibration (i.e., seamlessly putting IMUs into use without the need for an explicit calibration process), as well as the first to enable long-term and accurate motion capture using sparse IMUs. The code and dataset are available at https://github.com/ZuoCX1996/TIC.