To enable large-scale reuse of real-world 3D assets-where garments and characters rarely share skeletons, templates, or dense correspondences-we present a fully automated virtual try-on system that dresses complex, multi-layer garments onto diverse, arbitrarily posed humanoids. Our key idea is to use SMPL as an intermediate proxy and decompose clothing-to-body transfer into two correspondence tasks with distinct challenges: (1) clothing-to-SMPL (partial-to-complete alignment) and (2) body-to-SMPL (large pose/shape variation and stylization). We address clothing-to-SMPL using a geometry-driven correspondence model, and introduce a diffusion-based body-to-SMPL correspondence approach that leverages multi-view consistent appearance features together with a pretrained 2D foundation model. Using these correspondences, we register SMPL/SMPL+D (Displacement) to the garment and target body and then perform simulator-driven fitting by transferring the garment along a smooth SMPL→SMPL+D transition, producing physically plausible draping on the target. Our system handles complex garment topology (including non-manifold meshes) and generalizes to a wide range of humanoid characters (e.g., humans, robots, cartoons, and creatures) while remaining computationally practical. Upon draping, our system also supports fast customization of clothing size. We show that our system can produce high-quality 3D clothing fittings without any human labor, even when 2D clothing sewing patterns are not available. Our project page is: https://cao-cong0.github.io/LUIVITON-Learned-Universal-Interoperable-VIrtual-Try-ON/.
论文检索
输入标题、作者或关键词，从 2,942 篇学术成果中精准定位
Retargeting human kinematic reference motion onto a robot's morphology remains a formidable challenge. Existing methods often produce physical inconsistencies, such as foot sliding, self-collisions, or dynamically infeasible motions, which hinder downstream imitation learning. We propose a bilevel optimization framework that jointly adapts reference motions to a robot's morphology while training a tracking policy using reinforcement learning. To make the optimization tractable, we derive an approximate gradient for the upper-level loss. Our framework requires only a sparse set of semantic rigid-body correspondences and eliminates the need for manual tuning by identifying optimal values for a parameterization expressive enough to preserve characteristic motion across different embodiments. Moreover, by integrating retargeting directly with physics simulation, we produce physically plausible motions that facilitate robust imitation learning. We validate our method in simulation and on hardware, demonstrating challenging motions for morphologies that differ significantly from a human, including retargeting onto a quadruped.
Generating realistic 3D human motions in real-time within interactive applications is key for animation, simulation, and humanoid robotics. While recent offline motion generation approaches offer precise control via text and kinematic constraints, they lack the inference speed required for interactive settings. Conversely, existing online methods enable real-time synthesis but often sacrifice controllability or struggle with complex text semantics and long-horizon goals due to limited context windows. In this work, we introduce ARDY, a streaming generation framework that bridges this gap by enabling high-fidelity motion generation controllable via online text prompts and flexible kinematic constraints. ARDY employs a hybrid representation that combines explicit root features with a latent body embedding, balancing precise trajectory control with efficient generative learning. We propose a two-stage autoregressive transformer denoiser that features variable history context and supports conditioning on flexible, long-horizon kinematic constraints. By training on a large-scale motion capture dataset and being directly conditioned on text labels and kinematic constraints sampled from ground truth poses, ARDY natively learns controllable generation that supports online prompting and flexible long-horizon goals. Extensive evaluations on the HumanML3D benchmark and the large-scale, high-fidelity Bones Rigplay dataset demonstrate ARDY's high motion quality and constraint adherence, validating the efficacy of our key architectural decisions. Finally, we demonstrate the method's practical versatility through an interactive demo featuring dynamic text control, diverse keyframe pose constraints, path following, and interactive locomotion control via mouse and keyboard. Supplementary video results, code, and model releases can be found at https://research.nvidia.com/labs/sil/projects/ardy/.
Despite transformative advances in generative motion synthesis, real-time interactive motion control remains dominated by traditional techniques. In this work, we identify two key challenges in bridging research and production: 1) Real-time scalability: Industry applications demand real-time generation of a vast repertoire of motion skills, while generative methods exhibit significant degradation in quality and scalability under real-time computation constraints, and 2) Integration: Industry applications demand fine-grained multi-modal control involving velocity commands, style selection, and precise keyframes, a need largely unmet by existing text- or tag-driven models. Moreover, a systematic motion design interface for generative models remains absent. To overcome these limitations, we introduce MotionBricks: a large-scale, real-time generative framework with a two-fold solution. First, we propose a large-scale modular latent generative backbone tailored for robust real-time motion generation, effectively modeling a dataset of over 350,000 motion clips with a single model. Second, we introduce smart primitives that provide a unified, robust, and intuitive interface for authoring both navigation and object interaction. Notably, MotionBricks applies to new downstream tasks in a zero-shot manner, where no fine-tuning or task-specific tagging is required. Applications can be designed in a plug-and-play manner like assembling bricks without expert animation knowledge, enabling an accessible interface for applications in animation and robotics. Quantitatively, we show that MotionBricks produces state-of-the-art motion quality on open-source and proprietary datasets of various scales, while also achieving a real-time throughput of 15,000 FPS with 2ms latency. We demonstrate the flexibility and robustness of MotionBricks in a complete production-level animation demo, covering navigation and object-scene interaction across various styles with a unified model. To showcase our framework's application beyond animation, we deploy MotionBricks on the Unitree G1 humanoid robot to demonstrate its flexibility and generalization for real-time robotic control.
Online 3D reconstruction from monocular image sequences is a challenging and ongoing research topic. 3D Gaussian Splatting (3DGS), leveraging its high-quality real-time rendering capability, empowers online 3D reconstruction to represent dense scenes with enhanced expressiveness, and thus holds great promise for a wide range of applications such as robotics and AR/VR. However, existing online 3DGS methods still suffer from some key challenges: fragile camera pose estimation due to the lack of global optimization, and low optimization efficiency in large-scale or long-sequence scenarios. To address these issues, we propose a robust and efficient online voxelized 3DGS reconstruction framework integrated with global Sim(3) optimization, which enables reliable camera tracking and efficient global loop closure for both camera poses and voxelized 3DGS. To accelerate the convergence of the voxelized 3DGS, we further introduce a color residual learning strategy, which not only boosts optimization speed but also enhances rendering quality. Extensive experiments on diverse indoor and outdoor datasets demonstrate that our method achieves state-of-the-art performance in both camera pose estimation accuracy and rendering quality, while retaining real-time efficiency. Additionally, we develop and deploy a real-world UAV-based active reconstruction system grounded on our proposed method, validating its robustness and generalizability for practical online 3D reconstruction tasks. Our code and data are available at https://github.com/TrickyGo/MoonSplat..
Anisotropic friction is a critical source of propulsion for efficient locomotion of many terrestrial animals. The interplay between animal morphology, control, and anisotropically frictional contact makes designing optimal anisotropic friction for terrestrial locomotion intriguing and challenging. We propose a computational pipeline for co-designing anisotropic friction and controllers of terrestrial robots with diverse morphologies. Our pipeline presents a co-design algorithm that alternates between optimizing direction of anisotropic friction and training a neural network controller to improve the locomotion performance of a given robot morphology. Based on the intuition that controller’s performance does not change significantly when the frictional force differs slightly, we introduce the concept of trust-region into robot co-design, allowing the controller network to continue training from the previous iteration. Our evaluation on various morphologies show that anisotropic friction is critical for terrestrial robot locomotion, and our pipeline is statistically better than current state-of-the-art methods. Furthermore, we reveal that large language models (LLM) constitute a strong baseline for this kind of co-design problems, worth receiving more attention. We demonstrate that co-designing anisotropic friction and control unlocks effective locomotion in various downstream tasks, including locomotion on uneven terrain, navigation in a maze, and object manipulation. To validate our pipeline in the real world, we design and 3D print a variety of scales and systematically measure their anisotropic friction coefficients. Then we construct a multi-link robot with anisotropic scales designed by our pipeline and compare its performance with isotropic scales. Our real-world experiments confirm that isotropic scales are insufficient to support terrestrial robots’ locomotion abilities, and computationally co-designing friction and control enables robots to perform tasks including turning, slithering, and other non-trivial locomotion tasks.
Coordinate-motion assemblies can only be disassembled by the simultaneous motion of multiple parts along distinct paths, providing high structural stability and enabling efficient robotic assembly. Existing examples are largely limited to architectural structures using joint-based connections, or puzzles created through trial-and-error, as the relationship between part geometry and coordinate motion remains poorly understood. Computationally designing such assemblies is challenging because it requires jointly achieving distributed contacts across the entire assembly and a unique coordinate motion for disassembly that rules out other feasible motions. We address this challenge by establishing a theoretical connection between part geometry and unique coordinate motion, enabling us to rigorously verify whether a given assembly admits a unique coordinate motion. Building on this theory, we introduce a two-stage algorithm that optimizes contact interfaces for a target motion and constructs physically feasible part geometries that conform to a user-specified global shape. We demonstrate our approach on models with complex geometries and topologies, including assemblies with large part counts, and validate it through physical fabrication and experiments.
Imaging objects hidden outside the direct line of sight expands the effective field of view and is critical for applications such as autonomous driving and robotic perception. Despite impressive progress in time-of-flight (ToF)-based non-line-of-sight (NLOS) imaging, real-world deployment remains challenging because practical measurements are often collected over spatially limited, arbitrarily shaped relay regions—conditions that violate the planar-wall and dense-sampling assumptions made by most existing methods. To address these limitations, we propose a LOS-guided NLOS imaging pipeline that imposes no geometric assumptions on the relay surface and naturally supports both confocal and non-confocal configurations. Our method represents the hidden scene using 3D Gaussian primitives and couples them with an efficient, differentiable transient rendering model, enabling end-to-end optimization directly from measured transients. We validate our approach on real-world measurements from both a public dataset and a custom-built capture system. Across settings, our method achieves state-of-the-art reconstruction fidelity under spatially limited, sparsely sampled conditions, and significantly outperforms existing methods on complex, arbitrary relay surface geometries. We release our code and datasets at nlos-3d-gtr.
Volumetric video (VV) streaming delivers truly immersive viewing experiences over the Internet, serving as a critical foundation for next-generation applications, including immersive telepresence in the metaverse, the surveillance of remote ecological systems, and robotic teleoperation for embodied AI, and beyond. Beyond immersive viewing, these applications turn VV streaming into a real-time interface to remote physical environments, imposing new system-level demands for photorealistic scene representation, low-latency interaction, and robust performance under heterogeneous network conditions. 3D Gaussian Splatting (3DGS) has been widely used for real-time photorealistic rendering, offering superior visual quality and rendering performance, but it faces challenges due to bandwidth consumption. Furthermore, as the foundation of adaptive VV streaming, existing Levels of Detail (LoD) methods based on density are not well-suited to Gaussian representations, leading to visible gaps and severe quality degradation. Recent studies have also explored attribute compression techniques to reduce bandwidth consumption. Our preliminary studies reveal that aggressive attribute compression primarily causes color distortion, which can be effectively corrected in the rendered image using a reference image. Motivated by these findings, we propose a novel Color-Adaptive scheme for adaptive VV streaming that uses vector quantization (VQ) to establish LoDs and correct color distortions with low-resolution reference images. We further present CAGS, an adaptive VV streaming system compatible with diverse Gaussian representations, which integrates the Color-Adaptive scheme by rendering reference images on the streaming server and performing color restoration on the client. Extensive experiments on our prototype system demonstrate that CAGS outperforms the existing adaptive streaming systems in PSNR by 5 ∼ 20 dB under fluctuating bandwidth, operates significantly faster than existing scalable Gaussian compression methods, and generalizes across different Gaussian representations. The code is available at https://github.com/yindaheng98/ColorAdaptiveGaussianSplatting.
Per- and polyfluoroalkyl substances (PFAS) are persistent environmental contaminants that require integrated, semantically structured representations of chemical identity, classification, and properties to support integrated contaminant monitoring and analysis. This work presents the CompTox ontology, an expert-guided ontology describing commonly analyzed PFAS and designed to support PFAS data integration and querying. The ontology organizes PFAS hierarchically according to key physicochemical characteristics and incorporates authoritative identifiers and properties from EPA's CompTox Chemicals Dashboard. Individual PFAS are annotated with core chemical identifiers, including DTXSID, CASRN, InChIKey, and SMILES; with physicochemical attributes such as molecular mass, carbon chain length, and functional group information; and with observed or predicted environmental fate and transport and toxicological information. The ontology was constructed using the Knowledge Acquisition and Representation Methodology (KNARM), employing a template-driven workflow implemented with the ROBOT tool to generate an OWL-formatted ontology. An expert-guided hierarchy captures major PFAS classes, including fluorotelomers, perfluoroalkyl acids (both Perfluoroalkyl Carboxylic and Sulfonic Acids), and perfluoroalkyl ether acids. Human-readable IRIs and SKOS alternative labels enhance usability. The ontology helps facilitate integrated querying and analysis of PFAS contamination within the SAWGraph knowledge graphs but also serves as a flexible and extensible framework for unified chemical identification and classification.
Vision–Language–Action (VLA) models underpin robotic and other embodied agents by mapping visual observations and language instructions into executable actions. Their wide adoption through open web model repositories, however, introduces new supply-chain risks: adversaries can launch adversarial attacks to manipulate the action outputs of VLAs, potentially leading to harmful real-world outcomes for embodied agents. To exploit this vulnerability, we propose MAVLA, a novel multimodal adversarial attack framework. MAVLA serves as a modular front-end that integrates seamlessly with a target VLA model, injecting perturbations into task-relevant and structure-sensitive image regions to disrupt cross-modal alignment and induce deviations in the generated action instructions. To balance attack effectiveness with stealth, we design four loss functions that jointly maximize multimodal misalignment while preserving visual stealthiness. Extensive evaluations in simulated and real-world scenarios show that at a 40% perturbation ratio, the task success rate of VLAs drops by about 70%. Compared to conventional attack baselines, MAVLA achieves superior attack effectiveness and stealthiness with low overhead. Our work reveals a practical and previously underexplored threat to embodied systems, and offers a red-team baseline to inform future defensive strategies and promote safer VLA deployment.
The Digital Roads project at Cambridge University is leveraging digitalisation, automation, and low-carbon materials to build an Automated Maintenance Plant (AMP) for UK road networks, aimed at minimising repair times to reduce congestion, improving safety, and contributing to the UK’s net-zero goals through faster, more accurate, and efficient road maintenance.
Cities deploying autonomous vehicles face an urgent policy question: would the adoption of autonomous vehicles (AVs) improve the congestion rate or worsen it? What would be the optimal adoption rate to minimize the congestion rate? How would cautious AVs (Waymo-style) and aggressive AVs (Tesla "Mad Max'"-style) interact with human drivers and delivery robots on shared roads? We present UrbanMix, an interactive simulation platform that embeds cognitively diverse agents (human drivers, cautious AVs, aggressive AVs, and delivery robots) with distinct behavioral profiles inside a Simulation of Urban MObility (SUMO) framework of real urban road networks. Our LLM planner operates as an urban policy coordinator, setting traffic rules through a bounded action interface, while a regulation shield enforces infrastructure constraints. Our experiments reveal three key phenomena: (i) roads throughput may substantially drop with the increase in AVs adoption; (ii) an aggressive cascade, where runtime behavior switching modeling Tesla user-selectable "Mad Max" mode triggers up to 60 times increase in emergency braking events on a real Austin Downtown network; and (iii) a delivery bottleneck, showing 24-36% throughput reduction from slow robots. Results validated on synthetic and real data from Austin, TX, demonstrate that real network topology amplifies cascade effects by more than four times.
While Open-Set Semantic Mapping and 3D Semantic Scene Graphs (3DSSGs) have become established paradigms in robotic perception in recent years, most existing works are limited to small environments or sacrifice geometric detail and instance granularity for scalability. Deploying these systems at scale for large multi-room environments remains a major challenge due to the computational overhead of high-dimensional feature integration and the maintenance of the 3DSSG structure. In this paper, we demonstrate a modular mapping architecture that establishes 3D Semantic Scene Graphs (3DSSGs) as its foundational backend. Unlike approaches that generate scene graphs as a post-processing step, our system maintains the graph as the primary, incrementally updated knowledge representation. Our architecture is optimized for GPU-accelerated operations, enabling the dense representation of extensive environments containing thousands of unique object instances, supporting open-vocabulary queries via CLIP features without requiring any additional post-processing steps. In this live demonstration, we showcase our pipeline processing large-scale data from the Habitat Matterport 3D (HM3D) dataset as well as live data collected from a handheld device. Attendees will interact with the generated maps by performing real-time, open-set queries (e.g., “find the vintage wooden chair”) across complex, multi-story environments, highlights the system's capability to represent dynamic, human-aligned environmental understanding suitable for downstream robotic tasks.
In agricultural tasks, manual grape harvesting remains a labor-intensive activity facing challenges of efficiency, labor shortages, and sustainability. To this end, tailor-made robotic systems have been designed with the capabilities to transfer heavy boxes, navigate vineyard terrains, communicate, accurately locate, and safely interact with humans. The introduction of collaborative robotic fleets alongside human workers in large-scale vineyard harvesting effectively presents a Multi-Robot Task Allocation (MRTA) problem, where the real-world domain possesses characteristics that, when combined, pose a challenging research endeavor and align with open issues in MRTA research. Here, we present RoboVineSim, a simulation tool that can capture any vineyard area using geographical data and model the behavior of humans and robots in the environment. In addition, we have established the necessary mechanisms to facilitate the development of novel MRTA methods in this domain.
While we have witnessed extraordinary progress across various subfields of AI, the deployment of these systems in safety-critical and mission-critical domains has lagged behind. A key requirement for such deployments is the availability of AI systems capable of generating optimal behavior that can be formally verified and effectively used by non-AI experts from diverse backgrounds. While significant advances have been made toward building such systems, we still lack comprehensive formal frameworks to model and analyze the complex dynamics of human-AI interaction. In this talk, I will introduce the Human-Aware AI framework, a multi-agent planning framework specifically designed to support and reason about human-AI interaction. I will show how this framework provides novel solutions to challenges such as explainability, value alignment, and proactive assistance, and demonstrate its application in domains including intelligent tutoring systems, cybersecurity, and robotics.
Recent progress in generalizable embodied control has been driven by large-scale pretraining of Vision–Language–Action (VLA) models. However, most existing approaches rely on large collections of robot demonstrations, which are costly to obtain and tightly coupled to specific embodiments. Human videos, by contrast, are abundant and capture rich interactions, providing diverse semantic and physical cues for real-world manipulation. Yet, embodiment differences and the frequent absence of task-aligned annotations make their direct use in VLA models challenging. This survey provides a unified view of how human videos are transformed into effective knowledge for VLA models. We categorize existing approaches into four classes based on the action-related information they derive: (i) latent action representations that encode inter-frame changes; (ii) predictive world models that forecast future frames; (iii) explicit 2D supervision that extracts image-plane cues; and (iv) explicit 3D reconstruction that recovers geometry or motion. Beyond this taxonomy, we highlight three key open challenges in this area: structuring unstructured videos into training-ready episodes, grounding video-derived supervision into robot-executable actions under embodiment and viewpoint heterogeneity, and designing evaluation protocols that better predict real-world deployment performance and transfer efficiency, thereby informing future research directions.
We examine a particular adversarial human--robot interaction for which mobile robots influence the behavior of people role-playing as a school shooter. Using a high-fidelity virtual reality environment, we conducted a controlled study with 150 participants. Two autonomous robots predicted participant movement and positioned themselves to interfere and distract. The robots' approach strategy was manipulated---either moving directly into the participant's path (aggressive) or maintaining distance (passive)---along with the level of distraction, ranging from no additional cues (low), to siren and lights (medium), to siren, lights, and smoke to impair visibility (high). An aggressive, high-distraction robot configuration reduced the number of victims by 46.6% relative to a no-robot control. These results show that robot-based distraction can meaningfully alter human behavior and outcomes in adversarial settings, while also raising important ethical questions about the use of such systems in school environments.
Motion planning among multiple robots in a shared space is a fundamental yet computationally challenging problem in robotics, with applications ranging from warehouse automation to autonomous fleets. In this work, we introduce a fast, scalable motion planner that achieves real-time, collision-free trajectory planning via a two-staged algorithm combining deterministic search-based planning with machine learning-driven conflict resolution. We present a prioritized Safe Interval Path Planning algorithm (SIPP-PP) with a novel limited goal reservation strategy to prevent goal-blocking conflicts while allowing shared goal regions. We added a second layer of ML-guided Large Neighborhood Search (LNS) procedure to our SIPP-PP algorithm for improving success rates in highly congested environments via intelligent selection of conflict resolution actions. The result is a planning system that generates collision-free paths for multiple robots in complex environments within tens of milliseconds. For example, compared to recent advanced learning-based methods such as diffusion planners, our planner is two-to-three orders of magnitude faster. Our work demonstrates a multi-robot planner capable of real-time operation in dense scenarios, satisfying the stringent requirements of industrial applications such as drive units in fulfillment centers.
This paper addresses the Motion Execution Gap, the disconnect between high-level symbolic task descriptions using semantic constraints and executable robot motions. \textbf{Motion Statecharts} are introduced as an executable symbolic representation for complex motions. They allow the arbitrary arrangement of motion constraints, monitors or nested statecharts in parallel and sequence. World-centric motion specification and generalization across embodiments are enabled through the use of a unified differentiable kinematic world model of both, robots and environments. Motion execution is realized through a \ac{lmpc}-based implementation of the task-function approach, in which smooth transitions during task switches are ensured using jerk bounds. Cross-platform transferability was demonstrated by deploying the method on eight robot platforms, operating in diverse environments. The proposed framework is called Giskard and is available open source (https://github.com/cram2/cognitive_robot_abstract_machine).