论文检索

Hanxiao Chen

Large Language Models (LLMs) have revolutionized the modern society significantly with the numerous advanced interactions between humans and AI agents, whereas the usage of most large language models including ChatGPT are not friendly open-sourced and must require users paying a lot for such AI service continuously. Therefore, deploying open-sourced large language models on local servers can be considered as an efficient method to design and implement creative embodied AI algorithms with lower cost and more stable free usage. Inspired by this ordinary motivation, we originally propose and implement the “Socratic Models-ChatGLM”, which is a well-performed algorithm for multi-modal interactive control of robotic arm based on offline large language models via the facile PyBullet platform, even presents extraordinary potential to address complicated text-image multi-step long-horizon robotic manipulation tasks.

Jiankai Sun

My research aims to pioneer efficient and reliable wearable intelligence algorithms that transform healthcare robotics into adaptive, patient-centered systems. I take a four-step approach: (1) design multimodal wearable sensing platforms to capture human and biometric signals; (2) train a foundation model that learns from these rich datasets to reason about human behaviors and health states; (3) validate the model through large-scale simulation and principled uncertainty quantification; and (4) deploy it in rehabilitation and assistive robots for intelligent, personalized care. This research not only advances fundamental understanding of multimodal human behavior, but also opens new pathways for early disease diagnosis, adaptive treatment, and accessible digital health. By bridging AI, wearables, and robotics, my work aspires to lay the groundwork for the next generation of healthcare technologies that are proactive, trustworthy, and deeply aligned with human well-being.

Chang Shi

The lack of large-scale clean data for learning has been a challenge that significantly hinders robots from developing superior level of autonomous intelligence. This urges the necessity to utilize diverse data in a more efficient way. This work approaches the challenge from four perspectives: efficient learning from expert demonstration, efficient dynamics modeling from in-the-wild videos, efficient learning from heuristics guidance, and adjustment for efficient deployment. We provide an overview of preliminary results in each area and outline proposed research on extracting controllable representation from data, aiming at efficient cross-embodiment learning, as well as learning from multi fidelity data.

Connor Mattson

Learning from human feedback enables AI systems and robots to learn policies that align with human intent. While existing work has primarily examined learning from demonstrations, corrections, and preferences in single-agent settings, these ideas have yet to be fully extended to multi-agent domains—where cooperation, decentralization, and non-stationary dynamics demand new methods. In this thesis summary, I highlight my current work and outline future directions for multi-robot learning from human feedback, offering deployment strategies that align supervisor intent with robot teams in the real world.

Rishabh Agrawal

My doctoral research develops a unified framework for offline imitation learning (IL) that tackles three central challenges: achieving sample efficiency in strictly batch settings, ensuring robustness and generalization under dynamics shifts, and learning from demonstrations of varying quality. At the core of this work is a new paradigm for strictly offline IL based on enforcing the Markov Balance Equation (MBE), a fundamental structural property of trajectory data. Using advanced conditional density estimation, I developed two algorithms, CKIL and MBIL, which achieve state-of-the-art performance in high-dimensional continuous-control tasks. Building upon this foundation, I developed the first Distributionally Robust Offline IL framework under a stationarity constraint, enabling robustness to transition-model mismatch without requiring any additional interaction. I am now extending this direction through Robust Behavior Foundation Models (RBFMs), which aim to generalize across dynamics shifts for a wide range of tasks. Finally, I propose a variational approach for learning from crowdsourced demonstrations by inferring and accounting for demonstrator expertise. Together, these contributions yield principled and practical IL algorithms with strong performance and robustness, broadening the applicability of IL to real-world domains such as robotics, healthcare, and autonomous systems.

Oren Salzman, Carlos Hernández Ulloa, Ariel Felner, Sven Koenig

Multi-objective search (MOS) has emerged as a unifying framework for planning and decision-making problems where multiple, often conflicting, criteria must be balanced. While the problem has been studied for decades, recent years have seen renewed interest in the topic across AI applications such as robotics, transportation, and operations research, eflecting the reality that real-world systems rarely optimize a single measure. This paper surveys developments in MOS while highlighting cross-disciplinary opportunities, and outlines open challenges that define the emerging frontier of MOS research.

Scarlett Raine, Tobias Fischer

Marine ecosystems face increasing pressure due to climate change, driving the need for scalable, AI-powered monitoring solutions to inform effective conservation and restoration efforts. This paper examines the rapid emergence of underwater AI as a major research frontier and analyzes the factors that have transformed marine perception from a niche application into a catalyst for AI innovation. We identify three convergent drivers: i) environmental necessity for ecosystem-scale monitoring, ii) democratization of underwater datasets through citizen science platforms, and iii) researcher migration from saturated terrestrial computer vision domains. Our analysis reveals how unique underwater challenges—turbidity, cryptic species detection, expert annotation bottlenecks, and cross-ecosystem generalization—are driving fundamental advances in weakly supervised learning, open-set recognition, and robust perception under degraded conditions. We survey emerging trends in datasets, scene understanding and 3D reconstruction, highlighting the paradigm shift from passive observation toward AI-driven, targeted intervention capabilities. The paper demonstrates how underwater constraints are pushing the boundaries of foundation models, self-supervised learning, and perception, with methodological innovations that extend far beyond marine applications to benefit general computer vision, robotics, and environmental monitoring.

Hendrik Baier, Mark T. Keane, Sarath Sreedharan, Silvia Tulli, Abhinav Verma

In this paper, we highlight the field of explainable sequential decision making. We discuss how the problem of explaining sequential decisions gives rise to problems and challenges that are absent from scenarios that focus on explaining single-shot decision making. We provide a short survey of some of the more prominent subareas within explainable sequential decision-making and their unique focuses and blind spots. Here, we argue that we need to go beyond simply focusing on individual subareas like explainable planning, reinforcement learning, or robotics, and move towards studying and tackling the more general problem of explainable sequential decision-making. Such a holistic approach will not only allow us to identify previously ignored problems, but also provide us with the ability to transfer ideas and intuitions from one subarea of explainable sequential decision-making to another. We end the paper with a discussion on future directions and some of the most pressing open questions.

Jessica Vandenberg, Keisha Bailey, Claire Aguiar, Cecilia Xuning Zhang, Danny Schmidt, Treshonda Rutledge, Bradford Mott, Joseph P. Wilson

Introducing AI concepts in the earliest years of schooling can help children make sense of intelligent technologies, yet few resources exist for K–2 classrooms. This paper presents the design and outcomes of a professional development (PD) program supporting K–2 teachers as they explored AI literacy and co-designed unplugged classroom activities. Grounded in AI4K12's Five Big Ideas in AI framework, the PD combined hands-on learning, collaborative design, and micro-teaching opportunities. Guiding activities included Train the AI (pattern recognition), What Happens Next? (consequences of AI use), Who Did the Robot Hear? (data diversity), and Teach the Robot (model training). Educators then created screen-free, English Language Arts-aligned activities using storytelling, sorting, and embodied play to introduce AI topics such as machine learning and fairness in age-appropriate ways. The PD emphasized integrating AI into existing K–2 literacy routines, lowering implementation barriers while supporting vocabulary development, reasoning, and empathy. Teacher reflections revealed growing confidence in adapting AI topics for young learners and highlighted the value of peer collaboration, clear language, and tactile materials.

Sandra Roach, Karis Boyd-Sinkler, Visrut Sudhakar, Shaundra Daily, Miroslav Pajic, Whitney McCoy

As artificial intelligence (AI) becomes increasingly prevalent in society, there is a critical need for accessible K-12 educational resources that introduce students to AI and robotics concepts through engaging, hands-on experiences. In this paper, we present a scalable workshop framework that uses narrative-driven problem solving to teach fundamental AI and autonomous systems concepts to students in grades 5-12. Developed through a collaboration between AI researchers and education specialists, Bot Blitz employs Sphero RVR+ robots within immersive storylines ranging from fairground rescue missions for younger students to urban traffic management scenarios for high schoolers. Preliminary observations from workshops with 56 students show high engagement levels and successful completion of programming challenges.

Maria Kasinidou, Styliani Kleanthous, Jahna Otterbacher

AI has great potential to transform education and daily life. However, before integrating AI into classrooms, it is crucial to first educate children on what AI is and how to use it responsibly. Effective AI education should build on children's existing perceptions, address misconceptions, and establish a solid foundation for AI literacy. This study explores primary school students’ perception of AI and its relationship to their demographic characteristics and digital skills. A survey was conducted in seven local schools, with 233 students participating. The results indicate that most of them were unfamiliar with AI, and those who attempted to define or depict it often associated it with robots or digital devices. The study also found significant differences in students’ AI perceptions based on factors like gender, grade, and prior digital skills training. These variables were also linked to students’ awareness and understanding of AI. These findings underscore the need for targeted AI educational interventions for primary school students, leveraging their existing perceptions.

Benedikt Schlereth-Groh, Sakir Furkan Yöndem, Ramin Tavakoli Kolagari

Reinforcement Learning (RL) has shown significant promise in developing autonomous navigation algorithms for complex environments. However, the direct application of RL policies trained in simulation to real-world scenarios often faces challenges due to the reality gap. This paper proposes a two-stage system incorporating a segmentation strategy and a bird’s-eye-view (BEV) representation to mitigate the domain gap between simulation and reality. In the first stage, the segmentation transforms sensor data into a simplified and interpretable representation of the surrounding area, facilitating transferability across different deployments. In the second stage, the agent navigates through the BEV map, which can be trained using a vectorized simulation environment---a setup that runs multiple parallel instances of the environment to provide a wide range of training scenarios. This vectorization enables rapid exposure to varied environmental conditions, thereby accelerating and diversifying the training of a deep RL agent to achieve optimal navigation behaviors while maintaining high-speed, in-bound trajectories. The segmentation is crucial because it supports generalization of the learned policy across different robotic platforms. The contribution of this paper lies in combining real-time semantic segmentation with a bird’s-eye-view navigation policy, resulting in a transferable and scalable framework for real-world deployment of RL-based navigation agents. Experimental results demonstrate that agents trained with this methodology exhibit robust navigation performance and adaptability in both simulated and real-world environments, validating the efficacy of combining vectorized simulation with real-world segmentation for practical robotic navigation.

Shaohong Zhong, Alessandro Albini, Perla Maiolino, Ingmar Posner

Recent advances in machine learning have driven a step-change in robot perception with modalities such as vision, where large amounts of training data are readily available or cheap to collect. However, in tactile perception, the relatively high cost of data collection still largely impedes the adoption of such data-driven learning solutions. In this article, we introduce TactGen, a novel, cross-modal framework to tackle this challenge. In particular, using a two-step data generation pipeline, we leverage easily accessible vision data to synthesise artificial tactile data for downstream classifier training. Specifically, we use readily collected video data of objects of interest to efficiently learn neural radiance field (NeRF) representations. The NeRF models are then used to render red–green–blue-depth (RGBD) images from any desired vantage points. In the second stage, the RGBD images are translated into corresponding tactile images typically generated by camera-based tactile sensors using a conditional generative adversarial network (cGAN). The cGAN model is itself trained with a large set of visuo-tactile images collected in simulation, and can be transferred into the real world without fine-tuning or additional data collection. We extensively validate this approach in the context of tactile object classification, showing that it effectively reduces data collection time by a factor of 20 while achieving similar performance to training a classifier on manually collected real data.

Matthias Zeller, Vardeep Singh Sandhu, Benedikt Mersch, Jens Behley, Michael Heidingsfeld, Cyrill Stachniss

The perception of moving objects is crucial for autonomous robots performing collision avoidance in dynamic environments. LiDARs and cameras tremendously enhance scene interpretation but do not provide direct motion information and face limitations under adverse weather. Radar sensors overcome these limitations and provide Doppler velocities, delivering direct information on dynamic objects. In this article, we address the problem of moving instance segmentation in radar point clouds to enhance scene interpretation for safety-critical tasks. Our radar instance transformer enriches the current radar scan with temporal information without passing aggregated scans through a neural network. We propose a full-resolution backbone to prevent information loss in sparse point cloud processing. Our instance transformer head incorporates essential information to enhance segmentation but also enables reliable, class-agnostic instance assignments. In sum, our approach shows superior performance on the new moving instance segmentation benchmarks, including diverse environments, and provides model-agnostic modules to enhance scene interpretation.

Erfan Shahriari, Petr Svarny, Seyed Ali Baradaran Birjandi, Matej Hoffmann, Sami Haddadin

Robots have surpassed humans in terms of strength and precision, yet humans retain an unparalleled ability for decision-making in the face of unpredictable disturbances. This article aims to combine the strengths of both entities within a singular task: human motion guidance under strict geometric constraints, particularly adhering to predetermined paths. To tackle this challenge, a modular haptic guidance law is proposed that takes the human-applied wrench as an input. Using an auxiliary variable called phase, the generated desired motion is guaranteed to consistently adhere to the constraint path. The guidance policy can be generalized into physically interpretable terms, adjustable either prior to initiating the task or dynamically while the task is in progress. An illustrative guidance adaptation policy is showcased that takes into account the human's manipulability. Passivity analysis is used to ensure overall system stability. Experiments, including a 20-participant user study, explore various aspects of the approach in practice.

Sepehr Samavi, James R. Han, Florian Shkurti, Angela P. Schoellig

Robots need to predict and react to human motions to navigate through a crowd without collisions. Many existing methods decouple prediction from planning, which does not account for the interaction between robot and human motions and can lead to the robot getting stuck. We propose SICNav, a Model Predictive Control (MPC) method that jointly solves for robot motion and predicted crowd motion in closed-loop. We model each human in the crowd to be following an Optimal Reciprocal Collision Avoidance (ORCA) scheme and embed that model as a constraint in the robot's local planner, resulting in a bilevel nonlinear MPC optimization problem. We use a KKT-reformulation to cast the bilevel problem as a single level and use a nonlinear solver to optimize. Our MPC method can influence pedestrian motion while explicitly satisfying safety constraints in a single-robot multi-human environment. We analyze the performance of SICNav in two simulation environments and indoor experiments with a real robot to demonstrate safe robot motion that can influence the surrounding humans. We also validate the trajectory forecasting performance of ORCA on a human trajectory dataset.

Sipu Ruan, Weixiao Liu, Xiaoli Wang, Xin Meng, Gregory S. Chirikjian

This paper proposes a learning-from-demonstration (LfD) method using probability densities on the workspaces of robot manipulators. The method, named PRobabilistically-Informed Motion Primitives (PRIMP), learns the probability distribution of the end effector trajectories in the 6D workspace that includes both positions and orientations. It is able to adapt to new situations such as novel via points with uncertainty and a change of viewing frame. The method itself is robot-agnostic, in that the learned distribution can be transferred to another robot with the adaptation to its workspace density. Workspace-STOMP, a new version of the existing STOMP motion planner, is also introduced, which can be used as a post-process to improve the performance of PRIMP and any other reachability-based LfD method. The combination of PRIMP and Workspace-STOMP can further help the robot avoid novel obstacles that are not present during the demonstration process. The proposed methods are evaluated with several sets of benchmark experiments. PRIMP runs more than 5 times faster than existing state-of-the-art methods while generalizing trajectories more than twice as close to both the demonstrations and novel desired poses. They are then combined with our lab’s robot imagination method that learns object affordances, illustrating the applicability to learn tool use through physical experiments.

Akmaral Moldagalieva, Joaquim Ortiz-Haro, Wolfgang Hönig

Kinodynamic motion planning for a multirobot system with different dynamics and actuation limits is a challenging problem. The difficulty increases with the presence of aerodynamic interaction forces that occur when aerial robots fly in close proximity. Due to these complexities, existing planners either rely on simplified assumptions (like ignoring robot dynamics and interaction forces) or produce highly suboptimal solutions. This article presents a kinodynamic motion planner for a heterogeneous team of robots that respects robot dynamics, scales well to 16 robots, and directly reasons about interaction forces between aerial robots operating in close proximity. Our method, db-ECBS, generalizes the multiagent path-finding method Enhanced Conflict-Based Search (ECBS) to the continuous domain by using the single-robot kinodynamic motion planner discontinuity-bounded A. The planner db-ECBS operates on three levels. Initially, individual robot trajectories are computed using a graph search that allows bounded discontinuities between precomputed motion primitives. The second level identifies interrobot collisions or interaction force violations and resolves them by imposing constraints on the first level. The third and final level uses the resulting solution with discontinuities as an initial guess for a joint-space trajectory optimization. The procedure is repeated with a reduced discontinuity bound, resulting in an anytime, probabilistically complete, and asymptotically bounded suboptimal planner. We provide a benchmark of 65 problems with six different dynamics. We demonstrate that db-ECBS produces trajectories that are less than half the cost of existing planners. We show that the interaction-awareness is particularly important for very dense scenarios.

Dejan Milojevic, Gioele Zardini, Miriam Elser, Andrea Censi, Emilio Frazzoli

This article discusses the integration challenges and strategies for designing mobile robots, by focusing on the task-driven, optimal selection of hardware and software to balance safety, efficiency, and minimal usage of resources such as costs, energy, computational requirements, and weight. We emphasize the interplay between perception and motion planning in decision-making by introducing the concept of occupancy queries to quantify the perception requirements for sampling-based motion planners. Sensor and algorithm performance are evaluated using false negative rate and false positive rate across various factors such as geometric relationships, object properties, sensor resolution, and environmental conditions. By integrating perception requirements with perception performance, an integer linear programming approach is proposed for efficient sensor and algorithm selection and placement. This forms the basis for a co-design optimization that includes the robot body, motion planner, perception pipeline, and computing unit. We refer to this framework for solving the co-design problem of mobile robots as CODEI, short for co-design of embodied intelligence. A case study on developing an autonomous vehicle for urban scenarios provides actionable information for designers, and shows that complex tasks escalate resource demands, with task performance affecting choices of the autonomy stack. The study demonstrates that resource prioritization influences sensor choice: cameras are preferred for cost-effective and lightweight designs, while lidar sensors are chosen for better energy and computational efficiency.

Robert McCarthy, Daniel C.H. Tan, Dominik Schmidt, Fernando Acero, Nathan Herr, Yilun Du, Thomas G. Thuruthel, Zhibin Li

Scaling deep learning to massive and diverse internet data has driven remarkable breakthroughs in domains such as video generation and natural language processing. Robot learning, however, has thus far failed to replicate this success and remains constrained by a scarcity of available data. Learning from Videos (LfV) methods aim to address this data bottleneck by augmenting traditional robot data with large-scale internet video. This video data provides foundational information regarding physical dynamics, behaviours, and tasks, and can be highly informative for general-purpose robots. This survey systematically examines the emerging field of LfV. We first outline essential concepts, including detailing fundamental LfV challenges such as distribution shift and missing action labels in video data. Next, we comprehensively review current methods for extracting knowledge from large-scale internet video, overcoming LfV challenges, and improving robot learning through video-informed training. The survey concludes with a critical discussion of future opportunities. Here, we emphasize the need for scalable foundation model approaches that can leverage the full range of available internet video and enhance the learning of robot policies and dynamics models. Overall, the survey aims to inform and catalyse future LfV research, driving progress towards general-purpose robots.