Autonomous driving must handle motion blur, low light, and fast-changing scenes, where RGB frames and event cameras provide complementary strengths. This thesis explores how to fuse them across the perception–reasoning–planning pipeline. It introduces FlexEvent, a frequency-robust detector with adaptive fusion and label-efficient training; Talk2Event, the first benchmark for event–language grounding with attribute-aware modeling; and the EventDrive, an event–frame VLM covering the full driving loop. Together, these contributions advance robust perception, interpretable reasoning, and reliable planning for safety-critical driving through event–frame fusion.
论文检索
输入标题、作者或关键词,从 3,752 篇学术成果中精准定位
Autonomous systems operating in uncertain environments without human intervention must consider several factors, including safety, reliability, and task success. State-of-the-art methods have made progress in addressing these factors individually, but often fail to unify them for deployment in real-world systems. My dissertation aims to combine methods in planning under uncertainty, failure recovery, and explainability, providing a holistic framework for comprehensive safe autonomy in real-world deployment.
Scaling Up Cooperative Multi-Agent Reinforcement Learning Through Hierarchical Heterogeneous Modular Architectures
PDF ↗Multi-agent reinforcement learning enables sophisticated collaborative behaviors in autonomous systems, yet fundamental scalability barriers persist: existing methods struggle to coordinate large agent populations and face challenges with extended decision-making horizons. This research develops hierarchical approaches to scale up multi-agent learning systems through two complementary directions: structural scaling for coordinating increasing numbers of agents and temporal scaling for extending decision-making horizons. This paper presents four integrated contributions: a taxonomic survey establishing hierarchical architectures as the theoretical foundation for scalable multi-agent learning systems, a benchmark for long-horizon multi-objective multi-agent reinforcement learning, a framework integrating self-organizing neural networks with multiple reinforcement learning agents for hierarchical tri-level control, and a framework leveraging large language models for zero-shot multi-agent planning. Through comprehensive validation, this work demonstrates that hierarchical, heterogeneous, modular architectures provide unified, interpretable solutions to multi-agent scalability, bridging theoretical multi-agent reinforcement learning research with real-world deployment requirements.
Autonomous driving has shown significant progress in recent years. The combination of advanced sensors, ample data, and machine learning algorithms has led to the deployment of autonomous vehicles (AVs) in cities like Los Angeles, San Francisco, and Phoenix. While not all humans can drive perfectly, AVs should be able to plan, adapt, and react to environmental disturbances, including irrational human drivers. My research focuses on applying reinforcement learning (RL) techniques to validate AV-related cyber-physical systems (CPS) in realistic environments. I develop a custom RL environment that simulates highway driving scenarios with multiple vehicles. This environment includes a CPS model of adaptive cruise control (ACC), a lane-changing model (MOBIL), and an adversarial agent that learns to drive irrationally. My work extends interpretable RL techniques to continuous control tasks like autonomous driving.
Multi-objective search (MOS) has emerged as a unifying framework for planning and decision-making problems where multiple, often conflicting, criteria must be balanced. While the problem has been studied for decades, recent years have seen renewed interest in the topic across AI applications such as robotics, transportation, and operations research, eflecting the reality that real-world systems rarely optimize a single measure. This paper surveys developments in MOS while highlighting cross-disciplinary opportunities, and outlines open challenges that define the emerging frontier of MOS research.
Autoformalization has emerged as a term referring to the automation of formalization in the context of the formalization of mathematics using interactive theorem provers (proof assistants). Its rapid development has been driven by progress in deep learning, especially large language models (LLMs). More recently, usage of the term has expanded beyond mathematics to describe tasks that involve translating natural language input into verifiable logical representations. At the same time, a growing body of research explores using LLMs to translate informal language into formal representations for reasoning, planning, and knowledge representation, but without explicitly referring to this process as autoformalization. As a result, despite addressing similar tasks, the largely independent development of these research areas has limited opportunities for shared methodologies, benchmarks, and theoretical frameworks that could accelerate progress. Our goal is to review - explicit or implicit - instances of what can be considered autoformalization and to propose a unified framework, encouraging cross-pollination between different fields to advance the development of next generation AI systems.
In this paper, we highlight the field of explainable sequential decision making. We discuss how the problem of explaining sequential decisions gives rise to problems and challenges that are absent from scenarios that focus on explaining single-shot decision making. We provide a short survey of some of the more prominent subareas within explainable sequential decision-making and their unique focuses and blind spots. Here, we argue that we need to go beyond simply focusing on individual subareas like explainable planning, reinforcement learning, or robotics, and move towards studying and tackling the more general problem of explainable sequential decision-making. Such a holistic approach will not only allow us to identify previously ignored problems, but also provide us with the ability to transfer ideas and intuitions from one subarea of explainable sequential decision-making to another. We end the paper with a discussion on future directions and some of the most pressing open questions.
SlideBot: A Multi-Agent Framework for Generating Informative, Reliable, Multi-Modal Presentations
PDF ↗Large Language Models (LLMs) have shown immense potential in education, automating tasks like quiz generation and content summarization. However, generating effective presentation slides introduces unique challenges due to the complexity of multimodal content creation and the need for precise, domain-specific information. Existing LLM-based solutions often fail to produce reliable and informative outputs, limiting their educational value. To address these limitations, we introduce SlideBot - a modular, multi-agent slide generation framework that integrates LLMs with retrieval, structured planning, and code generation. SlideBot is organized around three pillars: informativeness, ensuring deep and contextually grounded content; reliability, achieved by incorporating external sources through retrieval; and practicality, which enables customization and iterative feedback through instructor collaboration. It incorporates evidence-based instructional design principles from Cognitive Load Theory (CLT) and the Cognitive Theory of Multimedia Learning (CTML), using structured planning to manage intrinsic load and consistent visual macros to reduce extraneous load and enhance dual-channel learning. Within the system, specialized agents collaboratively retrieve information, summarize content, generate figures, and format slides using LaTeX, aligning outputs with instructor preferences through interactive refinement. Evaluations from domain experts and students in AI and biomedical education show that SlideBot consistently enhances conceptual accuracy, clarity, and instructional value. These findings demonstrate SlideBot’s potential to streamline slide preparation while ensuring accuracy, relevance, and adaptability in higher education.
SAGE: A Compositional Multi-Agent LLM Framework with Pedagogical Reasoning for Structured Collaborative Problem Solving
PDF ↗While AI can simulate virtual classrooms, effective collaborative learning requires both dynamic interaction and a well-structured pedagogical plan. To address this, we introduce SAGE (Scaffolded Agent-Guided Education), a novel, compositional two-phase framework. First, a planning module automatically generates an optimized pedagogical scenario using a dedicated team of agents. Second, this scenario is used to configure a conversation module, where autonomous agents engage a student in a structured, real-time dialogue. This approach ensures that dynamic, multi-agent interactions are grounded in a pedagogically sound foundation. We evaluate SAGE through simulation and a study with real students. Results show improved performance against a next-speaker prediction baseline (achieving a 72.13% win rate) and demonstrate effective group dynamics. Specifically, our study with students reveals high role adherence from AI agents, a balanced progression between task-oriented and socio-emotional interactions, and a clear scaffolding effect where instructional support fades as learner autonomy increases. Our findings highlight the significant potential of synergizing automated instructional design with autonomous conversational execution for collaborative learning.
Catching the First Light of Tomorrow: A Hackathon-Based Framework for Introducing High School Students to AI Agents
PDF ↗Artificial Intelligence (AI), particularly in the form of intelligent AI agents, is transforming education, industry, and everyday life. These agents extend the capabilities of Large Language Models (LLMs) by integrating planning, decision-making, tool use, and multi-agent collaboration, enabling systems that can reason, adapt, and act in dynamic environments. As such systems become integral to modern workplaces and everyday problem solving, early exposure equips high school students with systems thinking, practical problem-solving skills, and ethical awareness, while preparing them to create applications that address real-world needs. Yet most high school AI programs focus on basic model usage and overlook the skills required to design and deploy agentic systems. Existing resources are largely aimed at university learners and assume substantial programming expertise, creating a significant accessibility gap. To address this need, we present a structured hackathon-based framework for introducing high school students to the design and application of AI agents. The framework combines expert-led lectures on core topics such as agent architectures, prompting strategies, reasoning methods, and tool-use protocols with a guided hackathon in which students collaboratively develop domain-specific agent-based chatbots. We provide a complete suite of instructional materials, step-by-step tutorials, and starter code to support hands-on learning, enabling participants to build functional agents capable of reasoning and interacting with external tools. Our approach bridges the gap between AI literacy and practical deployment while fostering creativity, collaboration, and responsible innovation, and our findings suggest that early engagement with agent-based AI design equips students with both technical proficiency and the mindset to shape the AI-driven future.
Generative AI has moved from pilots to everyday practice, delivering gains in productivity and accessibility while surfacing present-day risks—hallucinations and reliability failures, bias and unfairness, prompt-injection attacks, and so on. These trends make AI safety education a core competency. In this paper, we survey global AI safety curricula and, in the Japanese context, observe strong policy momentum but relatively few courses that explicitly combine capability instruction with systematic safety evaluation. In response, we developed a 7-week, graduate-level intensive at a private science and engineering university in Japan, with enrollment open to international exchange students at both the undergraduate and graduate levels. The curriculum progresses from machine-learning foundations to generative models and alignment, with introductory agent topics included to support risk reasoning. Delivery combines weekly lectures, invited talks from academia and industry, structured group discussions, and a final presentation plus a paper-style final project focused on risk evaluation and mitigation planning. An end-of-course survey indicates high perceived learning and positive experience and one student project later resulted in a peer-reviewed workshop paper at ICLR 2025.
GAICo: A Deployed and Extensible Framework for Evaluating Diverse and Multimodal Generative AI Outputs
PDF ↗The rapid proliferation of Generative AI (GenAI) into diverse, high-stakes domains necessitates robust and reproducible evaluation methods. However, practitioners often resort to ad-hoc, non-standardized scripts, as common metrics are often unsuitable for specialized, structured outputs (e.g., automated plans, time-series) or holistic comparison across modalities (e.g., text, audio, and image). This fragmentation hinders comparability and slows AI system development. To address this challenge, we present GAICo (Generative AI Comparator): a deployed, open-source Python library that streamlines and standardizes GenAI output comparison. GAICo provides a unified, extensible framework supporting a comprehensive suite of reference-based metrics for unstructured text, specialized structured data formats, and multimedia (images, audio). Its architecture features a high-level API for rapid, end-to-end analysis, from multi-model comparison to visualization and reporting, alongside direct metric access for granular control. We demonstrate GAICo's utility through a detailed case study evaluating and debugging complex, multi-modal AI Travel Assistant pipelines. GAICo empowers AI researchers and developers to efficiently assess system performance, make evaluation reproducible, improve development velocity, and ultimately build more trustworthy AI systems, aligning with the goal of moving faster and safer in AI deployment. Since its release on PyPI in Jun 2025, the tool has been downloaded over 16K times, across versions, by Dec 2025, demonstrating growing community interest.
A Unified Geospatial Clustering Framework to Identify Varying Density Clusters in E-Commerce Logistics
PDF ↗In e-commerce logistics, accurate geospatial clustering is essential for optimizing resource allocation, manpower planning, and delivery network design. However, existing density-based clustering approaches, particularly their reliance on heuristic parameter tuning, have been underexplored in datasets with significant density variations, limiting robustness and scalability. This study presents an unsupervised framework that extends DBSCAN by leveraging Gaussian Mixture Models (GMM). First, we propose a method that systematically identifies suitable clustering scales through statistical modeling. Second, the approach iteratively applies DBSCAN to extract clusters from dense to sparse regions, overcoming single-parameter limitations. Finally, we validate the method through large-scale offline experiments using data from over 200 last-mile dispatch centers (LMDC). The results demonstrate the framework’s effectiveness in identifying heterogeneous geographic demand patterns and supporting workforce planning and operational benchmarking. This framework provides a scalable solution to a critical challenge in e-commerce logistics, offering a valuable reference for strategic and operational decision-making.
In daily fantasy sports (DFS), match participation is highly time-sensitive. Users must act within a narrow window before a game begins, making match recommendation a time-critical task to prevent missed engagement and revenue loss. Existing recommender systems, typically designed for static item catalogs, are ill-equipped to handle the hard temporal deadlines inherent in these live events. To address this, we designed and deployed a recommendation engine using the Deep Interest Network (DIN) architecture. We adapt the DIN architecture by injecting temporality at two levels: first, through real-time urgency features for each candidate match (e.g., time-to-round-lock), and second, via temporal positional encodings that represent the time-gap between each historical interaction and the current recommendation request, allowing the model to dynamically weigh the recency of past actions. This approach, combined with a listwise neuralNDCG loss function, produces highly relevant and urgency-aware rankings. To support this at industrial scale, we developed a multi-node, multi-GPU training architecture on Ray and PyTorch. Our system, validated on a massive industrial dataset with over 650k users and over 100B interactions, achieves a +9% lift in nDCG@1 over a heavily optimized LightGBM baseline with handcrafted features. The strong offline performance of this model establishes its viability as a core component for our planned on-device (edge) recommendation system, where online A/B testing will be conducted.
Behavioral health conditions, which include mental health and substance use disorders, are the leading disease burden in the United States. Peer-run behavioral health organizations (PROs) critically assist individuals facing these conditions by combining mental health services with assistance for needs such as income, employment, and housing. However, limited funds and staffing make it difficult for PROs to address all service user needs. To assist peer providers at PROs with their day-to-day tasks, we introduce PeerCoPilot, a large language model (LLM)-powered assistant that helps peer providers create wellness plans, construct step-by-step goals, and locate organizational resources to support these goals. PeerCoPilot ensures information reliability through a retrieval-augmented generation pipeline backed by a large database of over 1,300 vetted resources. We conducted human evaluations with 15 peer providers and 6 service users and found that over 90% of users supported using PeerCoPilot. Moreover, we demonstrate that PeerCoPilot provides more reliable and specific information than a baseline LLM. PeerCoPilot is now used by a group of 5-10 peer providers at CSPNJ, a large behavioral health organization serving over 10,000 service users, and we are actively expanding PeerCoPilot's use.
Numerous social movements (SMs) around the world help support the UN's Sustainable Development Goals (SDGs). Understanding how key events shape SMs is key to the achievement of the SDGs. We have developed SMART (Social Media Analysis & Reasoning Tool) to track social movements related to the SDGs. SMART was designed by a multidisciplinary team of AI researchers, journalists, communications scholars and legal experts. This paper describes SMART's transformer-based multivariate time series Discourse Evolution Engine for Predictions about Social Movements (DEEP) to predict the volume of future articles/posts and the emotions expressed. DEEP outputs probabilistic forecasts with uncertainty estimates, providing critical support for editorial planning and strategic decision-making. We evaluate DEEP with a case study of the #MeToo movement by creating a novel longitudinal dataset (433K Reddit posts and 121K news articles) from September 2024 to June 2025, which is publicly available for research purposes.
Deployed AI Agents for Industrial Asset Management: CodeReAct Framework for Event Analysis and Work Order Automation
PDF ↗Maintenance of mission-critical industrial assets is frequently hindered by fragmented data, inconsistent record-keeping, and limited access to analytical expertise, resulting in reactive rather than predictive practices. We present \textit{CodeReAct}, an AI-powered agentic framework deployed in large-scale facilities to automate event analysis and work order (WO) management.CodeReAct extends the ReAct paradigm by embedding executable Python code within the Thought--Action--Observation (TAO) loop, enabling natural language interaction, grounding heterogeneous alerts and work orders into structured Business Objects (BOs), and dynamically invoking analytic functions for forecasting, anomaly correlation, and maintenance recommendations. This architecture reduces manual data science intervention, improves adaptability, and supports reuse across asset types. Deployed in a mission-critical data center and productionized in Maximo, CodeReAct manages pumps, chillers, AHUs, compressors, cooling towers, and other mechanical and electrical systems. Evaluation with 36 representative maintenance utterances showed that outer-loop reflection and adaptive temperature improved task completion by up to 20%, while ablation studies confirmed the importance of reasoning in addition to code execution. Business validation revealed seasonal failure patterns, bundling opportunities, and predictive accuracy trends. In production, site engineers reported 25--40% faster diagnostics, fewer unplanned downtime events, and reduced reliance on specialized analysts. Lessons learned highlight the importance of structured BOs for grounding analytics, runtime safeguards to mitigate hallucinations, and adaptive model control for consistent execution. These results demonstrate how deployed agentic AI can deliver measurable business value in predictive and strategic maintenance planning.
LLM-based autonomous agents have recently shown strong capabilities in solving complex industrial design tasks. However, in domains aiming for carbon neutrality and high-performance renewable energy systems, current AI-assisted design automation methods face critical challenges in explainability, scalability, and practical usability. To address these limitations, we introduce PHIA (Physics-Informed Autonomous Agent), an LLM-driven system that automates modulation design for power converters in Power Electronics Systems with minimal human intervention. In contrast to traditional pipeline-based methods, PHIA incorporates an LLM-based planning module that interactively acquires and verifies design requirements via a user-friendly chat interface. This planner collaborates with physics-informed simulation and optimization components to autonomously generate and iteratively refine modulation designs. The interactive interface also supports interpretability by providing textual explanations and visual outputs throughout the design process. Experimental results show that PHIA reduces standard mean absolute error by 63.2% compared to the second-best benchmark and accelerates the overall design process by over 33 times. A user study involving 20 domain experts further confirms PHIA’s superior design efficiency and usability, highlighting its potential to transform industrial design workflows in power electronics.
Talking Trails: LLM-Enhanced Spatiotemporal Trajectory Modeling for E-Bike Delivery Route Planning
PDF ↗Electric bicycles (e-bikes) have become the dominant mode of transportation in China’s urban instant delivery industry. However, many riders lack the experience to navigate complex traffic networks and diverse road conditions, leading to reduced delivery efficiency. To address this issue, we present Talking Trails, an e-bike delivery route planning system built upon an LLM-enhanced spatiotemporal trajectory model. Trained on millions of real-world delivery trajectories, fused with spatiotemporal and semantic data information, the model achieves a top-5 rider displacement prediction accuracy of 95% and a route optimization rate of 82.1%. In practice, we augment the core planner with an LLM-driven semantic layer that translates high-level user intent into executable tasks, then pair it with a battery-swap module that continuously validates route feasibility so the vehicle never runs out of charge mid-mission. Currently serving tens of thousands of riders, the system is projected to reduce average delivery mileage by 17% and lower annual carbon emissions by 3978 tons. Overall, Talking Trails significantly improves delivery efficiency, offering a scalable and sustainable solution for instant delivery operations.
SICNav: Safe and Interactive Crowd Navigation Using Model Predictive Control and Bilevel Optimization (Abstract Reprint)
PDF ↗Robots need to predict and react to human motions to navigate through a crowd without collisions. Many existing methods decouple prediction from planning, which does not account for the interaction between robot and human motions and can lead to the robot getting stuck. We propose SICNav, a Model Predictive Control (MPC) method that jointly solves for robot motion and predicted crowd motion in closed-loop. We model each human in the crowd to be following an Optimal Reciprocal Collision Avoidance (ORCA) scheme and embed that model as a constraint in the robot's local planner, resulting in a bilevel nonlinear MPC optimization problem. We use a KKT-reformulation to cast the bilevel problem as a single level and use a nonlinear solver to optimize. Our MPC method can influence pedestrian motion while explicitly satisfying safety constraints in a single-robot multi-human environment. We analyze the performance of SICNav in two simulation environments and indoor experiments with a real robot to demonstrate safe robot motion that can influence the surrounding humans. We also validate the trajectory forecasting performance of ORCA on a human trajectory dataset.