Enhancing diverse human decision-making processes in an urban environment is a critical issue across various applications, including ride-sharing vehicle dispatching, public transportation management, and autonomous driving. Offline reinforcement learning (RL) is a promising approach to learn and optimize human urban strategies (or policies) from pre-collected human-generated spatial-temporal urban data. However, standard offline RL faces two significant challenges: (1) data scarcity and data heterogeneity, and (2) distributional shift. In this paper, we introduce MODA - a Multi-Task Offline Reinforcement Learning with Contrastive Data Sharing approach. MODA addresses the challenges of data scarcity and heterogeneity in a multi-task urban setting through Contrastive Data Sharing among tasks. This technique involves extracting latent representations of human behaviors by contrasting positive and negative data pairs. It then shares data presenting similar representations with the target task, facilitating data augmentation for each task. Moreover, MODA develops a novel model-based multi-task offline RL algorithm. This algorithm constructs a robust Markov Decision Process (MDP) by integrating a dynamics model with a Generative Adversarial Network (GAN). Once the robust MDP is established, any online RL or planning algorithm can be applied. Extensive experiments conducted in a real-world multi-task urban setting validate the effectiveness of MODA. The results demonstrate that MODA exhibits significant improvements compared to state-of-the-art baselines, showcasing its capability in advancing urban decision-making processes. We also made our code available to the research community.
论文检索
输入标题、作者或关键词，从 3,752 篇学术成果中精准定位
Understanding urban regional characteristics is pivotal in driving critical insights for urban planning and management. We have witnessed the successful application of pre-trained Foundation Models (FMs) in generating universal representations for various downstream tasks. However, applying this principle to the geospatial domain remains challenging, primarily due to the difficulty of gathering extensive data for developing a dedicated urban foundation model. Though there have been some attempts to empower the existing FMs with urban data, most of them focus on single-modality FMs without considering the multi-modality nature of urban region understanding tasks. To address this gap, we introduce ReFound - a novel framework for Re-training a Foundation model for urban region understanding, harnessing the strengths of both language and visual FMs. In this framework, we first invent a Mixture-of-Geospatial-Expert (MoGE) Transformer, to effectively integrate the embedding of multi-source geospatial data. Building on this, ReFound is enhanced by jointly distilling knowledge from language, visual, and visual-language FMs respectively, thus augmenting its generalization capabilities. Meanwhile, we design a masked geospatial data modeling approach alongside a cross-modal spatial alignment mechanism, to enhance the spatial knowledge of ReFound derived from geospatial data. Extensive experiments conducted on six real-world datasets over three urban region understanding tasks demonstrate the superior performance of our framework.
Crime risk map plays a crucial role in urban planning and public security management. Traditionally, it is obtained solely from historical crime incidents or inferred from limited environmental factors, which are not sufficient to accurately model the occurrences of crimes over the geographical space well. Motivated by the impressive and realistic conditional generating power of diffusion models, in this paper, we propose a multimodal conditional diffusion method, namely, DiffCrime, to infer the crime risk map based on datasets in various domains, i.e., historical crime incidents, satellite imagery, and map imagery. It is equipped with a history-gated multimodal denoising network, i.e., HamNet, dedicated to the crime risk map inference. HamNet emphasizes the importance of historical crime data via a Gated-based History Fusion (GHF) module and adaptively controls multimodal conditions to be fused across different diffusion time steps via a Time step-Aware Modality Fusion (TAMF) module. Extensive experiments on two real-world datasets demonstrate the effectiveness of DiffCrime, which outperforms baselines by at least 43% and 31% in terms of RMSE, respectively.
Next location prediction is a crucial task in human mobility modeling, and is pivotal for many downstream applications like location-based recommendation and transportation planning. Although there has been a large body of research tackling this problem, the usefulness of user preference and temporal regularity remains underrepresented. Specifically, previous studies usually neglect the explicit user preference information entailed from human trajectories and fall short in utilizing the arrival time of next location, as a key determinant on next location. To address these limitations, we propose a Multi-Context aware Location Prediction model (MCLP) to predict next locations for individuals, where it explicitly models user preference and the next arrival time as context. First, we utilize a topic model to extract user preferences for different types of locations from historical human trajectories. Second, we develop an arrival time estimator to construct a robust arrival time embedding based on the multi-head attention mechanism. The two components provide pivotal contextual information for the subsequent prediction. Finally, we utilize the Transformer architecture to mine sequential patterns and integrate multiple contextual information to predict the next locations. Experimental results on two real-world mobility datasets show that our proposed MCLP outperforms baseline methods.
Recently, Large Language Model (LLM)-empowered recommender systems (RecSys) have brought significant advances in personalized user experience and have attracted considerable attention. Despite the impressive progress, the research question regarding the safety vulnerability of LLM-empowered RecSys still remains largely under-investigated. Given the security and privacy concerns, it is more practical to focus on attacking the black-box RecSys, where attackers can only observe the system's inputs and outputs. However, traditional attack approaches employing reinforcement learning (RL) agents are not effective for attacking LLM-empowered RecSys due to the limited capabilities in processing complex textual inputs, planning, and reasoning. On the other hand, LLMs provide unprecedented opportunities to serve as attack agents to attack RecSys because of their impressive capability in simulating human-like decision-making processes. Therefore, in this paper, we propose a novel attack framework called CheatAgent by harnessing the human-like capabilities of LLMs, where an LLM-based agent is developed to attack LLM-Empowered RecSys. Specifically, our method first identifies the insertion position for maximum impact with minimal input modification. After that, the LLM agent is designed to generate adversarial perturbations to insert at target positions. To further improve the quality of generated perturbations, we utilize the prompt tuning technique to improve attacking strategies via feedback from the victim RecSys iteratively. Extensive experiments across three real-world datasets demonstrate the effectiveness of our proposed attacking method.
Car-hailing services play an important role in the modern transportation system, and the utilities of the service providers highly depend on the efficiency of route planning algorithms. A widely adopted route planning framework is to assign weights to roads and compute the routes with the shortest path algorithms. Existing techniques of weight-assigning often focus on the traveling time and length of the roads, but cannot incorporate with the preferences of the passengers (users). In this paper, a set of preference weight estimation models is employed to capture the users' preferences over paths in car-hailing with their historical choices. Since the user preferences may vary dynamically over time, it is a challenging task to make real-time decisions over the models. The main technical contribution of this paper is to propose an online learning-based preference weight chasing (PWC) algorithm to solve this problem. The worst-case performance of PWC is analyzed with the metric regret, and it is proved that PWC has a vanishing regret, which means that the time-averaged loss concerning the fixed in-hindsight best model tends to zero. Experiments based on real-world datasets are conducted to verify the effectiveness and efficiency of our algorithm. The code is available at https://github.com/GaoYucen/PWC.
Reinforcement learning, optimization, and planning/search are interconnected domains in artificial intelligence. Algorithms within these domains share many similarities. They complement each other in solving complex decision-making problems, and also offer opportunities for cross-disciplinary integration. However, conducting research on algorithms across these domains typically requires learning the specialized libraries. These libraries often couple algorithms with domain-specific problem classes, making it difficult to conduct cross-disciplinary researches. In order to solve this problem, we developed a generic and lightweight framework for reinforcement learning, optimization, and planning/search algorithms (RLOP). It implements only the core logic of algorithms, abstracting away domain-specific details by defining interface functions, which enables flexible customization and efficient integration across different domains. The framework has been open-sourced at https://github.com/songzhg/RLOP.
Reinforcement Learning for Athletic Intelligence: Lessons from the 1st “AI Olympics with RealAIGym” Competition
PDF ↗As artificial intelligence gains new capabilities, it becomes important to evaluate it on real-world tasks. In particular, the fields of robotics and reinforcement learning (RL) are lacking in standardized benchmarking tasks on real hardware. To facilitate reproducibility and stimulate algorithmic advancements, we held an AI Olympics competition at IJCAI 2023 conference based on the double pendulum system in the RealAIGym project where the participants were asked to develop a controller for the swing up and stabilization task. This paper presents the methods and results from the top participating teams and provides insights into the real-world performance of RL algorithms with respect to a baseline time-varying LQR controller.
A carbon market is a market-based tool that incentivizes economic agents to align individual profits with the global utility, i.e., reducing carbon emissions to tackle climate change. Cap and trade stands as a critical principle based on allocating and trading carbon allowances (carbon emission credit), enabling economic agents to follow planned emissions and penalizing excess emissions. A central authority is responsible for introducing and allocating those allowances in cap and trade. However, the complexity of carbon market dynamics makes accurate simulation intractable, which in turn hinders the design of effective allocation strategies. To address this, we propose an adaptive mechanism design framework, simulating the market using hierarchical, model-free multi-agent reinforcement learning (MARL). Government agents allocate carbon credits, while enterprises engage in economic activities and carbon trading. This framework illustrates agents’ behavior comprehensively. Numerical results show MARL enables government agents to balance productivity, equality, and carbon emissions. Our project is available at https: //anonymous.4open.science/r/Carbon-Simulator.
Between 2021 and 2023, AI-Olympics---a series of online AI competitions, was hosted by the online evaluation platform Jidi in collaboration with the IJCAI committee. In these competitions, an agent is required to accomplish diverse sports tasks in a two-dimensional continuous world, while competing against an opponent. This paper provides a brief overview of the competition series and highlights notable findings. We aim to contribute insights to the field of multi-agent decision-making and explore the generalization of agents through engineering efforts.
Dealing with the ever-increasing demand for traffic management is one of the main challenges of the 21st century. The issue is much more apparent in urban areas during rush hours. Traffic congestion causes economic losses due to delays and increased fuel consumption and, on top of that, is a major health risk. Intelligent centralized traffic routing is an important concept aiming at reducing traffic congestion in urban areas by more effectively utilizing road networks. In this demo, we present a framework that, in a nutshell, integrates techniques for intelligent centralized traffic routing into the well-known SUMO simulator, so these techniques can be evaluated in realistic settings on real/realistic datasets. In particular, the framework automatically identifies ``problematic'' urban regions by analyzing historical traffic data, then simplifies the road networks by precomputing promising routes (for each considered traffic flow), and finally, leverages a planning-based approach to generate routes. Our framework is evaluated on a real dataset from Dublin's metropolitan area.
New rideshare and shared mobility services have transformed urban mobility in recent years. Such services have the potential to improve efficiency and reduce costs by allowing users to share rides in high-capacity vehicles and vans. Most transit agencies already operate various ridepooling services, including microtransit and paratransit. However, the objectives and constraints for implementing these services vary greatly between agencies and can be challenging. First, off-the-shelf ridepooling formulations must be adapted for real-world conditions and constraints. Second, the lack of modular and reusable software makes it hard to implement and evaluate new ridepooling algorithms and approaches in real-world settings. We demonstrate a modular on-demand public transportation scheduling software for microtransit and paratransit services. The software is aimed at transit agencies looking to incorporate state-of-the-art rideshare and ridepooling algorithms in their everyday operations. We provide management software for dispatchers and mobile applications for drivers and users and conclude with results from the demonstration in Chattanooga, TN.
AUTODRAITEC: An Infrastructure-Based AUTOnomous DRiving System Using Artificial Intelligence and TEleCommunication Technologies
PDF ↗This paper introduces AUTODRAITEC, a novel AI-based system that is deployed on the road infrastructure to control the driving of Connected and Autonomous Vehicles (CAVs). For this purpose, we present a convincing proof of concept that demonstrates the effectiveness of our solution. The system deploys a hybrid machine learning approach comprised of a supervised learning classifier to characterize the behaviors of human drivers, with a deep reinforcement learning policy to provide speed recommendations for CAVs. This system is implemented using perception sensors and an industrial computer (IPC), which are intended to be deployed on the road infrastructure. Using a 1:18 scale testbed that faithfully replicates real-world driving scenarios, we demonstrate that AUTODRAITEC improves driving safety and efficiency while preserving the traffic flow rate.
Manufacturing companies are experiencing a transformative journey, moving from labor-intensive processes to integrating cutting-edge technologies such as digitalization and AI. In this demo paper, we present a novel AI tool to enhance manufacturing processes. Remarkably, our work has been developed in collaboration with Agrati S.p.A., a worldwide leading company in the bolts manufacturing sector. In particular, we propose an AI-powered tool to address the problem of automatically generating the production cycle of a bolt. Currently, this decision-making task is performed by process engineers who spend several days to study, draw, and test multiple alternatives before finding the desired production cycle. We cast this task as a model-based planning problem, mapping bolt technical drawings and metal deformations to, potentially continuous, states and actions, respectively. Furthermore, we resort to computer vision tools and visual transformers to design efficient heuristics that make the search affordable in concrete applications. Agrati S.p.A.'s process engineers extensively validated our tool, and they are currently using it to support their work. To the best of our knowledge, ours is the first AI tool dealing with production cycle design in bolt manufacturing.
Integrating LLM, VLM, and Text-to-Image Models for Enhanced Information Graphics: A Methodology for Accurate and Visually Engaging Visualizations
PDF ↗This study presents an innovative approach to the creation of information graphics, where the accuracy of content and aesthetic appeal are of paramount importance. Traditional methods often struggle to balance these two aspects, particularly in complex visualizations like phylogenetic trees. Our methodology integrates the strengths of Large Language Models (LLMs), Vision Language Models (VLMs), and advanced text-to-image models to address this challenge. Initially, an LLM plans the layout and structure, employing Mermaid—a JavaScript-based tool that uses Markdown-like scripts for diagramming—to establish a precise and structured foundation. This structured script is crucial for ensuring data accuracy in the graphical representation. Following this, text-to-image models are employed to enhance the vector graphic generated by Mermaid, adding rich visual elements and enhancing overall aesthetic appeal. The integration of text-to-image models is a key innovation, enabling the creation of graphics that are not only informative but also visually captivating. Finally, a VLM performs quality control, ensuring that the visual enhancements align with the informational accuracy. This comprehensive approach effectively combines the accuracy of structured data representation, the creative potential of text-to-image models, and the validation capabilities of VLMs. The result is a new standard in information graphic creation, suitable for diverse applications ranging from education to scientific communication, where both information integrity and visual engagement are essential.
Across disciplines, cooperation is a fundamental research topic. While socially desirable to a population, it often bears a cost to the individual who, in their own self-interest, rationally chooses not to engage in costly cooperation. As such, much work has been done in understanding the biological mechanisms behind cooperation in human and animal populations. In my PhD project, I develop and apply these mechanisms both to artificial multi-agent systems and real social systems. I examine how factors such as agent heterogeneity and different learning algorithms affect not only the level of cooperation within a system, but also the level of fairness in the distribution of payoffs. In previous work, I showed how the effectiveness of the social norm-based mechanism of indirect reciprocity is affected when in-group biased cooperation is present. Beyond my future work on online platforms, I also plan to explore the effects of space, gossip, and partial and subjective observations to widen the potential scope of applications.
Sequential decision-making under uncertainty in multi-agent environments is a fundamental problem in artificial intelligence. Games serve as a base model for these problems. Finding optimal plans in games that model real-world scenarios necessitates scalable algorithms. In games with perfect information, algorithms that use a combination of search and deep reinforcement learning can scale to arbitrary-sized games and achieve superhuman performance. In games with imperfect information, the situation is more challenging due to the nature of the search. This work aims to develop algorithms that use search but can scale into larger games than currently possible.
Smart grid system encompasses large power plants in the wholesale market and retail customers in the tariff market. An electricity broker liaises between the wholesale and tariff markets by procuring electricity from the power plants and selling it to subscribed customers. In our work, we address the prominent challenges in the smart grid system to achieve better efficiency. We discuss the wholesale market, for which we design efficient bidding strategies in periodic double auctions (PDAs), and the tariff market, which includes tariff contract generation strategies and peak demand mitigation strategies. We use the PowerTAC simulator as a test-bed; also utilise these strategies for our autonomous broker, VidyutVanika, which has been proven efficient in the PowerTAC tournaments.
Hierarchical Decompositions and Termination Analysis for Generalized Planning (Abstract Reprint)
PDF ↗This paper presents new methods for analyzing and evaluating generalized plans that can solve broad classes of related planning problems. Although synthesis and learning of generalized plans has been a longstanding goal in AI, it remains challenging due to fun- damental gaps in methods for analyzing the scope and utility of a given generalized plan. This paper addresses these gaps by developing a new conceptual framework along with proof techniques and algorithmic processes for assessing termination and goal-reachability related properties of generalized plans. We build upon classic results from graph theory to decompose generalized plans into smaller components that are then used to derive hi- erarchical termination arguments. These methods can be used to determine the utility of a given generalized plan, as well as to guide the synthesis and learning processes for generalized plans. We present theoretical as well as empirical results illustrating the scope of this new approach. Our analysis shows that this approach significantly extends the class of generalized plans that can be assessed automatically, thereby reducing barriers in the synthesis and learning of reliable generalized plans.
Partially observable Markov decision processes (POMDPs) provide a flexible representation for real-world decision and control problems. However, POMDPs are notoriously difficult to solve, especially when the state and observation spaces are continuous or hybrid, which is often the case for physical systems. While recent online sampling-based POMDP algorithms that plan with observation likelihood weighting have shown practical effectiveness, a general theory characterizing the approximation error of the particle filtering techniques that these algorithms use has not previously been proposed. Our main contribution is bounding the error between any POMDP and its corresponding finite sample particle belief MDP (PB-MDP) approximation. This fundamental bridge between PB-MDPs and POMDPs allows us to adapt any sampling-based MDP algorithm to a POMDP by solving the corresponding particle belief MDP, thereby extending the convergence guarantees of the MDP algorithm to the POMDP. Practically, this is implemented by using the particle filter belief transition model as the generative model for the MDP solver. While this requires access to the observation density model from the POMDP, it only increases the transition sampling complexity of the MDP solver by a factor of O(C), where C is the number of particles. Thus, when combined with sparse sampling MDP algorithms, this approach can yield algorithms for POMDPs that have no direct theoretical dependence on the size of the state and observation spaces. In addition to our theoretical contribution, we perform five numerical experiments on benchmark POMDPs to demonstrate that a simple MDP algorithm adapted using PB-MDP approximation, Sparse-PFT, achieves performance competitive with other leading continuous observation POMDP solvers.