论文检索

Ilham Wicaksono, Zekun Wu, Rahul Patel, Theo King, Adriano Koshiyama, Philip Colin Treleaven

We present AgentSeer, an interactive observability framework for agentic AI systems. Unlike conventional tracing tools that expose raw spans or model-centric metrics, AgentSeer introduces a dual graph decomposition constructed through a deterministic rule-based parser: a temporal action graph, where each prompt or tool invocation is represented as a distinct action, and a component graph capturing architectural relations among agents, tools, and memory modules. Beyond visualization, AgentSeer enables action-level red teaming, where jailbreak payloads are systematically attached to every action node (including agent messages, tool calls, and memory retrievals) to uncover vulnerabilities invisible to model-level testing. Our demonstration features a six-agent hierarchical testbed with interactive visualization and deployment-oriented safety evaluation applied directly on the same prompts and contexts, systematically revealing high-risk interactions, context-dependent vulnerabilities, and emergent behaviors. By combining structured decomposition, automated red teaming, and rule-based reliability, AgentSeer establishes a safety-first methodology for observability in multi-agent AI.

You-Teng Lin, Li-Yang Zhang, Yi-Tang Chen, Jen-Tzung Chien

This work presented a web-based system which introduces an active-listening strategy in a spoken dialogue for self-disclosure to support mental health of a campus user. To enhance the system usability and safety, this demo is developed to conduct the bilingual (Mandarin/English) spoken dialogue where a high-risk dialogue detection during speech interaction is reliably augmented. In particular, a prompt-driven GPT classifier identifies the utterances indicating self-harm or suicide intent and triggers safety alerts with help center and counselor notification. We also integrate a TTS module for Taiwanese Mandarin and standard English, and redesign the user interface to automatically pop up alert messages when high-risk dialogue is detected. In addition, we collect speech data under diverse mental dialogue scenarios with bilingual speech to enable system analysis, evaluation and refinement. Overall, these extensions build a framework that promotes empathetic interactions, enables timely alert in critical cases, and improves the accessibility for diverse users.

Jianyu Jiang, Zequan Wang, Liang Yao, Shengxiang Xu, Fan Liu

Testing autonomous navigation algorithms of Unmanned Aerial Vehicles (UAVs) in real-world scenarios often entails significant safety risks. In this paper, we aim to build a flexible yet user-friendly UAV autonomous navigation simulator. Ideally, it should closely emulate real-world environments, support diverse UAV models and algorithms, and provide a flexible evaluation framework. Existing frameworks fail to satisfy all three requirements simultaneously. To this end, we present AirNavigation, an integrated simulation platform designed to support the end-to-end workflow of UAV navigation research. Specifically, our system leverages Unreal Engine to simulate highly realistic environments and diverse UAV models. It further facilitates semi-automated scene generation and multi-modal synthetic training data production. To lower the barrier of adoption, we develop a suite of user-friendly interfaces to enable seamless integration of diverse navigation algorithms. Moreover, we introduce a novel evaluation system powered by large language models to deliver personalized and fine-grained performance analysis.

Hai Le

Post-training quantization is widely used to compress large language models (LLMs) for efficient deployment in resource-constrained environments. However, recent work shows that quantization, especially aggressive schemes such as 4-bit QLoRA, can substantially degrade safety alignment, making models more vulnerable to harmful completions and jailbreaks. In this work, we investigate these safety risks and propose a mitigation strategy: projecting quantized parameters back into safety-aligned subspaces. First, we empirically measure safety degradation on benchmark datasets using both safety and utility metrics. Next, we explore projection-based restoration methods to recover alignment-preserving directions in the LoRA adapters of quantized models. Finally, we study how quantization affects mechanistic safety neurons and how hybrid-precision designs can preserve them. By foregrounding the safety implications of model compression, this work aims to support more robust, deployment-ready, and ethically aligned LLMs.

Jeong Hwan Ryu, Azimjon Akhtamov, Md Azher Uddin, Aziz Nasridinov

Ensuring proper use of personal protective equipment (PPE), especially helmets, is essential for workplace safety. Conventional object detectors often fail to distinguish whether a helmet is worn correctly, and existing approaches relying on single-model pipelines are prone to localization errors and false alarms. Moreover, most prior studies do not guarantee real-time performance. To resolve these challenges, we propose a lightweight multimodal approach that integrates a YOLO11-based object detector with a pose estimation model, achieving higher F1 scores and lower false alarm rates while maintaining real-time performance.

Manasa Mariam Mammen, Zafer Kayatas, Stefan Wagner

Scenario-based testing is an important approach for the development and validation of autonomous driving systems, as it enables evaluation across different driving situations. Safety-critical scenarios are especially relevant, but they occur rarely in real-world data, which creates the need for generation methods. In this paper, we present a scalable AI-based approach based on a variational autoencoder that unifies the generation of different types of critical scenarios while introducing controllability through a structured latent space. The integration of unified generation and latent space control advances AI-based scenario generation towards practical use, thereby supporting the requirements of industrial validation pipelines.

Malak Mahdy, Carlos Rubio-Medrano

Autonomous computer-use agents (ACUAs) enable end-to-end computer operation with human-like capabilities, executing commands across applications and making independent decisions. However, their real-world effectiveness and security remain largely untested. A systematic evaluation of ACUAs from Anthropic, OpenAI, and open-source projects categorized them into full computer access and browser-based agents. Findings reveal substantial limitations, with success rates dropping as low as 28% in some cases. Additionally, a 100% rate of unauthorized software installation was observed in certain tasks. The agents also demonstrated susceptibility to prompt injection attacks. The impact of varied prompting strategies on performance was also examined. In response to these weaknesses, a new agent framework designed to address these limitations is proposed. This work bridges agentic AI, human-computer interaction (HCI), and security to address the observed limitations of ACUAs, prioritizing both capability and safety.

Xiaotao Liu, Prashant Mohit, Arvind Easwaran

In safe reinforcement learning (SRL), there exists an inherent conflict between maximizing reward and minimizing cost. We propose a novel approach that effectively resolve the conflict between maximizing reward and minimizing cost in joint optimization.When the cost exceeds the threshold, we perform cost-reducing updates. Otherwise, we compute policy gradients that maximize expected rewards, while using second-order Taylor approximation to evaluate whether these reward-maximizing gradients would violate the cost constraint. If constraint violation is detected, we adjust the gradient direction to maintain safety compliance; otherwise, we execute standard reward-increasing policy updates. This approach helps ensure that reward-seeking updates do not inadvertently increase costs, thereby reducing the likelihood of constraint violations. Empirical tests show our framework successfully manages reward-cost trade-offs through reward augmentation and cost shaping, improving both performance and safety without switching optimization strategies. Results demonstrate that concurrent treatment of both objectives in one policy gradient update is viable for improving safe reinforcement learning methods.

Tae Hoon Lee, Joongheon Kim

In adaptive cruise control (ACC), balancing safety, comfort, and sustainability still remains challenging. Accordingly, we propose a hybrid reinforcement learning framework combining proximal policy optimization (PPO) and deep Q-network (DQN) with a multi-objective reward for autonomous carbon-neutral eco-driving. Experimental results revealed the contrasts between eco and non-eco modes, underscoring how reward design shapes driving behaviors.

Abhilekh Borah, Niranjan Chebrolu, Kokil Jaidka

Refusals must be resilient, not brittle.” Yet guarding refusals against adversarial phrasing and shifting user contexts remains difficult: large language models (LLMs) still yield to jailbreak prompts that evade safety filters and surface harmful content. We propose Refusal Activation Steering (RAS), a training-free, inference-time method that uses contrastive activations to shift LLM responses, biasing generation trajectories toward refusals without altering model weights. The approach is modular and domain-targetable, avoiding collateral refusals on benign queries while strengthening activation- space boundaries for unsafe content. On adversarial evaluations with an 8B instruction-tuned model, we find that steering improves refusal rate by ∼ 52% and reduces attack success rate by ∼ 40%, establishing a lightweight and interpretable safety layer for robust refusal consistency. To foster further research in this domain, we have made our implementation publicly available.

Gouranga Bala, Abhimanyu Chauhan, Amit Sethi

Reliable uncertainty quantification (UQ) is crucial for deploying deep learning models in safety-critical domains. Existing UQ methods often either rely on multi-pass inference, which increases computational cost, or restrict expressiveness by using only final-layer embeddings. In this work, we propose a lightweight evidential meta-model that leverages multi-layer feature fusion from a pretrained backbone, capturing both low-level features and high-level semantics to better estimate uncertainty. To further enhance epistemic fidelity, we integrate maximum weight-entropy (Max-WEnt) regularization, which encourages hypothesis diversity without altering the base network or adding test-time overhead. Experiments across two benchmark settings, medical (BACH, HAM10000, BreakHIS, DIV2K) and natural (ImageNet, SVHN, Fashion-MNIST, ImageNet-C) datasets, demonstrate consistent improvements in AUROC of out-of-distribution detection compared to prior post-hoc UQ methods. Our findings show that combining multi-layer evidential modeling with Max-WEnt provides a robust, efficient, and practical framework for trustworthy AI in high-stakes applications. The meta-model adds only ~0.8M parameters and trains in under four hours on a single 48GB GPU, making it practical for real-world deployment.

Tanmay Ambadkar, Darshan Chudiwal, Greg Anderson, Abhinav Verma

Ensuring safety in deep reinforcement learning is challenging, as formal methods that provide strong guarantees often fail to scale to complex, high-dimensional systems. We introduce RAMPS, a scalable shielding framework that pairs a general-purpose, learned linear dynamics model with a robust, multi-step Control Barrier Function (CBF) for real-time safety interventions. Experiments show RAMPS significantly reduces safety violations in high-dimensional environments compared to state-of-the-art methods, without sacrificing task performance.

Ruihan Zhang

Machine learning is widely used in various areas. However, the current machine learning framework remains vulnerable to issues such as adversarial attacks, fairness violations, and data leakage. These problems are not adequately captured by fitting models to collected data and focusing on test performance metrics alone, like accuracy or F1-score. In practice, machine learning tasks often involve additional quantities of interest, which turns an originally unconstrained optimisation problem (only optimising toward accuracy) into a constrained one. This thesis formally studies machine learning under different types of commonly concerning constraints, such as robustness, fairness, and privacy. I first focus on how the formal machine learning framework can be extended to incorporate robustness, which is a critical factor for safety. After that, I turn to more ethics-related aspects like fairness and privacy, to explore the possibility of formally fitting them into machine learning. My approach differs from empirically pushing up multiple metrics and instead emphasises fundamental ways to understand and address the underlying challenges.

Abdelrahman Sayed Sayed

Higher autonomy is an increasingly common goal in the design of transportation systems for the cities of the future. Recently, part of this autonomy in both rail and maritime transport has come from the field of artificial intelligence and machine learning, particularly for perception tasks (detection and recognition of rail signals, other vessels, or other elements in the vehicle environment) using neural networks. Although AI-based approaches have gained significant popularity in many application fields due to their good performance, their unpredictability and lack of formal guarantees regarding their desired behavior present a major issue for the deployment of such safety-critical systems in urban areas. The goal of my PhD thesis is to design new formal methods to analyze and ensure the safety of such AI-based perception modules in autonomous vehicles. More specifically, my PhD topic aims to formally evaluate the safety of a recently introduced class of continuous AI models which are neural ODE. Neural ODE have already been used successfully for image recognition tasks, showing higher performance compared to classical neural networks, but current work in the literature primarily focuses on their training performance, while they have been barely studied in terms of safety and formal guarantees. The main research directions that will be investigated during my PhD include: • R.D 1: Establishing formal relations between discrete and continuous neural models, and using them to deduce the safety of one model based on the safety verification of the other. • R.D 2: Analysis of the mathematical properties satisfied by the new continuous models (continuity, monotonicity, contraction, stability, etc.). • R.D 3: Exploiting these mathematical properties to study and/or enforce the overall behavior of the neural ODE with respect to various features: stability, stabilization, reachability analysis, safety, formal verification. • Verification and testing of theoretical findings on autonomous underwater vehicles (AUVs) during a 6 month mobility at the Department of Marine Technology, NTNU.

Dongyue Lu

Autonomous driving must handle motion blur, low light, and fast-changing scenes, where RGB frames and event cameras provide complementary strengths. This thesis explores how to fuse them across the perception–reasoning–planning pipeline. It introduces FlexEvent, a frequency-robust detector with adaptive fusion and label-efficient training; Talk2Event, the first benchmark for event–language grounding with attribute-aware modeling; and the EventDrive, an event–frame VLM covering the full driving loop. Together, these contributions advance robust perception, interpretable reasoning, and reliable planning for safety-critical driving through event–frame fusion.

Michelle Ho

Autonomous systems operating in uncertain environments without human intervention must consider several factors, including safety, reliability, and task success. State-of-the-art methods have made progress in addressing these factors individually, but often fail to unify them for deployment in real-world systems. My dissertation aims to combine methods in planning under uncertainty, failure recovery, and explainability, providing a holistic framework for comprehensive safe autonomy in real-world deployment.

Tanmay Ambadkar

While Reinforcement Learning (RL) has demonstrated remarkable success in solving complex sequential decision-making problems, its application in real-world, safety-critical systems is hindered by its reliance on carefully engineered reward functions. Designing effective rewards is notoriously challenging and can lead to unintended or unsafe behaviors, a phenomenon known as reward hacking. Specification-guided RL has emerged as a principled alternative, leveraging formal methods to directly encode high-level objectives, safety requirements, and behavioral constraints. However, the practical utility of this approach is often limited by coarse or under-specified logical formulas and the computational challenge of enforcing safety at scale. This thesis addresses these limitations by developing a unified framework for the automated refinement, scalable enforcement, and flexible adaptation of formal specifications in RL.

Sonal Allana

Explainability has emerged as a pillar of Trustworthy AI for ensuring safety in high-risk application domains. However, the incorporation of explainability to boost the transparency of black-box AI systems can inadvertently introduce unforeseen vulnerabilities. Previous research has drawn attention to privacy leakage, malicious or otherwise, from explainable interfaces leading to identification of individuals and exposure of sensitive personal information. Privacy preservation methods used in response to this leakage are found to adversely affect the utility of the system, including the degradation of model accuracy and explanation quality. The proposed thesis will examine the advancement of Privacy Enhancing Technologies (PETs) in Explainable AI (XAI) while ensuring that users remain at the core of the design process. The main objectives of this research are: (1) determining defenses for privacy attacks in XAI (2) building interpretable algorithms for private models and (3) examining user requirements for privacy preserving XAI. This research is expected to yield characteristics of privacy preserving XAI, guidelines and recommendations for effectively building privacy compliant XAI while considering the diverse needs of end users. The research outcomes will enable developers and researchers in designing XAI that is safe for deployment and considers the balance between privacy, explainability and utility.

Phan Xuan Tan, Eiji Kamioka, Van Nguyen

Generative AI has moved from pilots to everyday practice, delivering gains in productivity and accessibility while surfacing present-day risks—hallucinations and reliability failures, bias and unfairness, prompt-injection attacks, and so on. These trends make AI safety education a core competency. In this paper, we survey global AI safety curricula and, in the Japanese context, observe strong policy momentum but relatively few courses that explicitly combine capability instruction with systematic safety evaluation. In response, we developed a 7-week, graduate-level intensive at a private science and engineering university in Japan, with enrollment open to international exchange students at both the undergraduate and graduate levels. The curriculum progresses from machine-learning foundations to generative models and alignment, with introductory agent topics included to support risk reasoning. Delivery combines weekly lectures, invited talks from academia and industry, structured group discussions, and a final presentation plus a paper-style final project focused on risk evaluation and mitigation planning. An end-of-course survey indicates high perceived learning and positive experience and one student project later resulted in a peer-reviewed workshop paper at ICLR 2025.

Mehdi Yekrangi, Houssem Chatbri, Claudia Beatrice Chianella, Owen O'Neill

The increasing adoption of large language models in the fi-nancial sector introduces significant challenges related to the handling of sensitive financial information (SFI). Existing general-purpose content safety solutions, or guardrails, often fall short in detecting domain-specific risks inherent in finan-cial data processing. This study addresses these gaps by de-veloping a comprehensive taxonomy of SFI, grounded in globally recognized financial, information security, and AI governance standards. Leveraging this taxonomy, we synthe-sized an extensive dataset encompassing diverse categories of SFI and trained GARD (Generative Adversarial network Risk Detection) model to detect sensitive content in both in-puts and outputs of GenAI systems within the financial do-main. Our evaluation compared GARD against commercial guardrail solutions, including the OpenAI Moderation API and Microsoft Azure Content Safety (ACS). The results demonstrated that while commercial solutions maintained high precision, their recall was substantially lower, indicating many risky instances went undetected. In contrast, our model achieved a recall score of 0.98, significantly outperforming the benchmarks and enhancing SFI detection. These findings underscore the necessity of domain-specific guardrails tai-lored to the financial sector to ensure robust AI safety and compliance. In conclusion, this work contributes (1) A de-tailed taxonomy of SFI tailored for GenAI applications, (2) A comprehensive synthetic dataset that encompasses a wide range of sensitive topics relevant to the domain and (3) A high-performance risk detection model that can be deployed independently or alongside existing solutions to improve con-tent safety in financial services. This approach promotes trust, mitigates financial, legal, and reputational risks, and supports the responsible adoption of GenAI technologies in sensitive domains.