论文检索

Applications · Everything Else

Ayano Hiranaka, Ya-Chuan Hsu, Stefanos Nikolaidis, Erdem Biyik, Daniel Seita

AI assistants in human-AI collaboration often correct suboptimal human actions through behavioral feedback (e.g., alerts or steering-wheel nudges in assistive driving). Such interventions can mitigate immediate errors, but long-term improvement requires addressing the underlying misconceptions that cause repeated mistakes. We introduce SENSEI, a framework that infers user misconceptions from interaction behavior and provides targeted, minimal yet sufficient suggestions to correct them. Our approach departs from action- or trajectory-level interventions by operating over a structured knowledge representation to localize and correct the sources of erroneous behavior. Across three long-horizon tasks with diverse misconceptions and corresponding behaviors, SENSEI demonstrates zero-shot compositional generalization, disentangling multiple overlapping misconceptions despite training only on single-misconception cases. A user study further shows that our method identifies real human misconceptions and provides effective guidance that improves long-horizon task performance, successfully correcting 90% of student misconceptions.

Social Aspects · Accountability, Transparency, and Interpretability

Hak Hyun Kim, Benjamin Huh, Soroush Vosoughi

Multi-Agent Systems (MAS) are deployed at unprecedented scale—from warehouse robot fleets to autonomous vehicle networks to collaborative LLM agents—yet methods for explaining their behavior remain fragmented and underspecified. We analyze 2,381 MAS-related papers from top machine learning venues (2021–2025) and find systematic gaps: 65% omit stakeholder specifications, 76% lack quantitative evaluation bounds, and 99% ignore auditability requirements. These gaps render current MAS XAI research non-comparable, non-reproducible, and disconnected from deployment requirements. We argue that MAS XAI research requires explicit specification of two contracts before developing methods. The **Research Contract** defines six elements: explanandum, stakeholder, intervention unit, evaluation bounds, adversarial context, auditability. The **Agent Contract** defines expected behaviors through obligations, permissions, prohibitions, violation criteria, and accountability chains—providing the baseline against which deviations are explained. These contracts are method-agnostic and architecture-agnostic, applicable to LLM-based, learning-based, and hybrid MAS. Through case studies spanning warehouse robotics, autonomous vehicles, and LLM agent systems, we demonstrate that contracts transform vague post-hoc descriptions into verifiable, actionable, and comparable explanations. We call on researchers to adopt contracts in their work, conferences to encourage specification in submissions, and platforms to integrate contract templates into MAS benchmarks.

Social Aspects · Alignment

Junhyuk Choi, Sohhyung Park, chanhee cho, Hyeonchu Park, Bugeun Kim

While LLM-as-a-Judge is widely used in automated evaluation, existing validation practices primarily operate at the level of observed outputs, offering limited insight into whether LLM judges themselves function as stable and reliable measurement instruments. To address this limitation, we introduce a two-phase diagnostic framework for assessing reliability of LLM-as-a-Judge, grounded in Item Response Theory (IRT). The framework adopts Graded Response Model (GRM) of IRT and formalizes reliability along two complementary dimensions: (1) intrinsic consistency, defined as the stability of measurement behavior under prompt variations, and (2) human alignment, capturing correspondence with human quality assessments. We empirically examine diverse LLM judges with this framework, and show that leveraging IRT-GRM yields interpretable signals for diagnosing judgments systematically. These signals provide practical guidance for verifying reliablity of LLM-as-a-Judge and identifying potential causes of unreliability.

Deep Learning · Large Language Models

Priya Pitre, Gaurav Srivastava, Lu Zhang, Le Wang, Naren Ramakrishnan, Xuan Wang

Multi-agent LLM debates are increasingly deployed in domains such as policy analysis and city planning, where no objective ground truth exists. Despite this, debate quality is typically evaluated using outcome-based proxies such as LLM-as-judge scores that provide little insight into whether meaningful deliberation has occurred. Additionally, consensus and majority vote are viewed as ideal goals without analyzing the underlying interaction dynamics beneath them. In this work, we introduce a diagnostic evaluation framework that measures debate quality by measuring both the outcome and the process. Grounded in deliberative theory, our framework defines four interpretable process-level metrics capturing engagement, responsiveness, influence asymmetry, and balance, and two outcome-based metrics capturing stability and agent utility. Across both objective benchmarks and real-world domains, we find that process-level diagnostics are consistently more informative than commonly used outcome-based proxies. They better reflect correctness when ground truth exists and align more closely with human judgments of deliberative quality when it does not, revealing interaction failures that outcome-only measures fail to capture. These results demonstrate that process-level diagnostics are necessary for reliable evaluation of multi-agent debates and provide a principled foundation for analyzing and designing deliberative LLM systems.

Reinforcement Learning · Everything Else

Xiaoyuan Cheng, Wenxuan Yuan, Boyang Li, Yuanchao Xu, Yiming Yang, Hao Liang, Bei Peng, Robert Loftin, Zhuo Sun, Yukun Hu

Diffusion policy sampling enables reinforcement learning (RL) to represent multimodal action distributions beyond suboptimal unimodal Gaussian policies. However, existing diffusion-based RL methods primarily focus on offline setting for reward maximization, with limited consideration of safety in online settings. To address this gap, we propose Augmented Lagrangian-Guided Diffusion (ALGD), a novel algorithm for off-policy safe RL. By revisiting optimization theory and energy-based modeling, we show that the instability of primal–dual methods arises from the non-convex Lagrangian landscape. In diffusion-based safe RL, the Lagrangian can be interpreted as an energy function guiding the denoising dynamics; counter-intuitively, direct usage destabilizes both policy generation and training. ALGD resolves this issue by introducing an augmented Lagrangian that locally convexifies the energy landscape, yielding a stabilized policy generation and training, without altering the distribution of optimal policy. Theoretical analysis and extensive experiments demonstrate that ALGD is both theoretically grounded and empirically effective, achieving strong and stable performance across diverse environments.

Social Aspects · Accountability, Transparency, and Interpretability

Harry Mayne, Justin S. Kang, Dewi Gould, Kannan Ramchandran, Adam Mahdi, Noah Siegel

LLM self-explanations are often presented as a promising tool for AI oversight, yet their faithfulness to the model's true reasoning process is poorly understood. Existing faithfulness metrics have critical limitations, typically relying on identifying unfaithfulness via adversarial prompting or detecting reasoning errors. These methods overlook the predictive value of explanations. We introduce *Normalized Simulatability Gain* (NSG), a general and scalable metric based on the idea that a faithful explanation should allow an observer to learn a model's decision-making criteria, and thus better predict its behavior on related inputs. We evaluate 18 frontier proprietary and open-weight models, e.g., Gemini 3, GPT-5.2, and Claude 4.5, on 7,000 counterfactuals from popular datasets covering health, business, and ethics. We find self-explanations substantially improve prediction of model behavior (11-37% NSG). Self-explanations also provide more predictive information than explanations generated by external models, even when those models are stronger. This implies an advantage from self-knowledge that external explanation methods cannot replicate. Our approach also reveals that, across models, 5-15% of self-explanations are egregiously misleading. Despite their imperfections, we show a positive case for self-explanations: they encode information that helps predict model behavior.

Deep Learning · Foundation Models

Anthony Bao, Venkata Hasith Vattikuti, Jeffrey Lai, William Gilpin

Time Series Foundation Models (TSFMs) leverage extensive pretraining to accurately predict unseen time series during inference, without the need for task-specific fine-tuning. Through large-scale evaluations on standard benchmarks, we find that leading transformer-based TSFMs exhibit redundant components in their intermediate layers. We introduce a set of tools for mechanistic interpretability of TSFMs, including ablations of specific components and direct logit attribution on the residual stream. Our findings are consistent across several leading TSFMs with diverse architectures, and across a diverse set of real-world and synthetic time-series datasets. We discover that all models in our study are robust to ablations of entire layers. Furthermore, we develop a theoretical framework framing transformers as kernel regressors, motivating a purely intrinsic strategy for ablating heads based on the stable rank of the per-head projection matrices. Using this approach, we uncover the specific heads responsible for degenerate phenomena widely observed in TSFMs, such as parroting of motifs from the context and seasonality bias. Our study sheds light on the universal properties of this emerging class of architectures for continuous-time sequence modeling.

Reinforcement Learning · Deep RL

Zhishuai Liu, Pan Xu

We study a simple yet principled modification of classical Q-learning that clips the value estimate in the Bellman backup by a threshold $\lambda$. The resulting algorithm, clipped Q-learning, is motivated by a key theoretical insight: the clipped Bellman backup is an unbiased one-sample estimation of a robust Bellman operator arising naturally from a transition-regularized MDP framework. This formulation corresponds to optimizing performance against a specific class of adversarial dynamics perturbations at the test time that reallocate transition probability mass away from high-value states, thereby inducing conservative but stable decision making. Under this interpretation, clipped Q-learning can be viewed as tracking the fixed point of the robust Bellman equation and learning policies that hedge against adversarial dynamics shifts at test time. We analyze two clipped Q-learning variants with an optimistic exploration bonus and establish polynomial regret guarantees, demonstrating statistical efficiency. Beyond the tabular setting, our framework suggests that value clipping is a modular mechanism that can be incorporated into general value-based RL algorithms with function approximation. As a proof of concept, we evaluate a clipped Double DQN algorithm on a control task and observe robustness improvements consistent with our theoretical predictions.

Charles Westphal, Keivan Navaie, Fernando Rosas

Fine-tuned LLMs can covertly encode prompt secrets into outputs via steganographic channels. Prior work demonstrated this threat but relied on trivially recoverable encodings. We formalize payload recoverability via classifier accuracy and show previous schemes achieve 100\% recoverability. In response, we introduce low-recoverability steganography, replacing arbitrary mappings with embedding-space-derived ones. For Llama-8B (LoRA) and Ministral-8B (LoRA) trained on TrojanStego prompts, exact secret recovery rises from 17$\rightarrow$30\% (+78\%) and 24$\rightarrow$43\% (+80\%) respectively, while on Llama-70B (LoRA) trained on Wiki prompts, it climbs from 9$\rightarrow$19\% (+123\%), all while reducing payload recoverability. We then discuss detection. We argue that detecting fine-tuning-based steganographic attacks requires approaches beyond traditional steganalysis. Standard approaches measure distributional shift, which is an expected side-effect of fine-tuning. Instead, we propose a mechanistic interpretability approach: linear probes trained on later-layer activations detect the secret with up to 33\% higher accuracy in fine-tuned models compared to base models, even for low-recoverability schemes. This suggests that malicious fine-tuning leaves actionable internal signatures amenable to interpretability-based defenses.

Applications · Computer Vision

Yeongtak Oh, Sangwon Yu, Junsung Park, Han Cheol Moon, Jisoo (Jenny) Mok, Sungroh Yoon

Despite recent progress in vision-language models (VLMs), existing approaches often fail to generate personalized responses based on the user's specific experiences, as they lack the ability to associate visual inputs with a user’s accumulated visual-textual context. We newly formalize this challenge as contextualized visual personalization, which requires the visual recognition and textual retrieval of personalized visual experiences by VLMs when interpreting new images. To address this issue, we propose CoViP, a unified framework that treats personalized image captioning as a core task for contextualized visual personalization and improves this capability through reinforcement-learning-based post-training and caption-augmented generation. We further introduce diagnostic evaluations that explicitly rule out textual shortcut solutions and verify whether VLMs truly leverage visual context. Extensive experiments demonstrate that existing open-source and proprietary VLMs exhibit substantial limitations, while CoViP not only improves personalized image captioning but also yields holistic gains across downstream personalization tasks. These results highlight CoViP as a crucial stage for enabling robust and generalizable contextualized visual personalization.

Social Aspects · Accountability, Transparency, and Interpretability

Xi Chen, Mingyu Jin, Jingcheng (Frank) Niu, Yutong Yin, Jinman Zhao, Bangwei Guo, Dimitris Metaxas, Zhaoran Wang, Yutao Yue, Gerald Penn

In this paper, we present empirical and theoretical evidence against a central but largely implicit assumption in circuit and sheaf discovery (CSD), which we term the *Functional Anisotropy Hypothesis*: the idea that functions in large language models (LLMs) are localised to a unique or near-unique internal mechanism. We show that a single LLM task can instead be supported by multiple, structurally distinct circuits or sheaves that are simultaneously faithful, sparse, and complete. To systematically uncover such competing mechanisms, we introduce Overlap-Aware Sheaf Repulsion, a method that augments the CSD objective with an explicit penalty on structural overlap across multiple discovery runs, enabling the discovery of circuits or sheaves with strong task performance but minimal shared structure across a plethora of common CSD benchmarks. We find that this phenomenon becomes increasingly pronounced as the number of discovered sheaves grows and persists robustly across major CSD methods. We further identify an ultra-sparse three-edge sheaf and show that none of its edges is individually indispensable, undermining even weakened notions of canonical or essential components. To explain these findings, we propose a *Distributive Dense Circuit Hypothesis* and provide a theoretical analysis demonstrating that non-unique, low-overlap circuit explanations arise naturally from high-dimensional superposition under mild assumptions. Together, our results suggest that mechanistic explanations in LLMs are inherently non-canonical and call for a rethinking of how CSD results should be interpreted and evaluated.

General Machine Learning · Evaluation

Ghanem BAHRINI, Morgane Barbet-Massin, Sebastien Razakarivony, Valerie Gares, Jean-François Dupuy

Evaluating survival models under censoring is inherently challenging, yet standard evaluation practices are often applied without explicitly assessing how censoring distorts metric reliability. Performing a large experimental study, we analyze and quantify how survival evaluation metrics are affected in fundamentally different ways by the censoring rate and the censoring mechanism. Using a controlled semi-synthetic framework, we vary both the censoring mechanism (administrative, independent, covariate-dependent) and the censoring rate, and compare standard evaluations based on censored data with oracle evaluations using fully observed event times. This controlled setting enables us to quantify distortions along two complementary axes: numerical bias and preservation of model ranking. Across datasets and metric families, we find that censoring induces systematic, mechanism-dependent distortions. Moderate numerical bias, if not properly addressed, can lead to unreliable model comparison as censoring increases. These findings reveal fundamental limitations of common benchmarking practices and call for more careful interpretation of survival evaluation under realistic censoring.

Social Aspects · Accountability, Transparency, and Interpretability

Deepika Vemuri, Sayanta Adhikari, Ankit Saha, Krishn Vishwas Kher, Vineeth N Balasubramanian

Learning semantics is essential for deep learning models to be interpretable and better aligned with human reasoning. Concept-based models approach this by representing classes through meaningful semantic abstractions, but typically treat all concepts as a flat, unstructured set learned at a single neural network layer. This overlooks a fundamental property of human semantic understanding: concepts being organized hierarchically, from general to specific. While deep networks do learn a hierarchy of visual features, this structure is rarely aligned with explicit semantic hierarchies. Drawing on Formal Concept Analysis, we demonstrate that formal concept lattices provide principled semantic scaffolds to guide neural network learning. These lattices naturally identify where in the network concepts should be learned based on their level of generality. This allows the model to develop staged, semantically grounded representations throughout its depth. Empirical results on real-world datasets show that our models produce more interpretable embeddings, support more effective interventions, and learn concept representations that are both meaningful and hierarchically structured.

Applications · Health / Medicine

Guanghui Min, Tianhao Huang, Ke Wan, Qi Wang, Chen Chen

Reliable epidemic forecasting is critical for public health decision-making yet remains challenging due to data sparsity and the non-stationary nature of disease dynamics. While recent hybrid models attempt to integrate mechanistic principles with data-driven approaches, they often relegate mechanistic priors to merely auxiliary features or regularization terms. This design not only obscures the interpretability of the mechanistic contribution but also fails to inherit the capability of physical models to generalize under non-stationary dynamics, as the core architecture remains predominantly data-driven. To address these limitations, we propose EpiDiff, a unified framework that synergizes epidemiological domain knowledge with the generative power of diffusion models. Unlike methods that rigidly fuse features, EpiDiff employs a novel uncertainty-aware steering mechanism during inference. Specifically, we quantify the posterior uncertainty of mechanistic estimations and use it to dynamically modulate the diffusion process. Extensive experiments on real-world datasets demonstrate that EpiDiff consistently outperforms state-of-the-art baselines in accuracy and robustness, particularly under non-stationary distributions, while offering transparent insights into model reliance by explicitly visualizing when the forecast is governed by mechanistic laws versus data-driven patterns. Our code and datasets are available at https://anonymous.4open.science/r/epidiff-4782.

Social Aspects · Accountability, Transparency, and Interpretability

Eric Bigelow, Daniel Wurgaft, YingQiao Wang, Hidenori Tanaka, Tomer Ullman, Noah Goodman, Ekdeep Singh Lubana

Large language models (LLMs) can be controlled at inference time through prompts (in-context learning) and internal activations (activation steering). Different accounts have been proposed to explain these methods, yet their common goal of controlling model behavior raises the question of whether these seemingly disparate methodologies can be seen as specific instances of a broader framework. Motivated by this, we develop a unifying, predictive account of LLM control from a Bayesian perspective. Specifically, we posit that both context- and activation-based interventions impact model behavior by altering its belief in latent concepts: steering operates by changing concept priors, while in-context learning leads to an accumulation of evidence. This results in a closed-form Bayesian model that is highly predictive of LLM behavior across context- and activation-based interventions in a set of domains inspired by prior work on many-shot in-context learning. This model helps us explain prior empirical phenomena - e.g., sigmoidal learning curves as in-context evidence accumulates--while predicting novel ones--e.g., additivity of both interventions in log-belief space, which results in distinct phases such that sudden and dramatic behavioral shifts can be induced by slightly changing intervention controls. Taken together, this work offers a unified account of prompt-based and activation-based control of LLM behavior, and a methodology for empirically predicting the effects of these interventions.

General Machine Learning · Transfer, Multitask and Meta-learning

Gang Liu, Xiaoxuan Zhang, Yuhong Feng, Mingyang Zhou, Xiaoqun Wu, Hao Liao, Rui Mao

Few-shot classification aims to adapt a pretrained model to novel classes with limited examples. While current methods often heuristically combine pretrained knowledge and few-shot evidence, we seek a more principled understanding of their relationship. In this paper, we propose a Bayesian-inspired optimal integration framework(BOIF) that interprets pretrained models as priors and few-shot evidence as likelihoods. Under conditional independence approximation, we show that the optimal log-posterior decomposes into the sum of prior logits and likelihood logits. This leads to a simple yet effective design principle: decouple the prior and likelihood pathways and combine their logits additively. Guided by this principle, we implement BOIF using CLIP with two novel enhancements: (1) a multi-level feature adapter to enrich visual representations, and (2) a simplified cache module for likelihood estimation. Extensive experiments on 11 benchmarks show BOIF achieves state-of-the-art performance (e.g., 80.61\% average accuracy at 16-shot) and strong out-of-distribution robustness. Our work provides both a principled perspective and an effective instantiation for few-shot adaptation.

Deep Learning · Large Language Models

Uzay Macar, Li Yang, Atticus Wang, Peter Wallich, Emmanuel Ameisen, Jack Lindsey

Recent work shows that LLMs can sometimes detect when steering vectors are injected into their residual stream and identify the injected concept, a phenomenon cited as evidence of "introspective awareness." But what mechanisms underlie this capability, and do they reflect genuine introspective circuitry or more shallow heuristics? We investigate these questions in open-source models and establish three main findings. First, introspection is behaviorally robust: detection achieves moderate true positive rates with 0% false positives across diverse prompts. We also find this capability emerges specifically from post-training rather than pretraining. Second, introspection is not reducible to a single linear confound: anomaly detection relies on distributed MLP computation across multiple directions, implemented by interpretable gate and evidence-carrier features. Third, models possess greater introspective capability than is elicited by default: ablating refusal directions improves detection by ~50% and a trained steering vector improves detection by ~75%. Overall, our results suggest that introspective awareness is behaviorally robust, grounded in nontrivial internal anomaly detection, and likely could be substantially improved in future models.

Social Aspects · Accountability, Transparency, and Interpretability

Ariel Fargion, Lahav Dabah, Tom Tirer

Conformal Prediction (CP) has emerged as a powerful statistical framework for reliable classification, which generates a prediction set, guaranteed to include the true label with a pre-specified probability. The performance of CP methods is typically assessed by their average prediction set size. In setups where the classes can be partitioned into semantic groups, e.g., based on shared downstream actions or more interpretable coarse labels, users can benefit from prediction sets that are not only small but also contain a limited number of groups. This paper begins by addressing this problem and ultimately offers a widely applicable tool for boosting any CP method on any dataset. First, given a class partition, we propose augmenting the CP score function with a term that penalizes predictions with "out-of-group" errors. We theoretically analyze this strategy and prove its advantages for group-related metrics. Surprisingly, we show mathematically that, for common class partitions, it can also reduce the average set size of any CP score function. Our analysis reveals the class similarity factors behind this improvement and motivates us to propose a model-specific variant, which does not require any human semantic partition and can further reduce the prediction set size. Finally, we present an extensive empirical study, encompassing prominent CP methods, multiple models, and several datasets, which demonstrates that our class-similarity-based approach consistently enhances CP methods.

General Machine Learning · Scalable Algorithms

Manish Nagaraj, Sakshi Choudhary, Utkarsh Saxena, Deepak Ravikumar, Kaushik Roy

Instruction tuning is essential for aligning large language models (LLMs) to downstream tasks and commonly relies on large, diverse corpora. However, small, high-quality subsets, known as coresets, can deliver comparable or superior results, though curating them remains challenging. Existing methods often rely on coarse, sample-level signals like gradients, an approach that is computationally expensive and overlooks fine-grained features. To address this, we introduce TRIM (Token Relevance via Interpretable Multi-layer Attention), a forward-only, token-centric framework. Instead of using gradients, TRIM operates by matching underlying representational patterns identified via attention-based "fingerprints" from a handful of target samples. Such an approach makes TRIM highly efficient and uniquely sensitive to the structural features that define a task. Coresets selected by our method consistently outperform state-of-the-art baselines by up to 9% on downstream tasks and even surpass the performance of full-data fine-tuning in some settings. By avoiding expensive backward passes, TRIM achieves this at a fraction of the computational cost. These findings establish TRIM as a scalable and efficient alternative for building high-quality instruction-tuning datasets.

General Machine Learning · Representation Learning

Aviral Chawla, Galen Hall, Juniper Lovato

Foundation models must handle multiple generative processes, yet mechanistic interpretability largely studies capabilities in isolation; it remains unclear how a single transformer organizes multiple, potentially conflicting "world models". Previous experiments on Othello playing neural-networks test world-model learning but focus on a single game with a single set of rules. We introduce *MetaOthello*, a controlled suite of Othello variants with shared syntax but different rules or tokenizations, and train small GPTs on mixed-variant data to study how multiple world models are organized in a shared representation space. We find that transformers trained on mixed-game data do not partition their capacity into isolated sub-models; instead, they converge on a mostly shared board-state representation that transfers causally across variants. Linear probes trained on one variant can intervene on another's internal state with effectiveness approaching that of matched probes. For isomorphic games with token remapping, representations are equivalent up to a single orthogonal rotation that generalizes across layers. When rules partially overlap, early layers maintain game-agnostic representations while a middle layer identifies game identity, and later layers specialize. *MetaOthello* offers a path toward understanding not just whether transformers learn world models, but how they organize many at once.