论文检索

Social Aspects · Accountability, Transparency, and Interpretability

Constantin Venhoff, Iván Arcuschin, Phil Torr, Arthur Conmy, Neel Nanda

Why do thinking language models outperform their base counterparts, and what exactly do they learn during training? We introduce constructive model diffing, a framework for understanding fine-tuned models by explicitly constructing the base-to-fine-tuned difference from interpretable components to produce hybrid models, and measuring how well they recover the fine-tuned model's performance. For thinking models, we decompose the diff into two components: reasoning mechanisms (steering vectors that activate specific behaviors in the base model) and reasoning heuristics (a classifier that determines when each mechanism should fire). To ground this decomposition, we develop an unsupervised methodology using Sparse Autoencoders to discover interpretable taxonomies of reasoning behaviors. Evaluating nine model configurations (five RL-trained, four distilled), we find a striking difference between training methods: the hybrid models for the five RL-trained models achieve much higher performance recovery compared to the four distilled models. This indicates RL-trained models primarily learn sophisticated heuristics for deploying pre-existing base model mechanisms, while distillation affects the mechanisms themselves. These results provide a new lens for understanding what different training paradigms teach, with potential implications for efficient reasoning model development.

Social Aspects · Accountability, Transparency, and Interpretability

Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, Arthur Conmy

Recent studies indicate that when faced with explicit biases in prompts, models often omit mentioning these biases in their Chain-of-Thought (CoT) output, revealing that verbalized reasoning can give an incorrect picture of how models arrive at conclusions (unfaithfulness). In this work, we show that unfaithful CoT also occurs on naturally worded, non-adversarial prompts without adding artificial biases or editing model outputs. We find that when separately presented with the questions "Is X bigger than Y?" and "Is Y bigger than X?", models sometimes produce superficially coherent arguments to justify systematically answering Yes to both questions or No to both questions, despite such responses being logically contradictory. We present preliminary evidence that this is due to models' implicit biases towards Yes or No, labeling this *Implicit Post-Hoc Rationalization*. Our results reveal rates up to 13% for production models, and while frontier models are more faithful, none are entirely so, including thinking models like DeepSeek R1 (0.37%) and Sonnet 3.7 with thinking (0.04%). We also investigate *Unfaithful Illogical Shortcuts*, where models use subtly illogical reasoning to make speculative answers to hard math problems seem rigorously proven. Our findings indicate that while CoT can be useful for assessing outputs, it is not a complete account of a model's internal reasoning and should be used with caution in agentic or safety-critical settings.

Social Aspects · Alignment

Atticus Wang, Iván Arcuschin, Arthur Conmy

Large language model (LLM) post-training typically relies on a training signal from a reward model (RM), such as for reinforcement learning from human feedback. Previous work shows that this signal can be biased in attributes such as length, format, and sycophancy. In this work, we introduce and study the research problem of automatically finding reward model biases in natural language. We offer a simple approach of using an LLM to iteratively propose and refine candidate biases. Our method can recover known biases and surface novel ones: for example, we found that Skywork-V2-8B, a leading open-weight reward model, often mistakenly favors responses with redundant spacing and responses with hallucinated content. In addition, we show evidence that iteration provides benefits over flat best-of-N search. We hope our work contributes to further research on improving RMs through automated interpretability methods.

Social Aspects · Accountability, Transparency, and Interpretability

Iván Arcuschin, David Chanin, Adrià Garriga-Alonso, Oana-Maria Camburu

Large Language Models (LLMs) often provide chain-of-thought (CoT) reasoning traces that appear plausible, but may hide internal biases. We call these *unverbalized biases*. Monitoring models via their stated reasoning is therefore unreliable, and existing bias evaluations typically require predefined categories and hand-crafted datasets. In this work, we introduce a fully automated, black-box pipeline for detecting task-specific unverbalized biases. Given a task dataset, the pipeline uses LLM autoraters to generate candidate bias concepts. It then tests each concept on progressively larger input samples by generating positive and negative variations, and applies statistical techniques for multiple testing and early stopping. A concept is flagged as an unverbalized bias if it yields statistically significant performance differences while not being cited as justification in the model's CoTs. We evaluate our pipeline across six LLMs on three decision tasks (hiring, loan approval, and university admissions). Our technique automatically discovers previously unknown biases in these models (e.g., Spanish fluency, English proficiency, writing formality). In the same run, the pipeline also validates biases that were manually identified by prior work (gender, race, religion, ethnicity). More broadly, our proposed approach provides a practical, scalable path to automatic task-specific bias discovery.

Deep Learning · Large Language Models

Haonan Song, Qingchen Xie, Huan Zhu, Feng Xiao, Luxi Xing, Liu Kang, Fuzhen Li, Zhiyong Zheng, Feng Jiang, Ziheng Li 等

Generative Reward Models (GRMs) have demonstrated strong performance in reward modeling, due to their interpretability and potential for refinement through reinforcement learning (RL). However, widely used pairwise GRMs create a computational bottleneck in reinforcement learning from human feedback (RLHF), when calibrating or aggregating preference signals over $n$ candidates, often incurring $\mathcal{O}(n^2)$ pairwise judgments. To address this issue, we propose Intergroup Relative Preference Modeling (IRPM), an RL-based method that extends the Bradley--Terry preference-learning paradigm via intergroup comparisons to train \emph{pointwise} GRMs from pairwise preference data. IRPM derives pointwise reward for each response by contrasting groups of chosen vs.\ rejected samples, enabling pointwise scores comparable across candidate sets and $\mathcal{O}(n)$ reward evaluation for a variable number of candidates during RL training, while preserving interpretability and scalability. Experiments show that IRPM achieves state-of-the-art performance among pointwise GRMs on RM-Bench, JudgeBench and RewardBench, and approaches the performance of leading pairwise GRMs. In addition, IRPM achieves substantial gains in post-training evaluations, demonstrating its effectiveness.

Applications · Neuroscience, Cognitive Science

Ganchao Wei, Daniela de Albuquerque, Miles Martinez, Shiyang Pan, John Pearson

While neuroscience experiments have repeatedly demonstrated the involvement of large populations of neurons in even simple behaviors, these studies have just as often reported that the collective dynamics of neural activity are approximately low-dimensional. As a result, methods for identifying low-dimensional latent representations of time series data have become increasingly prominent in neuroscience. However, most existing methods either ignore temporal structure or model time evolution using latent dynamical systems approaches. In the first case, dynamics may be distorted or even scrambled in the latent space, while in the second, many possible latent dynamics may give rise to the same data. Here, we address these challenges using a novel flow-matching approach in which data are generated by a pair of flow fields, one governing time evolution, the other a mapping between data and a low-dimensional latent space. Importantly, the dimension-reducing flow is trained to minimize distortions of the temporal dynamics, learning an identifiable low-dimensional representation that preserves temporal relations in the original data. Additionally, we constrain our latent spaces to have low-dimensional support in a soft, parameterized manner, taking inspiration from ideas on nested dropout. Across both neural and behavioral data, we show that this dual flow approach produces both more interpretable dynamics and higher-quality reconstructions than competing models, including in noise-dominated data sets where conventional approaches fail.

Applications · Computer Vision

Zihao Zhu, Wenyuan Zhao, Nuo Chen, Chao Tian, Zhiwen Fan

Geometric foundation models hold promise for unconstrained dense geometry prediction from uncalibrated images; however, in current feed-forward designs, their predicted confidence scores are heuristic, lack probabilistic interpretation, and often fail to indicate where and how much the predicted geometry can be trusted. To fill this gap, we present ***Trust3R***, a trust-aware 3D reconstruction framework that pairs a lightweight gated residual mean refinement with evidential learning to predict pointmap evidence under a Normal-Inverse-Wishart prior and yield a closed-form multivariate Student-t predictive distribution. This design provides probabilistically grounded pointmap uncertainty estimates while adding moderate inference overhead. We evaluate on diverse indoor and outdoor benchmarks, and compare against MASt3R's built-in confidence map as well as common uncertainty-aware baselines spanning single-pass heteroscedastic regression and sampling-based methods such as MC dropout and deep ensembles. Experimental results show that Trust3R consistently improves risk--coverage and sparsification, and generally improves geometric accuracy. Trust3R consistently improves uncertainty ranking across benchmarks (e.g. on ScanNet++: 25\% lower AURC and 41\% lower AUSE), enabling uncertainty-aware weighting for downstream alignment and fusion.

Social Aspects · Accountability, Transparency, and Interpretability

Yining Jiao, Shankar Bhamidi, Carlton ZDANSKI, Julia Kimbell, Andrew Prince, Cameron Worden, Samuel Kirse, Christopher Rutter, Benjamin Shields, William Dunn 等

Understanding how anatomical shapes evolve in response to developmental covariates—and quantifying their spatially varying uncertainties—is critical in healthcare research. Existing approaches typically rely on global time-warping formulations that ignore spatially heterogeneous dynamics. We introduce \texttt{PRISM}, a novel framework that bridges implicit neural representations with uncertainty-aware statistical shape analysis. \texttt{PRISM} models the conditional distribution of shapes given covariates, providing spatially continuous estimates of both the population mean and covariate-dependent uncertainty at arbitrary locations. A key theoretical contribution is a closed-form Fisher Information metric that enables efficient, analytically tractable local temporal uncertainty quantification via automatic differentiation. Experiments on three synthetic datasets and one clinical dataset demonstrate \texttt{PRISM}'s strong performance across diverse tasks—from modeling shape evolution to anomaly detection—within a unified framework, while providing interpretable and clinically meaningful uncertainty estimates.

Reinforcement Learning · Policy Search

Doyeon Lee, Eunyi Lyou, Hyunsoo Cho, Soo Kyung Kim, Joonseok Lee, Jaemoo Choi

GRPO-style reinforcement learning (RL)-based LLM fine-tuning algorithms have recently gained popularity. Relying on heuristic trust-region approximations, however, they can lead to brittle optimization behavior, as global importance-ratio clipping and group-wise normalization fail to regulate samples whose importance ratios fall outside the clipping range. We propose Query-Adaptive Trust-Region policy Optimization (QUATRO), which directly enforces trust-region constraints through a principled optimization. This yields a clear and interpretable objective that enables explicit control over policy updates and stable, entropy-controlled optimization, with a stabilizer terms arising intrinsically from the exact trust-region formulation. Empirically verified on diverse mathematical reasoning benchmarks, QUATRO shows stable training under increased policy staleness and aggressive learning rates, maintaining well-controlled entropy throughout training.

Masahiro Kato

We propose \emph{ScoreMatchingRiesz}, a family of Riesz representer estimators based on score matching. The Riesz representer is a key nuisance component in debiased machine learning, enabling $\sqrt{n}$-consistent and asymptotically efficient estimation of causal and structural targets via Neyman-orthogonal scores. We formulate Riesz representer estimation as a score estimation problem. This perspective stabilizes representer estimation by allowing us to leverage denoising score matching and telescoping density ratio estimation. We also introduce the policy path, a parameter that captures how policy effects evolve under continuous treatments. We show that the policy path can be estimated via score matching by smoothly connecting average marginal effect (AME) and average policy effect (APE) estimation, which improves the interpretability of policy effects.

Applications · Social Sciences

Haofei Yu, Yining Zhao, Guanyu Lin, Jiaxuan You

Understanding and predicting how social beliefs evolve in response to events—from policy changes to scientific breakthroughs—remains a fundamental challenge in social science. Given LLMs’ commonsense knowledge and social intelligence, we ask: *Can LLMs model the dynamics of social beliefs following social events?* In this work, we introduce the concept of the Social World Model (SWM), a general framework designed to capture how social beliefs evolve in response to major events. SWM learns state-transition functions for social beliefs by mining temporal patterns in social data and optimizing evidence lower bound, without the need for explicit human annotations that link events to belief shifts or expensive census data. To evaluate SWM, we introduce a benchmark, SWM-Bench, derived from real-world prediction market data from both Kalshi and Polymarket. SWM-Bench includes over 10k datapoints for social belief prediction tasks spanning diverse domains such as politics, sports, cryptocurrency, and elections. Our experimental results show that SWM significantly outperforms time-series foundation models, achieving RMSE reductions of 8.4% and 11.2% on Polymarket and Kalshi respectively, while offering interpretable insights into the underlying mechanisms of social belief dynamics.

Applications · Chemistry, Physics, and Earth Sciences

Li Sun, Hongbo Lv, Zhikai Jiang, Zhongtian Sun, Lanxu Yang, Philip Yu

Partial Differential Equations (PDEs) play a fundamental role in scientific computing, and recent efforts have sought to extend the success of foundation models to PDE solving. However, multi-physics PDE pre-training faces the unique challenge of disentangling dynamic heterogeneity to learn universal, elementary patterns that generalize to new PDEs. Additionally, cross-physics transfer lacks a theoretical framework for interpretability—specifically, understanding which pre-trained operator knowledge is effectively transferred to target PDEs. To bridge these gaps, we introduce the theory of neural operator splitting, which decomposes PDE evolution into a modulated global spectral operator and sparse local constitutive mechanisms. A key innovation is Origo, which provides a neural operator bank that enables the identification of operator-level generalization patterns. Extensive experiments demonstrate strong zero-shot generalization and mechanism-level interpretability on unseen PDEs.

Applications · Time Series

Binqing Wu, Jian Zhou, Zongjiang Shang, Ling Chen

Spatial–temporal time series forecasting is challenging due to complex lead–lag dependencies, which are often ignored or inadequately modeled by existing methods. Thus, we propose LagLLM, the first LLM-empowered framework that explicitly models lead–lag dependencies by unifying data-driven dynamics modeling and knowledge-driven semantic reasoning. Specifically, LagLLM constructs a lead–lag graph by integrating learnable embeddings, spatial proximity, and prompt-guided reasoning from a frozen LLM, which can capture lead-lag dependencies informed by underlying data structure and semantic knowledge. In addition, LagLLM introduces structural token sorting based on the graph, which can make a fine-turned LLM explicitly perceive directional and delayed interactions. Experiments on eight real-world datasets show that LagLLM achieves the state-of-the-art performance with improved accuracy, robustness, and interpretability. The code is available at https://anonymous.4open.science/r/LagLLM.

Applications · Social Sciences

Caroline L Wang, Daniel Kasenberg, Kimberly Stachenfeld, Pablo Samuel Castro

As Large Language Models (LLMs) are increasingly deployed in social and strategic scenarios, it becomes critical to understand where and why their behavior diverges from that of humans. While behavioral game theory (BGT) provides a framework for analyzing behavior, existing models do not fully capture the idiosyncratic behavior of humans or black-box, non-human agents like LLMs. We employ AlphaEvolve, a cutting-edge program discovery tool, to directly discover *interpretable* models of human and LLM behavior from data, thereby enabling open-ended discovery of structural factors driving human and LLM behavior. Our analysis on iterated rock-paper-scissors reveals that frontier LLMs can be capable of deeper strategic behavior than humans. These results provide a foundation for understanding structural differences driving differences in human and LLM behavior in strategic interactions.

Theory · Everything Else

Suvrat Raju, Praneeth Kumar Netrapalli

We study the error rate of LLMs on tasks like arithmetic that require a deterministic output, and repetitive processing of tokens drawn from a small set of alternatives. We argue that incorrect predictions arise when small errors in the attention mechanism accumulate to cross a threshold, and use this insight to derive a quantitative two-parameter relationship between the accuracy and the complexity of the task. The two parameters vary with the prompt and the model; they can be interpreted in terms of an elementary noise rate, and the number of plausible erroneous tokens that can be predicted. Our analysis is inspired by an "effective field theory'' perspective: the LLM's many raw parameters can be reorganized into just two parameters that govern the error rate. We perform extensive empirical tests, using Gemini 2.5 Flash, Gemini 2.5 Pro and DeepSeek R1, and find excellent agreement between the predicted and observed accuracy for a variety of tasks, although we also identify deviations in some cases. Our model provides an alternative to suggestions that errors made by LLMs on long repetitive tasks indicate the "collapse of reasoning'', or an inability to express "compositional'' functions. Finally, we show how to construct prompts to reduce the error rate.

Applications · Health / Medicine

Maria Emilia Russo, Federico Di Valerio, Alessia Borghini, Alessio Ragno, Roberto Capobianco

Computational approaches have become central to Protein–Protein Interaction (PPI) research, complementing experimental techniques that remain costly and incomplete. While modern deep learning methods capture diverse biological signals and hold promise in expanding the known interactome, empirical validation remains a critical bottleneck due to its long and expensive procedures. To address this challenge, we introduce the problem of PPI candidate ranking, aiming to prioritize interactions for experimental testing. We propose a novel framework that leverages domain knowledge through interpretability-guided ranking and further refines prioritization by integrating complementary sources of evidence, including interaction scores, structural plausibility, and biomedical language features. Evaluations on a large-scale dataset constructed from successive STRING releases demonstrate that our approach yields significant improvements over two state-of-the-art PPI prediction models, providing more accurate and biologically coherent rankings.

Aryaman Arora, Zhengxuan Wu, Jacob Steinhardt, Sarah Schwettmann

The high-level concepts that a neural network uses to perform computation need not be aligned to individual neurons (Smolensky, 1986). Language model interpretability research has thus turned to techniques such as *sparse autoencoders* (SAEs) to decompose the neuron basis into more interpretable units of model computation, for tasks such as *circuit tracing*. However, not all neuron-based representations are uninterpretable. For the first time, we empirically show that **MLP neurons are as sparse a feature basis as SAEs**. We use this finding to develop an end-to-end pipeline for circuit tracing on the MLP neuron basis, which locates causal circuitry on a variety of tasks using gradient-based attribution. On a standard subject-verb agreement benchmark (Marks et al., 2025), a circuit of $\approx 10^2$ MLP neurons is enough to control model behaviour. On the multi-hop city $\to$ state $\to$ capital task from Lindsey et al., 2025, we find a circuit in which small sets of neurons encode specific latent reasoning steps (e.g. 'map city to its state'), and can be steered to change the model's output. This work thus advances automated interpretability of language models without additional training costs.

General Machine Learning · Data

Nick Jiang, Xiaoqing Sun, Lisa Dunlap, Lewis Smith, Neel Nanda

Analyzing large-scale text corpora is a core challenge in machine learning, crucial for tasks like identifying undesirable model behaviors. Current methods often rely on costly LLM-based techniques (e.g. annotating dataset differences) or dense embedding models (e.g. for clustering), which lack control over the properties of interest. We propose using sparse autoencoders (SAEs) to create *SAE embeddings*: representations whose dimensions map to interpretable concepts. Through four data analysis tasks, we show that SAE embeddings are more cost-effective and reliable than LLMs and offer the controllability that dense embeddings lack. Using the large hypothesis space of SAEs, we can uncover insights such as (1) semantic differences between datasets and (2) unexpected concept correlations in documents. For instance, by comparing model responses, we find that Grok-4 clarifies ambiguities more often than nine other frontier models. Relative to LLMs, SAE embeddings uncover bigger differences at 2-8x lower cost and identify biases more reliably. Additionally, SAE embeddings are controllable: by filtering concepts, we can (3) cluster documents along axes of interest and (4) outperform dense embeddings on property-based retrieval. Using SAE embeddings, we study model behavior with two case studies: investigating how OpenAI model behavior has changed over time and finding "trigger" phrases learned by Tulu-3 (Lambert et. al., 2024) from its training data. These results position SAEs as a versatile tool for unstructured data analysis and highlight the neglected importance of interpreting models through their *data*.

Peter Potaptchik, Lee Kit, Michael Albergo

We propose a simple, scalable algorithm based on stochastic interpolants for sampling from unnormalized densities and for fine-tuning generative models. The approach, Tilt Matching, arises from a dynamical equation relating the flow matching velocity to one targeting the same distribution tilted by a reward, implicitly solving a stochastic optimal control problem. The resulting velocity inherits the regularity of stochastic interpolant transports while minimizing an objective with strictly lower variance than flow matching itself. The update to the velocity field can be interpreted as the sum of all joint cumulants between the interpolant velocity and the reward, and to first order is their covariance. The method requires neither reward gradients nor backpropagation through trajectories of the flow or diffusion. We empirically demonstrate that the approach is efficient and highly scalable, providing state-of-the-art results on sampling under Lennard-Jones systems and competitive performance for fine-tuning Stable Diffusion, without requiring reward multipliers. The framework also applies directly to tilting few-step flow map models.

Deep Learning · Large Language Models

Raphael Bernas, Fanny Jourdan, Antonin Poché, Céline Hudelot

Since their introduction, Transformer architectures have dominated Natural Language Processing (NLP). However, recent research has highlighted an inherent anisotropy phenomenon in these models, presenting a significant challenge to their geometric interpretation. Previous theoretical studies on this phenomenon are rarely based on the underlying representation geometry. In this paper, we extend them by providing such theoretical arguments assessing the problematic nature of this phenomenon. Furthermore, to observe geometric internal model dynamics, we apply mechanistic interpretability (MI) techniques during the model's training checkpoints rather than post-hoc, as it is commonly done in the literature. By analyzing multiple models and their checkpoints -including EuroBERT, the Pythia suite, and SmolLM2- we investigate the structure of embedding representations and their correlation with the on manifold entropy of their underlying distribution.