论文检索

Deep Learning · Generative Models and Autoencoders

Ruchi Sandilya, Sumaira Perez, Charles Lynch, Lindsay Victoria, Benjamin Zebley, Derrick Buchanan, Mahendra Bhati, Nolan Williams, Timothy Spellman, FAITH GUNNING 等

Diffusion models excel at generation, but their latent spaces are high dimensional and not explicitly organized for interpretation or control. We introduce ConDA (Contrastive Diffusion Alignment), a plug-and-play geometry layer that applies contrastive learning to pretrained diffusion latents using auxiliary variables (e.g., time, stimulation parameters, facial action units). ConDA learns a low-dimensional embedding whose directions align with underlying dynamical factors, consistent with recent contrastive learning results on structured and disentangled representations. In this embedding, simple nonlinear trajectories support smooth interpolation, extrapolation, and counterfactual editing while rendering remains in the original diffusion space. ConDA separates editing and rendering by lifting embedding trajectories back to diffusion latents with a neighborhood-preserving kNN decoder and is robust across inversion solvers. Across fluid dynamics, neural calcium imaging, therapeutic neurostimulation, facial expression dynamics, and monkey motor cortex activity, ConDA yields more interpretable and controllable latent structure than linear traversals and conditioning-based baselines, indicating that diffusion latents encode dynamics-relevant structure that can be exploited by an explicit contrastive geometry layer.

Optimization · Non-Convex

Qiujing Lu, Tonmoy Monsoor, Ehsan Ebrahimzadeh, Kartik Sharma, Vwani Roychowdhury

Nonnegative matrix factorization (NMF) seeks a low-rank approximation $X \approx UV^T$ with nonnegative factors and is commonly solved using *interior* methods that enforce feasibility throughout optimization. We show that such constraint-driven approaches can impede progress in the nonconvex landscape, leading to slow convergence or convergence to suboptimal stationary points. We propose an *exterior* framework for NMF (eNMF) that separates low-rank approximation from nonnegativity enforcement. Our method initializes from the optimal unconstrained factorization and introduces a rotation procedure that maps unconstrained factors to an exterior point closest to the nonnegative orthant. This viewpoint yields an algorithmic framework in which simple iterative updates converge to KKT-satisfying stationary points on the boundary of the positive orthant. The exterior formulation also enables a geometric interpretation of NMF solutions, clarifying equivalence classes of factorizations under permutation and orthogonal transformations. An intriguing numerical result, involving 400 NMF experiments across both real and synthetic datasets, show that in 99\% of the cases, different algorithms tend to converge towards equivalent factor matrices. We benchmark eNMF against 9 state-of-the-art NMF algorithms with 9 initialization schemes across 3 real-world and 2 synthetic datasets. eNMF consistently outperforms all 81 competitors, achieving up to 30\% lower reconstruction error under equal-time settings and up to 150\% speedup under equal-error settings. The downstream experiments further demonstrate substantial performance gains in audio processing and recommendation tasks, corroborating the practical benefits of the proposed exterior optimization framework. Anonymized code is available at https://anonymous.4open.science/r/eNMF-6240/README.md

Social Aspects · Accountability, Transparency, and Interpretability

David Boetius, Shahaf Bassan, Guy Katz, Stefan Leue, Tobias Sutter

Shapley additive explanations (SHAP) are widely recognised as computationally intractable for neural networks, since they induce an exponential search space over the input features. In this work, we take a first step towards scaling exact SHAP computation to larger search spaces by introducing an algorithm that leverages recent advances in neural network verification to compute arbitrarily tight exact lower and upper bounds on SHAP values for neural networks, ultimately recovering the exact SHAP values. We demonstrate that our approach scales to orders of magnitude larger search spaces than state-of-the-art exact methods. This provides an important first step towards exact SHAP computation and establishes a principled cornerstone for evaluating statistical approximation methods on larger search spaces.

Deep Learning · Generative Models and Autoencoders

Xinyi Shang, Peng Sun, Jingyu Lin, Zhiqiang Shen

Analytical diffusion models offer a mathematically transparent path to generative modeling by formulating the denoising score as an empirical-Bayes posterior mean. However, this interpretability comes at a prohibitive cost: the standard formulation necessitates a full-dataset scan at every timestep, scaling linearly with dataset size. In this work, we present the first systematic study addressing this scalability bottleneck. We challenge the prevailing assumption that the entire training data is necessary, uncovering the phenomenon of \textit{Posterior Progressive Concentration}: the effective golden support of the denoising score is not static but shrinks asymptotically from the global manifold to a local neighborhood as the signal-to-noise ratio increases. Capitalizing on this, we propose \textit{Dynamic Time-Aware Golden Subset Diffusion} (\algopt), a training-free framework that decouples inference complexity from dataset size. Instead of static retrieval, \algopt uses a coarse-to-fine mechanism to dynamically pinpoint the ``Golden Subset'' for inference. Theoretically, we derive rigorous bounds guaranteeing that our sparse approximation converges to the exact score. Empirically, \algopt achieves a $\bf 71 \times$ speedup on AFHQ while matching or achieving even better performance than full-scan baselines. Most notably, we demonstrate the first successful scaling of analytical diffusion to ImageNet-1K, unlocking a scalable, training-free paradigm for large-scale generative modeling.

Optimization · Zero-order and Black-box Optimization

Sijia Liu, Yicheng Lang, Soumyadeep Pal, Changsheng Wang, Yancheng Huang, Chongyu Fan, James Diffenderfer, Bhavya Kailkhura, Yihua Zhang

Zeroth-order (ZO) optimization, learning from finite differences of function evaluations without backpropagation, has recently regained attention in deep learning due to its memory efficiency and applicability to gray- or black-box pipelines. Yet, ZO methods are often dismissed as fundamentally unscalable because of estimator variance and unfavorable query complexity. We argue that this conclusion might be misguided: ZO optimization is underexplored, not underpowered. We show that many perceived limitations stem from myopic development practices, most notably full-space, element-wise, estimator-centric designs. We articulate six positions spanning the algorithmic, systems, and evaluation stack. First, we revisit the feasibility boundaries of estimator-centric ZO methods through variance control, variance–query tradeoffs, and directional-derivative lenses. Then, we identify three underexplored opportunities: (i) subspace and spectral views of ZO that enable interpretable variance reduction with graceful query scaling, (ii) the forward-only nature of ZO as a systems advantage for communication-efficient, pipeline-friendly, and resource-constrained training, and (iii) the need to de-obfuscate ZO evaluations from task complexity. We strongly advocate rethinking ZO optimization around its unique strengths and acting accordingly, opening a viable path toward large-scale, system-aware, and resource-efficient learning with ZO optimization.

Deep Learning · Large Language Models

Yiting Lu, Fengbin Guan, Yixin Gao, Yan Zhong, Xinge Peng, Jiakang Yuan, Yihao Liu, Bo Zhang, Xin Li, Zhibo Chen 等

Image quality assessment (IQA) is inherently multi-mage quality assessment (IQA) is inherently multi-dimensional, yet existing reward models are typically limited to a single task and become unstable when extended to multi-task settings. In particular, heterogeneous reward scales and variances across tasks can lead to conflicting optimization signals during reinforcement learning. We propose VisualScore, a unified visual evaluation framework that formulates multi-task IQA as structured, task-aware reasoning followed by continuous reward optimization. VisualScore produces interpretable rationales together with scalar quality scores under explicit evaluation principles. We construct a reasoning-enhanced reward modeling dataset via rejection sampling and initialize the model through supervised fine-tuning. VisualScore is then optimized with Group Relative Policy Optimization (GRPO) using a Gaussian-based continuous reward. To address multi-task reward conflicts and stabilize training, we introduce standard deviation filtering and entropy gating to normalize task-wise reward signals and suppress noisy updates. Experiments on technical quality, aesthetic quality, and text–image alignment show that VisualScore improves robustness, generalization, and interpretability, and can effectively guide text-to-image generation at test time without retraining.

General Machine Learning · Unsupervised and Semi-supervised Learning

Dazhi Fu, Zhao Zhang, Jicong Fan

Density-based anomaly detection methods often provide accurate and interpretable predictions but their performance can be severely degraded by the inherent noise of data, such as changes arising from environmental conditions during data collection or background noise. To deal with such noise, we present noise-robust density estimation (NRDE) for tabular data anomaly detection. We aim to estimate the density of pure data with the influence of noises isolated, which is a non-trivial task since the data-generating process is completely unknown. Specifically, NRDE learns a Jacobian-regularized normalizing flow to estimate the sources of data and categorizes sources into two groups, where one group generates pure data and the other generates noise. After generating pure data, we can use the density of such pure data to detect anomalies caused by the sources of pure data solely. Therefore, NRDE is robust to inherent noise. We provide theoretical results to support the effectiveness of NRDE and compare NRDE with $17$ baselines on $47$ benchmark datasets under different settings, including vanilla anomaly detection, anomaly detection with anomaly contamination, anomaly detection on noisy data, and transductive outlier detection.

Deep Learning · Large Language Models

Yuxin Xiao, Zhen Huang, Wenxiao Wang, Yan Zhao, Zhihong Gu, Binbin Lin, Xiaofei He, Xu Shen, Jieping Ye

Unintended code-switching, which refers to the phenomenon where LLM unexpectedly switch languages, poses a fundamental challenge in the multilingual capabilities in LLMs. However, we still lack a mechanistic account of how this failure mode is implemented inside the model. For example, what internal components (i.e., circuits) give rise to unintended code-switching, where they emerge across layers, and how we can intervene to mitigate it. In this work, we introduce a scalable circuit discovery framework that causally localizes multilingual neurons and describes their functional patterns, then further groups them into interpretable circuits---without any additional training or manual annotation. Our findings lie in two folds: a) The model's ``speaking-a-language'' circuit decomposes into a language regime (detecting and maintaining language identity) and a semantic regime (retrieving language-agnostic semantics). b) The mechanism of unintended code-switching is a regime shift. Semantic regime suppresses the language regime, and overwhelms the multilingual circuit, leading the model to speak in unintended language. To validate these findings, we further fine-tune the identified language sub-circuit, reducing the code-switching rate by $20.8\%$ with minimal parameter updates ($\sim0.019\%$ of all neurons). This work serves as a preliminary exploration of multilingual generation mechanism, offering actionable insight for targeted training for multilingual LLMs.

General Machine Learning · Causality

Haorui Ma, Dennis Frauen, Stefan Feuerriegel

Structural nested mean models (SNMMs) are a principled approach to estimate the treatment effects over time. A particular strength of SNMMs is to break the joint effect of treatment sequences over time into localized, time-specific ``blip effects''. This decomposition promotes interpretability through the incremental effects and enables the efficient offline evaluation of optimal treatment policies without re-computation. However, neural frameworks for SNMMs are lacking, as their inherently sequential g-estimation scheme prevents end-to-end, gradient-based training. Here, we propose DeepBlip, the first neural framework for SNMMs, which overcomes this limitation with a novel double optimization trick to enable simultaneous learning of all blip functions. Our DeepBlip seamlessly integrates sequential neural networks like LSTMs or transformers to capture complex temporal dependencies. By design, our method correctly adjusts for time-varying confounding to produce unbiased estimates, and its Neyman-orthogonal loss function ensures robustness to nuisance model misspecification. Finally, we evaluate our DeepBlip across various clinical datasets, where it achieves state-of-the-art performance.

Reinforcement Learning · Batch/Offline

Ritam Majumdar, Finale Doshi-Velez, Sonali Parbhoo

Off-policy evaluation (OPE) is essential for deploying reinforcement learning in safety-critical settings, yet existing estimators such as importance sampling and doubly robust (DR) often exhibit prohibitively high variance when rewards are sparse. In this work, we introduce Reward-Shaping Control Variates, a new family of unbiased estimators that leverage potential-based reward shaping to construct additional zero-mean control variates. We prove that shaped estimators always yields valid variance reduction, and that combining shaping-based and Q-based control variates strictly expands the variance-reduction subspace beyond DR and its minimax variant MRDR. Empirically, we provide a systematic regime map across synthetic chains, a cancer simulator, 5 single-stock and 1 multi-stock DOW-30 trading environments and an ICU-sepsis benchmark showing that shaping-based OPE consistently outperforms DR in sparse-reward settings, while a hybrid estimator achieves state-of-the-art performance across sparse, noisy, and misspecified environments. Our results highlight reward shaping as a powerful and interpretable tool for robust OPE, offering both theoretical guarantees and practical improvements in domains where standard estimators fail.

General Machine Learning · Clustering

Kenneth Clarkson, Lior Horesh, Takuya Ito, Charlotte Park, Parikshit Ram

Although transformers are most commonly optimized as statistical sequence models, it is unclear to what extent they can implement and learn exact algorithmic computations. Here, we specify a transformer implementation from first principles that executes a fundamental and widely used method for $k$-means clustering: Lloyd's algorithm. We theoretically prove and empirically demonstrate that this implementation of a transformer architecture, which we term the _$k$-means transformer_, exactly implements Lloyd's algorithm for $k$-means clustering using the standard circuit mechanisms of modern transformers: attention block, residual connections, and feed-forward block. In learning experiments, we find that training this base architecture on $k$-means clustering yields a generalizable clustering algorithm that surpasses Lloyd's algorithm in terms of clustering quality. Finally, we demonstrate that interpretable alterations (e.g., inclusion of layer normalizations) to this architecture yields diverse and novel variants of clustering algorithms, including soft $k$-means, spherical $k$-means, trimmed $k$-means. Overall, our results show that transformer circuit mechanisms can instantiate exact algorithmic routines for clustering, while simultaneously providing an effective learnable model.

Reinforcement Learning · Everything Else

Yuetian Wang, Dianxi Shi, Yuanze Wang, Huanhuan Yang, Shiming Song, Chunping Qiu

Safe reinforcement learning (Safe RL) seeks to optimize long-term performance while ensuring adherence to safety constraints. However, most existing approaches address safety in a simplified manner, typically by linearly combining rewards and costs, which provides limited guidance when safety and performance interact in complex, nonlinear ways. We present USB-RL (Unsupervised Score-Balanced Reinforcement Learning), a model-based framework that learns implicit safety–performance preferences directly from experience. Our approach infers a monotone partial-order score through unsupervised pairwise comparisons of long-horizon outcomes, capturing nuanced trade-offs without relying on manually tuned cost weights. The learned score guides model-based policy optimization by dynamically balancing safety and performance, enabling flexible and adaptive multi-step planning in imagination-based control. Across diverse safety benchmarks, USB-RL achieves strong returns while substantially reducing safety violations, demonstrating stable and interpretable safety–performance trade-offs.

Deep Learning · Large Language Models

Haiyu Wang, Yuanyuan Lin

Recent studies have revealed two intriguing phenomena in large language models: massive activations, characterized by a small number of activations exhibiting abnormally large magnitudes, and attention sink, where a disproportionate amount of attention is consistently allocated to specific tokens regardless of their semantic relevance. However, the co-emergence and co-existence of these two phenomena remain poorly understood. In this work, we revisit the prevailing view that massive activations are the primary mechanism responsible for concentrating attention on sink tokens, and provide a more nuanced interpretation of their relationship. Through both theoretical analysis and empirical evidence, we demonstrate that massive activations and attention sink jointly act to prevent excessive token mixing in self-attention. Specifically, attention sink suppresses mixing among non-sink tokens, whereas massive activations suppress mixing between sink tokens and non-sink tokens. Furthermore, our theory provides a principled explanation of how the location of massive activations depends on the placement of layer normalization, and why KV-biases and gating mechanisms can remove massive activations while largely preserving attention sink. We further conduct intervention analyses and find that removing the value vector of the sink token can recover attention sink even when massive activations are entirely suppressed. Overall, this work provides a mechanistic perspective on how massive activations and attention sink interact under normalization and self-attention, offering new insights into their functional roles in Transformer models.

Applications · Chemistry, Physics, and Earth Sciences

Gnankan Landry Regis N'guessan, Bum Jun Kim

Physical systems near singularities, interfaces, and critical points exhibit power-law scaling, yet standard neural networks leave the governing exponents implicit. We introduce physics-informed M\"untz-Sz\'asz Networks (MSN-PINN), a power-law basis network that treats scaling exponents as trainable parameters. The model outputs both the solution and its scaling structure. We prove identifiability, or unique recovery, and show that, under these conditions, the squared error between learned and true exponents scales as $O(|\mu - \alpha|^2)$. Across experiments, MSN-PINN achieves single-exponent recovery with 1--5\% error under noise and sparse sampling. It recovers corner singularity exponents for the two-dimensional Laplace equation with 0.009\% error, matches the classical result of Kondrat'ev (1967), and recovers forcing-induced exponents in singular Poisson problems with 0.03\% and 0.05\% errors. On a 40-configuration wedge benchmark, it reaches a 100\% success rate with 0.022\% mean error. Constraint-aware training encodes physical requirements such as boundary condition compatibility and improves accuracy by three orders of magnitude over naive training. By combining the expressiveness of neural networks with the interpretability of asymptotic analysis, MSN-PINN produces learned parameters with direct physical meaning.

Social Aspects · Accountability, Transparency, and Interpretability

Guanzheng Qin, Chenghao Sun, Zhining Xie, Xinmei Tian

Large Language Models (LLMs) demonstrate remarkable potential in role-playing tasks but frequently suffer from personality decay—termed "Out-of-Character" (OOC) behavior—during prolonged interactions. While heuristic strategies exist to align model behaviors, the internal computational dynamics driving personality expression remain opaque. A fundamental barrier to decoding these mechanisms is a *metric gap*: while standard causal attribution paradigms target atomic, single-token outcomes, personality manifests as a holistic, multi-token behavioral tendency. We bridge this gap via the *Latent Persona Vector*, a differentiable proxy enabling the first fine-grained causal tracing of personality circuits. This metric reveals a structured "Preparation-Establishment-Expression" dynamic and identifies the mechanistic root of OOC behavior not as knowledge erasure, but as *generic prior dominance*. Specifically, we find that intrinsic assistant priors suppress emergent persona intents during the critical "Establishment" phase. Guided by this diagnosis, we propose surgically recalibrating the signal magnitude in fewer than 5% of attention heads. This targeted intervention effectively counteracts prior suppression, significantly restoring character consistency while preserving general reasoning capabilities.

Applications · Chemistry, Physics, and Earth Sciences

Chuyang Xiang, Yichen Wei, Junchi Yan

Symbolic regression (SR) aims to discover interpretable mathematical expressions from observed data. While recent generative approaches have shown promise in treating SR as machine translation or multimodal learning tasks using NN methods, they suffer from a fundamental limitation: training-evaluation misalignment. The training objectives (average cross-entropy loss on a token level across the distribution of historical data) differ from the evaluation metric (fitting error for every test data / complexity), necessitating extensive heuristic post-processing and constant optimization. On the other hand, direct optimization methods suffer from curse of dimensionality, non-differentiability and local optima traps. We propose MOD-SR, unifying multimodal distribution learning during training with direct optimization at inference time. This is achieved by modeling the task as $p(x_0 \mid \mathcal{D}, y^*)$ and employing gradient-guided diffusion in embedding space, enhanced by contrastive learning and representation alignment. Furthermore, we introduce DFEX, a fixed-depth tree relaxation method that ensures differentiability for effective gradient guidance during inference. Experiments demonstrate significant improvements over existing methods, achieving superior performance on diverse benchmarks through a unified framework integrating distribution learning and optimization.

Deep Learning · Large Language Models

Akanksha Narula, Mofasshara Rafique, Laurent Bindschaedler

Large language models (LLMs) now generate substantial production code, often for tasks with multiple valid algorithmic solutions. The hidden risk is that incidental prompt cues can steer \emph{which} algorithm is selected, even when all outputs pass the same tests. Prompt sensitivity is well studied as a tool to improve output quality, but we instead examine output policy: algorithm choice under fixed correctness. We define algorithm steering and run 55{,}545 controlled experiments across 11 tasks, 19 cue types (18 channels plus a memoization ablation), and 15 models. We find large, interpretable shifts in algorithm-family distributions (up to 100 percentage points, pp), including on applied tasks such as rate limiting, yielding an ``invisible lottery'' in which accidental context alters performance, security, and maintainability.

YiZhen Wang, Zheng Wang, EUN-HU KIM, Zunwei Fu

Most existing time series forecasting models are trained with backpropagation, which often brings high computational cost and limited transparency, so it can be hard to understand why a model makes a given prediction. This paper presents FIPN, a forward self-organizing interpretable polynomial network for time series forecasting. FIPN grows its architecture layer by layer and avoids backpropagation. Each neuron couples a fuzzy-rule antecedent with a Fourier-enhanced polynomial consequent: fuzzy clustering softly partitions the input space and produces interpretable rule weights for local regimes, while the consequent operates directly on the original features and uses Fourier functions to capture periodic and frequency-related structure. Forward growth can lead to redundancy, collinearity, and overfitting as depth increases, so FIPN introduces regularized node scoring, node-level dropout, and persistent access to raw inputs at every layer to stabilize closed-form estimation and improve generalization. Experiments on long-horizon forecasting benchmarks show that FIPN achieves competitive accuracy with a compact model size, and the learned fuzzy rules provide consistent, structure-based explanations. These results suggest that forward self-organizing polynomial networks offer a practical balance among accuracy, efficiency, and interpretability for long-term time series forecasting.

Probabilistic Methods · Bayesian Models and Methods

Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Minimum Bayes Risk (MBR) decoding yields more robust and higher-quality text generation than maximum a posteriori (MAP) decoding by selecting hypotheses that maximize expected utility over sampled pseudo-references. However, there exists discrepancy in the design: hypothesis selection calculates expected utility scores conditioned on given pseudo-references, while commonly used evaluation metrics, e.g., BLEU and COMET, are asymmetric. Therefore, it is important to consider both hypothesis-to-reference and reference-to-hypothesis directional effects. In this study, we introduce a noisy channel decomposition of MBR decoding that naturally incorporates bidirectional effects to account for these asymmetries. We decompose MBR decoding into four interacting components: hypothesis-to-reference likelihood, reference-to-hypothesis likelihood, hypothesis prior, and reference prior. This decomposition provides a unified interpretation of existing MBR variants and enables metric- and task-specific interpretability by isolating the contribution of each channel. Furthermore, our comprehensive analysis demonstrates that appropriate channel weighting consistently yields performance gains over original MBR decoding across tasks and utility functions.

Han Wang, Weijie Wang, Jiaqi Liu, Hilde Kuehne, Nicu Sebe

Interpreting residual architectures as discretizations of Ordinary Differential Equations characterizes the forward pass as a continuous latent flow. Although this framework defines the mechanics of inference, conventional training paradigms primarily constrain the terminal state, leaving the intermediate evolution unregulated. In this work, we formalize the forward pass as a Conditional Discriminative Flow and investigate its intrinsic kinematic laws. Using Tuned Lens analysis, we discover that the accumulation of semantic evidence follows a consistent exponential schedule. This finding confirms that deep models naturally require an extended phase of feature construction prior to a rapid transition toward categorical certainty in the terminal layers. Standard training ignores this latent progression. To resolve this impedance mismatch, we propose Deep Trajectory Supervision, a framework that aligns auxiliary supervision with this intrinsic exponential bias. By rectifying the trajectory of the inference flow, DTS functions as a critical physical inductive bias. Empirical evaluations on ImageNet-1K and various benchmarks demonstrate that DTS significantly accelerates convergence and improves terminal performance.