论文检索

Probabilistic Methods · Bayesian Models and Methods

Rajeev Yasarla, Shizhong Han, Hsin-Pai Cheng, Apratim Bhattacharyya, Shweta Mahajan, Litian Liu, Yunxiao Shi, Risheek Garrepalli, Hong Cai, Fatih Porikli

End-to-end (E2E) autonomous driving has recently emerged as a new paradigm, offering significant potential. However, few studies have looked into the practical challenge of deployment across domains (e.g., cities). Although several works have incorporated Large Language Models (LLMs) to leverage their open-world knowledge, LLMs do not guarantee cross-domain driving performance and may incur prohibitive retraining costs during domain adaptation. In this paper, we propose RoCA, a novel framework for robust cross-domain E2E autonomous driving. RoCA formulates the joint probabilistic distribution over the tokens that encode ego and surrounding vehicle information in the E2E pipeline. Instantiating with a Gaussian process (GP), RoCA learns a set of basis tokens with corresponding trajectories, which span diverse driving scenarios. Then, given any driving scene, it is able to probabilistically infer the future trajectory. By using RoCA together with a base E2E model in source-domain training, we improve the generalizability of the base model, without requiring extra inference computation. In addition, RoCA enables robust adaptation on new target domains, significantly outperforming direct finetuning. We extensively evaluate RoCA on various cross-domain scenarios and show that it achieves strong domain generalization and adaptation performance.

Hamish Flynn, Joe Watson, Ingmar Posner, Jan Peters

We analyze the Bayesian regret of the Gaussian process posterior sampling reinforcement learning (GP-PSRL) algorithm. Posterior sampling is an effective heuristic for decision-making under uncertainty that has been used to develop successful algorithms for a variety of continuous control problems. However, theoretical work on GP-PSRL is limited. All known regret bounds either fail to achieve a tight dependence on a kernel-dependent quantity called the maximum information gain or fail to properly account for the fact that the set of possible system states is unbounded. Through a recursive application of the Borell-Tsirelson-Ibragimov-Sudakov inequality, we show that, with high probability, the states actually visited by the algorithm are contained within a ball of near-constant radius. To obtain tight dependence on the maximum information gain, we use the chaining method to control the regret suffered by GP-PSRL. Our main result is a Bayesian regret bound of the order $\widetilde{\mathcal{O}}(H^{3/2}\sqrt{\gamma_{T/H} T})$, where $H$ is the horizon, $T$ is the number of time steps and $\gamma_{T/H}$ is the maximum information gain. With this result, we resolve the limitations with prior theoretical work on PSRL, and provide the theoretical foundation and tools for analyzing PSRL in complex settings.

Probabilistic Methods · Gaussian Processes

Jihao Andreas Lin, Sebastian Ament, Louis Tiao, David Eriksson, Maximilian Balandat, Eytan Bakshy

Gaussian processes (GPs) are powerful and widely used probabilistic regression models, but their effectiveness in practice is often limited by the choice of kernel function. This kernel function is typically handcrafted from a small set of standard functions, a process that requires expert knowledge, results in limited adaptivity to data, and imposes strong assumptions on the hypothesis space. We study Empirical GPs, a principled framework for constructing flexible, data-driven GP priors that overcome these limitations. Rather than relying on standard parametric kernels, we estimate the mean and covariance functions empirically from a corpus of historical observations, enabling the prior to reflect rich, non-trivial covariance structures present in the data. Theoretically, we show that the resulting model converges to the GP that is closest (in KL-divergence sense) to the real data generating process. Practically, we formulate the problem of learning the GP prior from independent datasets as likelihood estimation and derive an Expectation-Maximization algorithm with closed-form updates, allowing the model handle heterogeneous observation locations across datasets. We demonstrate that Empirical GPs achieve competitive performance on learning curve extrapolation and time series forecasting benchmarks.

Optimization · Zero-order and Black-box Optimization

Yan Zhang, Xuefeng Liu, Sipeng Chen, Sascha Ranftl, Chong Liu, Shibo Li

Standard Bayesian Optimization (BO) assumes uniform smoothness across the search space—an assumption violated in multi-regime problems such as molecular conformation search through distinct energy basins or drug discovery across heterogeneous molecular scaffolds. A single GP either oversmooths sharp transitions or hallucinates noise in smooth regions, yielding miscalibrated uncertainty. We propose RAMBO, a Dirichlet Process Mixture of Gaussian Processes that automatically discovers latent regimes during optimization, each modeled by an independent GP with locally-optimized hyperparameters. We derive collapsed Gibbs sampling that analytically marginalizes latent functions for efficient inference, and introduce adaptive concentration parameter scheduling for coarse-to-fine regime discovery. Our acquisition functions decompose uncertainty into intra-regime and inter-regime components. Experiments on synthetic benchmarks and real-world applications—including molecular conformer optimization, virtual screening for drug discovery, and fusion reactor design—demonstrate consistent improvements over state-of-the-art baselines on multi-regime objectives.

Probabilistic Methods · Gaussian Processes

Tim Weiland, Philipp Hennig

Nonlinear conservation laws are at the heart of many of the most important dynamical systems in science and engineering. In practical applications, such systems are often subject to various sources of uncertainty, e.g. due to sparse or noisy measurements. Inferring physical quantities and fields of interest then becomes an ill-posed problem which both classical numerical methods and modern deep learning-based methods struggle to treat appropriately. Recent work has framed classical numerical methods as Bayesian inference under Gaussian process priors, resulting in a physics-aware treatment of uncertainties. Following this line of work, we develop a novel numerically conservative method for uncertainty-aware simulations of nonlinear conservation laws. Our method uses recent sparse approximation techniques to scale up to large-scale forward and inverse problems. For forward simulation, we match the accuracy of classical solvers while providing structurally meaningful uncertainty. On inverse problems, we recover posteriors over nonparametric source fields in seconds --- outperforming neural baselines that take minutes to produce a less accurate point estimate.

General Machine Learning · Representation Learning

Chankyo Kim, Sicheng Zhao, Minghan Zhu, Tzu-Yuan Lin, Maani Ghaffari

Many scientific and geometric problems exhibit general linear symmetries, yet most equivariant neural networks are built for compact groups or simple vector features, limiting their reuse on matrix-valued data such as covariances, inertias, or shape tensors. We introduce \textbf{Reductive Lie Neurons (ReLNs)}, an exactly $\mathrm{GL}(n)$-equivariant architecture that natively supports matrix-valued and Lie-algebraic features. ReLNs resolve a central stability issue for reductive Lie algebras by introducing a non-degenerate adjoint (conjugation)-invariant bilinear form, enabling principled nonlinear interactions and invariant feature construction in a single architecture that \textit{transfers across subgroups without redesign}. We demonstrate ReLNs on algebraic tasks with $\mathfrak{sl}(3)$ and $\mathfrak{sp}(4)$ symmetries, Lorentz-equivariant particle physics, uncertainty-aware drone state estimation via joint velocity--covariance processing, learning from 3D Gaussian-splat representations, and EMLP double-pendulum benchmark spanning multiple symmetry groups. ReLNs consistently match or outperform strong equivariant and self-supervised baselines while using substantially fewer parameters and compute, improving the accuracy–efficiency trade-off and providing a practical, reusable backbone for learning with broad linear symmetries.

Applications · Everything Else

Yingming Pu, Tao LIN, Hongyu Chen

Large Language Model (LLM)-based scientific agents have accelerated scientific discovery, yet they often suffer from significant inefficiencies due to adherence to fixed initial priors. Existing approaches predominantly operate within a static hypothesis space, which restricts the discovery of novel phenomena, resulting in computational waste when baseline theories fail. To address this, we propose shifting the focus from searching hypotheses to evolving the underlying scientific principles. We present $\textbf{PiEvo}$, a principle-evolvable framework that treats scientific discovery as Bayesian optimization over an expanding principle space. By integrating Information-Directed Hypothesis Selection via Gaussian Process and an anomaly-driven augmentation mechanism, PiEvo enables agents to autonomously refine their theoretical worldview. Evaluation across four benchmarks demonstrates that PiEvo (1) achieves an average solution quality of up to 90.81\%$\sim$93.15\%, representing a 29.7\%$\sim$31.1\% improvement over the state-of-the-art, (2) attains an 83.3\% speedup in convergence step via significantly reduced sample complexity by optimizing the compact principle space, and (3) maintains robust performance across diverse scientific domains and LLM backbones.

Deep Learning · Generative Models and Autoencoders

Boyang Zhang, Zhiguo Wang, Ya-Feng Liu

Score-based diffusion models demonstrate superior performance in generative tasks but encounter fundamental bottlenecks in inverse problems due to the analytical intractability of the time-dependent likelihood score. To bridge this gap, we propose a novel proximal-based generative modeling (PGM) framework that rigorously circumvents explicit likelihood evaluation. Our framework is built upon a theoretical equivalence between Gaussian convolution in diffusion processes and Moreau-Yosida regularization in nonsmooth optimization. This enables a new sampling mechanism driven by the proposed Moreau score, which admits a closed-form expression via proximal operators. Moreover, we introduce Moreau score matching to learn the proximal operators that rely solely on samples drawn from the prior distribution. Theoretically, PGM eliminates the early-stopping bias inherent in the score-based diffusion model and achieves non-asymptotic convergence. Experiments demonstrate that PGM significantly surpasses state-of-the-art methods in reconstruction quality and sampling time.

Deep Learning · Large Language Models

Shu Ding, Yang Peng, Hangan Zhou, Xinyu Lu, Shangwei Chen, Junhua Huang, Mingxuan Yuan, Wei Wang

Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning technique, and previous works have studied the update dynamics of LoRA, showing that updating via the low-rank matrix $\mathbf{A}$ can be viewed as a process within the compressed subspace defined by $\mathbf{A}^{\top} \mathbf{A}$ of the gradient $\nabla f\left(\mathbf{W} \right)$. However, few works analyze how the properties of the low-rank matrices affect the performance of LoRA, since existing methods heuristically initialize the low-rank matrices as Gaussian matrices. In this paper, we provide a theoretical understanding of the update dynamics of LoRA. We reveal that the update dynamics can be viewed as a process within the subspace projected by $\mathbf{A}^{\top} (\mathbf{A} \mathbf{A}^{\top})^{\dagger} \mathbf{A}$, and prove that when the gradient $\nabla f\left(\mathbf{W} \right)$ is unavailable, if $\mathbf{A}$ is an Equiangular Tight Frame (ETF), $\mathbf{A}^{\top} \mathbf{A}$ and $\mathbf{A}^{\top} (\mathbf{A} \mathbf{A}^{\top})^{\dagger} \mathbf{A}$ can preserve the maximum information from the gradient $\nabla f\left(\mathbf{W} \right)$. Thus, initializing $\mathbf{A}$ as an ETF is the optimal solution for low-rank adaptation when the gradient $\nabla f\left(\mathbf{W} \right)$ is unavailable. Furthermore, we establish the convergence of Low-Rank Adaptation with a rate of $\mathcal{O}\left(\frac{1}{T}\right)$ when $\mathbf{A}$ is an ETF. Extensive experiments show that initializing the low-rank matrices as ETFs significantly outperforms the commonly used Gaussian initialization for existing primary LoRA variants.

Theory · Learning Theory

Yuanhe Zhang, Jason Lee, Fanghui Liu

We present the first comprehensive Lean 4 formalization of statistical learning theory (SLT) grounded in empirical process theory. Our end-to-end formal infrastructure implement the missing contents in latest Lean library, including a complete development of Gaussian Lipschitz concentration, the first formalization of Dudley’s entropy integral theorem for sub-Gaussian processes, and an application to least-squares regression with a sharp rate. The project was carried out using a human–AI collaborative workflow, in which humans design proof strategies and AI agents execute tactical proof construction, resulting in approximately 30,000 lines of human-verified Lean 4 code produced over 500 hours of supervised development. Beyond implementation, the formalization process exposes and resolves implicit assumptions and missing details in standard SLT textbooks, enforcing a granular, line-by-line understanding of the theory. This work establishes a reusable formal foundation for future developments in machine learning theory. The code is provided in the supplementary materials.

Deep Learning · Everything Else

Zihan Zhou, Yang Zhou, Ruoming Jin, Pan He, Patrick Emami

Vision–language models such as CLIP embed images and text into a shared space, but still suffer from a modality gap, where image and text features cluster separately and nearest neighbors are dominated by same-modality rather than true cross-modal matches. Existing works alleviate the modality gap by strengthening cross-modal losses, post-processing embeddings or similarities, or imposing geometric regularization, but they primarily enforce global alignment and can distort local geometry, limiting gains in local ranking and zero-shot accuracy. We propose Fractal Spectral Alignment (FSAlign), which reduces the modality gap by shaping and matching the multi-scale geometry of image and text embeddings. By enforcing Ahlfors-regularity and sub-Gaussian heat kernel bounds, FSAlign constructs a shared fractal multi-scale structure for multiple modalities. This structure captures geometry across scales, from local neighborhoods to global structure, and ensures shared fractal spectral geometry across modalities. Based on this structure, we introduce a fractal spectral zeta score derived from multi-scale heat kernels and minimize the discrepancy between pairwise image–text samples to align their multi-scale neighborhoods. We theoretically demonstrate that FSAlign can guarantee the alignment of local spectral measures and global fractional Dirichlet energies.

Applications · Health / Medicine

heng rao, Jason Zhang, Yu Gu, Zhenghao Liu, Ge Yu, Jeffrey Su, Yang Cao, Fan Yang, Minghan Chen

Predicting long-horizon trajectories of biological dynamical systems remains challenging due to substantial system heterogeneity. Most existing machine learning approaches are system-specific, requiring retraining for each new system and exhibiting limited generalization across distinct biological regimes. To address this limitation, we create a large-scale benchmark of over 1,000 ODE-based systems biology models spanning diverse organisms, biological processes, and dynamical behaviors. Building on this benchmark, we propose a regime-aware trajectory prediction framework that enables cross-system generalization and uncertainty quantification for unseen systems. Our approach introduces structured initial states derived from biological regime priors, such as growth trends and oscillatory rhythms, into conditional flow matching, replacing the standard Gaussian source distribution. We provide theoretical justification for this initialization and empirically demonstrate state-of-the-art accuracy (31\% MAE reduction), well-calibrated uncertainty (17\% CRPS improvement), and efficient long-horizon inference across the benchmark.

Probabilistic Methods · Gaussian Processes

Marcus Noack, Mark Risser, HENGRUI LUO, Vardaan Tekriwal, Ronald Pandolfi

Despite a large corpus of recent work on scaling up Gaussian processes, a stubborn trade-off between computational speed, prediction and uncertainty quantification accuracy, and customizability persists. This is because the vast majority of existing methodologies exploit various levels of approximations that lower accuracy and limit the flexibility of kernel and noise-model designs --- an unacceptable drawback at a time when expressive non-stationary kernels are on the rise in many fields. Here, we propose a methodology we term \emph{gp2Scale} that scales exact Gaussian processes to more than 10 million data points without relying on inducing points, kernel interpolation, or neighborhood-based approximations, and instead leveraging the existing capabilities of a GP: its kernel design. Highly flexible, compactly supported, and non-stationary kernels lead to the identification of naturally occurring sparse structure in the covariance matrix, which is then exploited for the calculations of the linear system solution and the log-determinant for training. We demonstrate our method's functionality on several real-world datasets and compare it with state-of-the-art approximation algorithms. Although we show superior approximation performance in many cases, the method's real power lies in its agnosticism toward arbitrary GP customizations --- core kernel design, noise, and mean functions --- and the type of input space, making it optimally suited for modern Gaussian process applications.

General Machine Learning · Online Learning, Active Learning and Bandits

Abdelhamid Ezzerg, Ilija Bogunovic, Jeremias Knoblauch

Bayesian Optimization is critically vulnerable to extreme outliers. Existing provably robust methods typically assume a bounded cumulative corruption budget, which makes them defenseless against even a single corruption of sufficient magnitude. To address this, we introduce a new adversary whose budget is only bounded in the frequency of corruptions, not in their magnitude. We then derive RCGP-UCB, an algorithm coupling the upper confidence bound (UCB) approach with a Robust Conjugate Gaussian Process (RCGP). We present stable and adaptive versions of RCGP-UCB, and prove that they achieve sublinear regret in the presence of up to $O(T^{1/4})$ and $O(T^{1/7})$ corruptions with possibly infinite magnitude. This robustness comes at near zero cost: without outliers, RCGP-UCB's regret bounds match those of the standard GP-UCB algorithm.

Theory · Online Learning and Bandits

Shogo Iwazaki

We study an algorithm-independent, worst-case lower bound for the Gaussian process (GP) bandit problem in the frequentist setting, where the reward function is fixed and has a bounded norm in the known reproducing kernel Hilbert space (RKHS). Specifically, we focus on the squared exponential (SE) kernel, one of the most widely used kernel functions in GP bandits. One of the remaining open questions for this problem is the gap in the *dimension-dependent* logarithmic factors between upper and lower bounds. This paper partially resolves this open question under a hyperspherical input domain. We show that any algorithm suffers $\Omega(\sqrt{T (\ln T)^{d} (\ln \ln T)^{-d}})$ cumulative regret, where $T$ and $d$ represent the total number of steps and the dimension of the hyperspherical domain, respectively. Regarding the simple regret, we show that any algorithm requires $\Omega(\epsilon^{-2}(\ln \frac{1}{\epsilon})^d (\ln \ln \frac{1}{\epsilon})^{-d})$ time steps to find an $\epsilon$-optimal point. We also provide the improved $O((\ln T)^{d+1}(\ln \ln T)^{-d})$ upper bound on the maximum information gain for the SE kernel. Our results guarantee the optimality of the existing best algorithm up to *dimension-independent* logarithmic factors under a hyperspherical input domain.

Social Aspects · Privacy

Ossi Räisä, Antti Koskela, Antti Honkela

The accuracy-first perspective of differential privacy addresses an important shortcoming by allowing a data analyst to adaptively adjust the quantitative privacy bound instead of sticking to a predetermined bound. Existing works on the accuracy-first perspective have neglected an important property of differential privacy known as post-processing immunity, which ensures that an adversary is not able to weaken the privacy guarantee by post-processing. We address this gap by determining which existing definitions in the accuracy-first perspective have post-processing immunity, and which do not. The only definition with post-processing immunity, pure ex-post privacy, lacks useful tools for practical problems, such as an ex-post analogue of the Gaussian mechanism, and an algorithm to check if accuracy on separate private validation set is high enough. To address this, we propose a new definition based on Rényi differential privacy that has post-processing immunity, and we develop basic theory and tools needed for practical applications. We demonstrate the practicality of our theory with applications to synthetic data generation and image classifier fine-tuning, where our algorithm successfully adjusts the privacy bound until an accuracy threshold is met on a private validation dataset.

Theory · Reinforcement Learning and Planning

Viktor Stein, Adwait Datar, Nihat Ay

Kullback-Leibler divergence (KL) regularization is widely used in reinforcement learning, but it becomes infinite under support mismatch and can degenerate in low-noise limits. Utilizing a unified information-geometric framework we introduce (Kalman)-Wasserstein-based KL analogues by replacing the Fisher–Rao geometry in the dynamical formulation of the KL with transport-based geometries, and we derive closed-form values for common distribution families. These divergences remain finite under support mismatch and yield a geometric interpretation of regularization heuristics used in Kalman ensemble methods. We demonstrate the utility of these divergences in KL-regularized optimal control. In the fully tractable setting of linear time-invariant systems with Gaussian process noise, the classical KL reduces to a quadratic control penalty that becomes singular as process noise vanishes. Our variants remove this singularity and yield well-posed problems. On a double integrator and a cart-pole example, the resulting controls outperform KL-based regularization.

Probabilistic Methods · Gaussian Processes

Armin Lederer, Anuj Srivastava, Marco Bagatella, Andreas Krause

Bayesian optimization (BO) is a method commonly used for policy search in problems with low-dimensional policy parameterizations. While it is generally considered data-efficient, existing BO approaches are agnostic to the sequential structure of the optimization objective induced by policy roll-outs. Thereby, valuable information is discarded that could improve the convergence of BO. We address this inefficiency by developing and rigorously analyzing a novel approach for BO that relies on a temporal difference learning formulation for discounted infinite-horizon value functions based on Gaussian process (GP) regression. We derive learning error bounds for the proposed temporal difference GPs, such that we can exploit upper confidence bounds to analyze the cumulative regret of our BO approach. This analysis is further refined by bounding the maximal information gain for our temporal difference GP model. In a comparison with relevant baseline methods, we demonstrate the practical advantages of our method.

Probabilistic Methods · Bayesian Models and Methods

Katarzyna Kobalczyk, Zhiyuan Jerry Lin, Benjamin Letham, Zhuokai Zhao, Maximilian Balandat, Eytan Bakshy

Many real-world optimization problems are guided by complex, subjective preferences that are difficult to express as explicit closed-form objectives. In response, we introduce Language-in-the-Loop Optimization (LILO), a Bayesian optimization (BO) framework that employs a large language model (LLM) to translate free-form natural language feedback and prior knowledge from a decision maker into structured preference signals, going beyond the restrictive scalar or pairwise feedback formats typically assumed in preferential BO. The LLM-derived preferences are integrated by a Gaussian process proxy model, enabling principled acquisition-driven exploration with calibrated uncertainty. By placing the LLM in a supporting role rather than as the optimizer itself, LILO preserves the sample efficiency and stability of BO while providing a flexible and expressive feedback interface. Across synthetic and real-world benchmarks, LILO consistently outperforms both conventional preference-based BO methods and LLM-only optimizers, with particularly strong gains in feedback-limited regimes.

Social Aspects · Accountability, Transparency, and Interpretability

David Rundel, Fabian Fumagalli, Maximilian Muschalik, Bernd Bischl, Matthias Feurer

Shapley values are a principled attribution measure widely used in interpretable machine learning, but their exact computation scales exponentially with the number of players, motivating a wide range of approximation methods based on value-function evaluations of sampled coalitions. This raises the question of whether approximation accuracy can be improved by *adaptively* selecting coalitions for evaluation based on previous outcomes. This is particularly relevant in settings where the value function is costly, and the number of evaluations is severely limited, such as retraining-based feature importance, data valuation, and hyperparameter importance. For this purpose, we propose $\texttt{ShaplEIG}$, a Bayesian experimental design approach that approximates the expensive value function via a Gaussian process surrogate and adaptively selects coalitions based on their expected information gain about the Shapley values. Since Shapley values are a linear function of the value function, we show that the expected information gain is available in *closed form* and *efficiently* computable. In extensive experiments across diverse costly applications, our method consistently improves estimation accuracy over state-of-the-art baselines.