论文检索

Deep Learning · Generative Models and Autoencoders

Jiawei Zhang, Ziyuan Liu, Leon Yan, Zhenyu Xiao, Yuantao Gu

The distortion–perception (D–P) tradeoff is a fundamental phenomenon of Bayesian inverse problems, which characterizes the inherent tension between distortion performance and perceptual quality. Enabling flexible traversal of the D-P tradeoff at inference time is crucial for practical applications. Despite the recent success of diffusion models in zero-shot inverse problem solving, efficient and principled strategies for D-P traversal in diffusion-based inverse algorithms remain inadequately characterized. In this paper, we propose a stage-wise framework for realizing D-P traversal using a single diffusion model in zero-shot inverse problems. Our proposed method, termed MAP-RPS, starts with an MAP estimation stage that approximates the MMSE solution and provides a low-distortion initialization, followed by a re-noised posterior sampling stage that progressively improves perceptual quality. We provide theoretical analyses for both stages, establishing the validity and effectiveness of the proposed design. Furthermore, we extend MAP-RPS to the latent space, yielding LMAP-RPS, which enjoys broader applicability by leveraging large-scale pre-trained latent diffusion backbones. Extensive experiments demonstrate that MAP-RPS and LMAP-RPS enable more effective D-P traversal on various tasks, while also exhibiting strong performance as efficient solvers for real-world inverse problems.

Probabilistic Methods · Gaussian Processes

Rui-Yang Zhang, Henry Moss, Lachlan Astfalck, Edward Cripps, David Leslie

We introduce a formal active learning methodology for guiding the placement of Lagrangian observers to infer time-dependent vector fields -- a key task in oceanography, marine science, and ocean engineering -- using a physics-informed spatio-temporal Gaussian process surrogate model. The majority of existing placement campaigns either follow standard `space-filling' designs or relatively ad-hoc expert opinions. A key challenge to applying principled active learning in this setting is that Lagrangian observers are continuously advected through the vector field, so they make measurements at different locations and times. It is, therefore, important to consider the likely future trajectories of placed observers to account for the utility of candidate placement locations. To this end, we present BALLAST: Bayesian Active Learning with Look-ahead Amendment for Sea-drifter Trajectories. We observe noticeable benefits of BALLAST-aided sequential observer placement strategies on both synthetic and high-fidelity ocean current models. In addition, we developed a novel GP inference method -- the Vanilla SPDE Exchange (VaSE) -- to boost the GP posterior sampling efficiency, which is also of independent interest.

Reinforcement Learning · Multi-agent

Daeyon Hwang, Raunaq Suri, Valentin Villecroze, Anthony Caterini, Jesse Cresswell, Noël Vouitsis, Brendan Ross

LLM agents operate in two distinct regimes: open-weight agents amenable to reinforcement learning (RL) and black-box agents whose behaviour must be controlled purely at test time. Although black-box agents are often backed by state-of-the-art proprietary LLMs, API-only access precludes parameter-level optimization, rendering most RL methods inapplicable. To address this limitation, we turn to a known equivalence between RL and Bayesian inference. We propose Agentic Monte Carlo (AMC) to directly sample from the optimal policy of a black-box agent rather than training it through RL. The optimal policy is a posterior over trajectories whose prior we define as the fixed black-box LLM agent. We employ Sequential Monte Carlo to sample from this posterior by learning a value function to steer the agent while leaving the underlying black-box model unchanged. We validate AMC on three diverse environments from the AgentGym benchmark, demonstrating significant improvements over prompting baselines and even outperforming Group Relative Policy Optimization (GRPO) as we scale the test-time compute of our method. AMC demonstrates the feasibility of performing principled RL-style optimization of black-box LLM agents.

Applications · Chemistry, Physics, and Earth Sciences

Badr MOUFAD, Albina Ilina, Hai Victor Habi, Salem Lahlou, Yazid Janati, HAGIT MESSER, Eric Moulines

Commercial Microwave Links (CMLs) offer dense spatial coverage for rainfall sensing but produce path-integrated measurements that make accurate ground-level reconstruction challenging. Existing methods typically oversimplify CMLs as point sensors and neglect the physical power-law relating rainfall to signal attenuation, resulting in degraded performance under heterogeneous precipitation. In this work, we view rain field reconstruction as a Bayesian inverse problem with Diffusion Models (DMs) as high-fidelity spatial priors. We show that diffusion models better preserve key rainfall statistics compared to censored Gaussian processes. Framing rainfall estimation as a Bayesian inverse problem with a DM prior enables training-free posterior sampling using a broad family of methods, including Plug-and-Play, Sequential Monte Carlo, and Replica Exchange methods. Experiments on synthetic and real-world datasets demonstrate consistent improvements over established CML-based reconstruction baselines.

Deep Learning · Generative Models and Autoencoders

Julien Lalanne, David Picard, Lionel Boillot, Lina-María GUAYACÁN-CARRILLO, Leon Barens, Jean-Michel Pereira

Generative modeling provides a powerful framework for learning data distributions. These models initially relied on probabilistic methods such as Gaussian Processes (GP) for uncertainty-aware predictions and shifted towards larger trainable models to learn more complex distributions. In this work, we introduce *Random Process (RP) Flow*, a Flow Matching-based framework that represents the vector field as a neural implicit function. Unlike modern generative methods, our setting involves a single observed field, from which only sparse measurements are available. RP Flow uses Random Fourier Features to learn an implicit signal representation that can be queried at any arbitrary location from a limited set of observations, while encoding uncertainty through ensemble sampling. We propose constructing a Bayesian posterior by GP regression in the source space to generate high-quality samples. Our empirical results demonstrate that this framework generates realistic samples along with calibrated uncertainty estimates, even under challenging conditions such as high frequency, high sparsity, or high dimensionality. These findings position RP Flow as a milestone towards generative models for reconstruction tasks where data is scarce and uncertainty must remain traceable.

Social Aspects · Everything Else

Rachael Hwee Ling Sim, Jue Fan, Xiao Tian, Xinyi Xu, Patrick Jaillet, Bryan Kian Hsiang Low

Collaborative machine learning involves training high-quality models using datasets from a number of sources. To incentivize sources to share data, existing data valuation methods fairly reward each source based on its data submitted as is. However, as these methods do not verify nor incentivize data truthfulness, the sources can manipulate their data (e.g., by submitting duplicated or noisy data) to artificially increase their valuations and rewards or prevent others from benefiting. This paper presents the first mechanism that provably ensures (**F**) collaborative fairness and incentivizes (**T**) truthfulness at equilibrium for Bayesian models. Our mechanism combines semivalues (e.g., Shapley value), which ensure fairness, and a truthful data valuation function (DVF) based on a validation set that is unknown to the sources. As semivalues are influenced by others' data, we introduce an additional condition to prove that a source can maximize its expected data values in coalitions and semivalues by submitting a dataset that captures its true knowledge. Additionally, we discuss the implications and suitable relaxations of (**F**) and (**T**) when the mediator has a limited budget for rewards or lacks a validation set. Our theoretical findings are validated on synthetic and real-world datasets.

Deep Learning · Self-Supervised Learning

Fabian A Mikulasch, Friedemann Zenke

Self-supervised learning (SSL) excels at finding general-purpose latent representations from complex data, yet lacks a unifying theoretical framework that explains the diverse existing methods and guides the design of new ones. We cast SSL as latent distribution matching (DM): learning representations that maximize their log-probability under an assumed latent model (alignment), while maximizing latent entropy to prevent collapse (uniformity). This view unifies independent component analysis with contrastive, non-contrastive, and predictive SSL methods, including stop-gradient approaches. Leveraging DM, we derive a nonlinear, sampling-free Bayesian filtering model with a Kalman-based predictor for high-dimensional timeseries. We further prove that predictive DM yields identifiable latent representations under mild assumptions, even with nonlinear predictors. Overall, DM clarifies the assumptions behind established SSL methods and provides principled guidance for developing new approaches.

Probabilistic Methods · Bayesian Models and Methods

Xiaoxian Zhu, Yingmeng Li, Shuangge Ma, Mengyun Wu

In modern applications such as ECG monitoring, neuroimaging, wearable sensing, and industrial equipment diagnostics, complex and continuously structured data are ubiquitous, presenting both challenges and opportunities for functional data analysis. However, existing methods face a critical trade-off: conventional functional models are limited by linearity, whereas deep learning approaches lack interpretable region selection for sparse effects. To bridge these gaps, we propose a sparse Bayesian functional deep neural network (sBayFDNN). It learns adaptive functional embeddings through a deep Bayesian architecture to capture complex nonlinear relationships, while a structured prior enables interpretable, region-wise selection of influential domains with quantified uncertainty. Theoretically, we establish rigorous approximation error bounds, posterior consistency, and region selection consistency. These results provide the first theoretical guarantees for a Bayesian deep functional model, ensuring its reliability and statistical rigor. Empirically, comprehensive simulations and real‑world studies confirm the effectiveness and superiority of sBayFDNN. Crucially, sBayFDNN excels in recognizing intricate dependencies for accurate predictions and more precisely identifies functionally meaningful regions, capabilities fundamentally beyond existing approaches.

Probabilistic Methods · Bayesian Models and Methods

Ibne Farabi Shihab, SANJEDA AKTER, Anuj Sharma

Deep protein structure predictors such as AlphaFold provide confidence estimates (e.g., pLDDT) that are not calibrated and degrade under distribution shifts across experimental modalities, temporal changes, and disordered regions. We introduce **CalPro**, a prior-aware evidential conformal framework for shift-robust uncertainty quantification. CalPro combines three components: (i) a geometric evidential head outputting Normal Inverse Gamma distributions via graph neural networks; (ii) a differentiable calibration surrogate that shapes representations during training, followed by split-conformal calibration for finite-sample coverage; and (iii) domain priors (disorder, flexibility) encoded as soft constraints on predicted uncertainty. Theoretically, we derive structure-aware *sensitivity bounds* for coverage degradation under distribution shift using PAC-Bayesian control over ambiguity sets, quantifying how miscoverage increases with model complexity and shift magnitude. Empirically, CalPro achieves at most 5 percentage points coverage degradation across modalities compared to 15 to 25 points for baselines, reduces calibration error by 30\% to 50\%, and improves downstream docking success from 52\% to 75\% when filtering by uncertainty. The framework extends beyond proteins to structured regression tasks where priors encode local reliability.

Social Aspects · Accountability, Transparency, and Interpretability

Zongxin Liu, Xiaoyong Xue, Sun Weidi, Shengchao Qin, Lijun Zhang

Precise, training-free editing of text-to-image diffusion models requires balancing alignment (faithful attribute manifestation), consistency (preserving non-target content), and quality (artifact-free textures). Sparse autoencoder (SAE) steering offers interpretable, smooth ``slider-like'' control by manipulating SAE feature activations derived from the text encoder; however, existing approaches rely on heuristic feature selection and manual tuning of the steering strength, leading to suboptimal trade-offs among the three objectives. We propose Sparse Relaxed-Lasso Steering (SRLS), which casts steering-vector discovery as a convex sparse recovery problem. Exploiting the affine structure of the SAE decoder, SRLS automatically identifies sparse, generalizable support sets via a Lasso objective, and then debiases the coefficients using support-restricted ridge regression. We further select the optimal steering strength using Bayesian optimization. Experiments across diverse attributes and subjects show that SRLS generally improves over other methods, yielding a better balance among alignment, consistency, and quality.

Probabilistic Methods · Everything Else

Kianoosh Ashouritaklimi, Stefano Cortinovis, Francois Caron

Bayes--assisted conformal prediction combines the strengths of Bayesian modelling with exact, distribution--free frequentist coverage guarantees. While validity holds even under model misspecification, the size of the prediction sets can degrade significantly when the prior is poorly aligned with the observed data. We address this limitation by introducing \textbf{RoBAS}: a novel Bayes--assisted nonconformity score which is motivated by a hierarchical Bayesian working model with heavy--tailed priors, and which we implement in practice via a computationally tractable empirical Bayes instantiation. Our proposed method is adaptive to the quality of the available working information in the prior. When reliable prior information is available and can be effectively encoded, we achieve set sizes lower than that of other sets with the same coverage. On the other hand, when such information is weak or inaccurate, our nonconformity scores revert to the Distance--To--Average score, a robust baseline that is well--suited to settings where accurate prior information is not available. We evaluate our method on tabular and image regression tasks in the setting where there exists distribution shift between the training and calibration/test data. We find that our approach is competitive with widely used nonconformity scores in the absence of distribution shift, while providing significant gains in the more challenging setting of distribution shift.

Probabilistic Methods · Variational Inference

Bohan Wu, David Blei

Variational inference (VI) has emerged as a popular method for approximate inference for high-dimensional Bayesian models. In this paper, we propose a novel VI method that extends the naive mean field via entropic regularization, referred to as $\Xi$-variational inference ($\Xi$-VI). $\Xi$-VI has a close connection to the entropic optimal transport problem and benefits from the computationally efficient Sinkhorn algorithm. We show that $\Xi$-variational posteriors effectively recover the true posterior dependency, where the likelihood function is downweighted by a regularization parameter. We analyze the role of dimensionality of the parameter space on the accuracy of $\Xi$-variational approximation and the computational complexity of computing the approximate distribution, providing a rough characterization of the statistical-computational trade-off in $\Xi$-VI, where higher statistical accuracy requires greater computational effort. We also investigate the frequentist properties of $\Xi$-VI and establish results on consistency, asymptotic normality, high-dimensional asymptotics, and algorithmic stability. We provide sufficient criteria for our algorithm to achieve polynomial-time convergence. Finally, we show the inferential benefits of using $\Xi$-VI over mean-field VI and other competing methods, such as normalizing flow, on simulated and real datasets.

Probabilistic Methods · Bayesian Models and Methods

Moule Lin, Shuhao Guan, Andrea Patane, David Gregg, Goetz Botterweck

Large Language Models usually put more emphasis on accuracy and therefore, will guess even when not certain about the prediction, which is especially severe when fine-tuned on small datasets due to the inherent tendency toward miscalibration. In this work, we introduce Bayesian-LoRA, which reformulates the deterministic LoRA update as a probabilistic low-rank representation inspired by Sparse Gaussian Processes. We identify a structural isomorphism (in the functional sense of shared bilinear form, not strict algebraic equivalence) between LoRA's factorization and Kronecker-factored SGP posteriors, and show that LoRA emerges as a limiting case when posterior uncertainty collapses. We conduct extensive experiments on various LLM architectures across commonsense reasoning benchmarks. With only approximately 0.42M additional parameters and ${\approx}1.2{\times}$ training cost relative to standard LoRA, Bayesian-LoRA significantly improves calibration across models up to 30B, achieving up to 84\% ECE reduction and 76\% NLL reduction while maintaining competitive accuracy for both in-distribution and out-of-distribution (OoD) evaluations.

Probabilistic Methods · Bayesian Models and Methods

Armin Beck, Peter Ochs

Symmetries are known to improve the empirical performance of machine learning models, yet theoretical guarantees explaining these gains remain limited. Prior work has focused mainly on compact group symmetries and often assumes that the data distribution itself is invariant, an assumption rarely satisfied in real-world applications. In this work, we extend generalization guarantees to the broader setting of non-compact symmetries, such as translations and to non-invariant data distributions. Building on the PAC-Bayes framework, we adapt and tighten existing bounds, demonstrating the approach on McAllester's PAC-Bayes bound while showing that it applies to a wide range of PAC-Bayes bounds. We validate our theory with experiments on a rotated MNIST dataset with a non-uniform rotation group, where the derived guarantees not only hold but also improve upon prior results. These findings provide theoretical evidence that, for symmetric data, symmetric models are preferable beyond the narrow setting of compact groups and invariant distributions, opening the way to a more general understanding of symmetries in machine learning.

Probabilistic Methods · Bayesian Models and Methods

Nicolas M Zilberstein, Florentin Guth, Santiago Segarra, Eero Simoncelli

Generative diffusion models can provide powerful priors for inverse problems in imaging, but existing implementations suffer from two key limitations: $(i)$ they learn only an implicit approximation of the prior density, and $(ii)$ they rely on crude likelihood approximations that introduce biases in the sampling. We address these challenges by introducing a new energy-based model trained using denoising score matching with a covariance-based regularization that enforces consistency across different inverse problems. Our approach learns explicit, normalized posterior densities for diverse linear inverse problems using a single model, while preserving the sampling capabilities of diffusion models. This enables new capabilities unavailable to score-based diffusion models: energy-guided adaptive sampling that adjusts schedules on-the-fly, unbiased MCMC correction with Metropolis-Hastings acceptance, and blind degradation estimation via Bayes rule. We validate our method on multiple datasets (MNIST, CelebA, ImageNet) and tasks (inpainting, deblurring), demonstrating competitive or superior performance to established baselines.

Probabilistic Methods · Bayesian Models and Methods

Sara Pérez-Vieites, Sahel Iqbal, Simo Särkkä, Dominik Baumann

Bayesian experimental design (BED) provides a principled framework for optimizing data collection by choosing experiments that are maximally informative about unknown parameters. However, existing methods cannot deal with the joint challenge of (a) *partially observable dynamical systems*, where only noisy and incomplete observations are available, and (b) *fully online inference*, which updates posterior distributions and selects designs sequentially in a computationally efficient manner. Under partial observability, dynamical systems are naturally modeled as state-space models (SSMs), where latent states mediate the link between parameters and data, making the likelihood---and thus information-theoretic objectives like the expected information gain (EIG)---intractable. We address these challenges by deriving new estimators of the EIG and its gradient that explicitly marginalize latent states, enabling scalable stochastic optimization in nonlinear SSMs. Our approach leverages nested particle filters for efficient online state-parameter inference with convergence guarantees. Applications to realistic models, such as the susceptible–infectious–recovered (SIR) and a moving source location task, show that our framework successfully handles both partial observability and online inference.

Deep Learning · Large Language Models

Bowen Tian, Caixue He, Jiemin Wu, Jingying Wang, Wenshuo Chen, Zexi Li, Yutao Yue

Editing complex, long-form knowledge in Large Language Models remains a significant challenge due to the difficulty of maintaining generation coherence. Existing autoregressive methods like AnyEdit alleviate length constraints but rely on Fixed-window Chunking, which disregards logical structure and compromises consistency. To address this, we present AnyEdit++, a structure-aware framework incorporating Bayes-Chunk, an adaptive segmentation mechanism that dynamically identifies semantic boundaries based on Bayesian Surprise. We underpin this approach with a theoretical framework establishing two key principles: (1) Structural Independence: we prove that cross-segment interference is minimized when anchor keys are geometrically orthogonal (a condition naturally satisfied by our surprisal-based boundaries but violated by fixed windows), and (2) Causal Locality: we demonstrate that updates injected at these semantic peaks yield strictly superior control compared to arbitrary split points. Extensive experiments across mathematical reasoning, code generation, and narrative tasks demonstrate that AnyEdit++ achieves superior performance and robustness compared to state-of-the-art baselines, validating that structural awareness is critical for effective long-form knowledge editing.

Probabilistic Methods · Bayesian Models and Methods

Jiaxiang Yi, Miguel Bessa

Real-world data contains aleatoric uncertainty -- irreducible noise arising from imperfect measurements or from incomplete knowledge about the data generation process. Mean variance estimation (MVE) networks can learn this type of uncertainty but require ad-hoc regularization strategies to avoid overfitting and are unable to predict epistemic uncertainty (model uncertainty). Conversely, Bayesian neural networks predict epistemic uncertainty but are notoriously difficult to train due to the approximate nature of Bayesian inference. We propose to cooperatively train a variance estimation network with a Bayesian neural network and empirically demonstrate that the resulting model disentangles aleatoric and epistemic uncertainties while improving the mean estimation. We demonstrate the effectiveness and scalability of this method across a diverse range of datasets, including a time-dependent heteroscedastic regression dataset we created where the aleatoric uncertainty is known, used to assess estimation accuracy. The proposed method is straightforward to implement, robust, and adaptable to various model architectures.

Social Aspects · Accountability, Transparency, and Interpretability

Eric Bigelow, Daniel Wurgaft, YingQiao Wang, Hidenori Tanaka, Tomer Ullman, Noah Goodman, Ekdeep Singh Lubana

Large language models (LLMs) can be controlled at inference time through prompts (in-context learning) and internal activations (activation steering). Different accounts have been proposed to explain these methods, yet their common goal of controlling model behavior raises the question of whether these seemingly disparate methodologies can be seen as specific instances of a broader framework. Motivated by this, we develop a unifying, predictive account of LLM control from a Bayesian perspective. Specifically, we posit that both context- and activation-based interventions impact model behavior by altering its belief in latent concepts: steering operates by changing concept priors, while in-context learning leads to an accumulation of evidence. This results in a closed-form Bayesian model that is highly predictive of LLM behavior across context- and activation-based interventions in a set of domains inspired by prior work on many-shot in-context learning. This model helps us explain prior empirical phenomena - e.g., sigmoidal learning curves as in-context evidence accumulates--while predicting novel ones--e.g., additivity of both interventions in log-belief space, which results in distinct phases such that sudden and dramatic behavioral shifts can be induced by slightly changing intervention controls. Taken together, this work offers a unified account of prompt-based and activation-based control of LLM behavior, and a methodology for empirically predicting the effects of these interventions.

General Machine Learning · Transfer, Multitask and Meta-learning

Gang Liu, Xiaoxuan Zhang, Yuhong Feng, Mingyang Zhou, Xiaoqun Wu, Hao Liao, Rui Mao

Few-shot classification aims to adapt a pretrained model to novel classes with limited examples. While current methods often heuristically combine pretrained knowledge and few-shot evidence, we seek a more principled understanding of their relationship. In this paper, we propose a Bayesian-inspired optimal integration framework(BOIF) that interprets pretrained models as priors and few-shot evidence as likelihoods. Under conditional independence approximation, we show that the optimal log-posterior decomposes into the sum of prior logits and likelihood logits. This leads to a simple yet effective design principle: decouple the prior and likelihood pathways and combine their logits additively. Guided by this principle, we implement BOIF using CLIP with two novel enhancements: (1) a multi-level feature adapter to enrich visual representations, and (2) a simplified cache module for likelihood estimation. Extensive experiments on 11 benchmarks show BOIF achieves state-of-the-art performance (e.g., 80.61\% average accuracy at 16-shot) and strong out-of-distribution robustness. Our work provides both a principled perspective and an effective instantiation for few-shot adaptation.