论文检索

Deep Learning · Generative Models and Autoencoders

Phil Sidney Ostheimer, Mayank Kumar Nagda, Andriy Balinskyy, Gabriel Rodrigues, Jean Radig, Carl Herrmann, Stephan Mandt, Marius Kloft, Sophie Fellenz

Diffusion models (DMs) excel on dense continuous data, but are not designed for sparse continuous data. They do not model exact zeros that represent the deliberate absence of a signal. As a result, they erase sparsity patterns and perform unnecessary computation on mostly zero entries. With Sparsity-Exploiting Diffusion (SED), we model only non-zero values, preserving sparsity. SED delivers computational savings while maintaining or improving generation quality by skipping zeros during training and inference. Across physics and biology benchmarks, SED matches or surpasses conventional DMs and domain-specific baselines, while vision experiments provide intuitive insights into the limitations of dense DMs and the benefits of SED.

Applications · Computer Vision

Nasar Iqbal, Dennis Wagner, Philipp Liznerski, Nabeel Hussain Syed, Sophie Fellenz, Niki Martinel, Marius Kloft

Inaccurate Visual Anomaly Detection (VAD) can lead to critical failures in safety-sensitive domains, including autonomous navigation and industrial surveillance. With the increasing abundance and rapid proliferation of VAD algorithms, their reliable evaluation has become increasingly important and challenging. Commonly used evaluation metrics often fail to capture practically relevant aspects of model behavior, yielding inconsistent or misleading assessments by overlooking errors such as redundant detections and the spatial distribution of false positives. In this paper, we formalize the requirements for VAD evaluation by introducing a set of axiomatic, verifiable properties that an evaluation metric should satisfy. Through a systematic analysis of state-of-the-art evaluation methods, we show that none satisfies all proposed properties. To address this gap, we introduce SAAM-ALARM, a novel evaluation metric that satisfies these properties. Our results show that SAAM-ALARM provides a more nuanced and theoretically sound assessment, establishing a stronger standard for performance benchmarking in VAD.

Applications · Chemistry, Physics, and Earth Sciences

Yaotian Yang, Yiwen Tang, Yizhe Chen, Xiao Chen, Jiangjie Qiu, Hao Xiong, Haoyu Yin, Zhiyao Luo, Yifei Zhang, Sijia Tao 等

Reconstructing atomistic crystal structures from a single noisy STEM projection is an ill-posed inverse problem: multiple lattices can explain similar contrast, and purely feed-forward models cannot verify physical validity. We present **AutoMat**, a failure-aware agentic *controller* that performs inference-time hypothesis search with *closed-loop verification* to convert Scanning Transmission Electron Microscopy (STEM) images into simulation-ready crystal structures and downstream properties. AutoMat composes perception and physics modules—pattern-adaptive denoising, physics-guided template retrieval (as a fallback), symmetry-constrained atomic reconstruction, and MLIP-based relaxation/validation—and triggers rollback-and-retry when verification fails. For systematic evaluation, we introduce **STEM2Mat-Bench**, a benchmark dataset containing 450+ annotated samples. Performance is assessed using lattice root-mean-square deviation (RMSD), formation energy mean absolute error (MAE), and structure matching accuracy. Results demonstrate that AutoMat outperforms existing approaches including SOTA models, specialized domain tools, and closed-source multimodal large models. This work establishes a direct pathway from microscopic characterization to atomic-scale modeling, addressing a fundamental challenge in materials science.

Deep Learning · Generative Models and Autoencoders

Zander Blasingame, Chen Liu

Deep generative models based on neural differential equations have quickly become the state-of-the-art for numerous generation tasks across many different applications. These models rely on ODE/SDE solvers which integrate from a prior distribution to the data distribution. In many applications it is highly desirable to then integrate in the other direction. The standard solvers, however, accumulate discretization errors which don’t align with the forward trajectory, thereby prohibiting an exact inversion. In applications where the precision of the generative model is paramount this inaccuracy in inversion is often unacceptable. Current approaches to solving the inversion of these models results in significant downstream issues with poor stability and low-order of convergence; moreover, they are strictly limited to the ODE domain. In this work, we propose a new family of reversible exponential (stochastic) Runge-Kutta solvers which we refer to as Rex developed by an application of Lawson methods to convert any explicit (stochastic) Runge-Kutta scheme into a reversible one. In addition to a rigorous theoretical analysis of the proposed solvers, we also empirically demonstrate the utility of Rex on improving the sample of Boltzmann distributions with flow models, and improving image generation and editing capabilities with diffusion models.

Deep Learning · Large Language Models

Shenghu Jiang, Ruihao Gong

We propose a novel algorithm for incremental Byte Pair Encoding (BPE) tokenization. The algorithm processes each input byte in **worst-case** $\mathcal{O}(\log^2 t)$ time, leading to an overall complexity of $\mathcal{O}(n \log^2 t)$, where $n$ is the input length and $t$ is the maximum token length. The algorithm incrementally maintains BPE tokenization results for every prefix of the input text, implementing the standard BPE merge procedure defined by a fixed set of merge rules. This enables efficient partial tokenization in streaming settings. Functioning as a drop-in replacement for standard BPE, our approach achieves up to $\sim$3$\times$ speedups over Hugging Face's tokenizers, and significant latency reductions over OpenAI's tiktoken on pathological inputs. We further introduce an eager output algorithm that enables streaming output, emitting tokens as soon as token boundaries are determined during incremental tokenization. Overall, our results demonstrate that BPE tokenization can be performed incrementally with strong worst-case guarantees, while providing practical latency benefits in modern large language model pipelines.

Applications · Chemistry, Physics, and Earth Sciences

Jephte Abijuru, Mayank Kumar Nagda, Phil Sidney Ostheimer, Jan Tauberschmidt, Sebastian Vollmer, Stephan Mandt, Marius Kloft, Sophie Fellenz

Physics-informed neural networks (PINNs) enforce physical laws by minimizing partial differential equation (PDE) residuals and auxiliary constraints. Standard training relies on a mean-squared error (MSE) objective, which implicitly assumes independent Gaussian residuals with a fixed global variance. We show theoretically and empirically that residuals encountered during PINN training are heterogeneous and heavy-tailed, revealing a systematic mismatch with this assumption. As a consequence, a small number of large residuals can disproportionately dominate both the loss and gradient, leading to poorly balanced optimization dynamics. Motivated by this mismatch, we adopt a Student-$t$ residual model to explicitly capture heavy-tailed behavior. An equivalent hierarchical representation yields an expectation–maximization (EM) algorithm that alternates between estimating residual-dependent weights and optimizing network parameters via a weighted MSE objective, allowing existing PINN solvers to be reused in the M-step. The resulting training dynamics bound the influence of extreme residuals and admit almost sure convergence guarantees under standard stochastic optimization assumptions. Experiments across a diverse suite of challenging PDE benchmarks demonstrate consistently improved solution accuracy and robustness compared to standard PINN training.

General Machine Learning · Evaluation

Zihan Dong, Zhixian Zhang, Yang Zhou, Can Jin, Ruijia Wu, Linjun Zhang

Evaluating mathematical reasoning in LLMs is constrained by limited benchmark sizes and inherent model stochasticity, yielding high-variance accuracy estimates and unstable rankings across platforms. On difficult problems, an LLM may fail to produce a correct final answer, yet still provide reliable pairwise comparison signals indicating which of two candidate solutions is better. We leverage this observation to design a statistically efficient evaluation framework that combines standard labeled outcomes with pairwise comparison signals obtained by having models judge auxiliary reasoning chains. Treating these comparison signals as control variates, we develop a semiparametric estimator based on the efficient influence function (EIF) for the setting where auxiliary reasoning chains are observed. This yields a one-step estimator that achieves the semiparametric efficiency bound, guarantees strict variance reduction over naive sample averaging, and admits asymptotic normality for principled uncertainty quantification. Across simulations, our one-step estimator substantially improves ranking accuracy, with gains increasing as model output noise grows. Experiments on GPQA Diamond, AIME 2025, and GSM8K further demonstrate more precise performance estimation and more reliable model rankings, especially in small-sample regimes where conventional evaluation is pretty unstable.

General Machine Learning · Evaluation

Till Aczel, Lucas Theis, Roger Wattenhofer

Evaluating generative models is challenging because standard metrics often fail to reflect human preferences. Human evaluations are more reliable but costly and noisy, as participants vary in expertise, attention, and diligence. Pairwise comparisons improve consistency, yet aggregating them into overall quality scores requires careful modeling. Bradley-Terry-based methods update item scores from comparisons, but existing approaches either ignore rater variability or lack convergence guarantees, limiting robustness and interpretability. We introduce BBQ, a Bayesian Bradley-Terry variant that explicitly models rater quality, downweighting or removing unreliable participants, and provides guaranteed monotonic likelihood convergence through an Expectation-Maximization algorithm. Empirical results show that BBQ achieves faster convergence, well-calibrated uncertainty estimates, and more robust, interpretable rankings compared to baseline Bradley-Terry models, even with noisy or crowdsourced raters. This framework enables more reliable and cost-effective human evaluation of generative models.

Applications · Chemistry, Physics, and Earth Sciences

Jephte Abijuru, Mayank Kumar Nagda, Phil Sidney Ostheimer, Sebastian Vollmer, Marius Kloft, Sophie Fellenz

Physics-Informed Neural Networks (PINNs) embed physical laws into deep learning models. However, conventional PINNs often suffer from failure modes leading to inaccurate solutions. We trace these failure modes to two structural pathologies: gradient shattering, where gradients degrade with depth and provide little training signal, and flow mismatch, where training pushes predictions along trajectories that diverge from the PDE solution path. We introduce ResPINNs, which reformulate PINNs as residual flows, networks that iteratively refine their own predictions through explicit corrective steps, in the spirit of classical iterative solvers. Our analysis shows that this design mitigates both pathologies by keeping updates aligned with descent and by preserving informative gradients across depth. Extensive experiments on PDE benchmarks confirm that ResPINNs achieve higher accuracy with substantially fewer parameters than conventional architectures.

Shenggui Li, Chao Wang, YIKAI ZHU, Yubo Wang, Fan Yin, Shuai Shi, YefeiChen, Xiaomin Dong, Qiaoling Chen, Jin Pan 等

Speculative decoding mitigates the memory-bound nature of LLM decoding by using a lightweight draft model to propose multiple tokens for parallel verification. However, its adoption has been limited by the lack of high-quality draft models and scalable training infrastructure. We introduce SpecForge, an open-source and efficient framework for training speculative decoding models with full support for EAGLE-3. SpecForge incorporates target–draft decoupling, hybrid parallelism, optimized training kernels, and tight integration with production-grade inference engines, enabling up to 9.9x faster EAGLE-3 training for Qwen3-235B-A22B compared to the baseline. We further release SpecBundle, a suite of production-grade EAGLE-3 draft models trained with SpecForge for mainstream open-source LLMs, achieving up to 4.48x end-to-end inference speedup on SGLang and addressing the scarcity of high-quality drafts. Finally, we distill a systematic study of speculative decoding training into practical and actionable recipes to guide real-world adoption.

Deep Learning · Large Language Models

Kuang-Da Wang, Zhao Wang, Wei-Yao Wang, Yotaro Shimose, Jaechang Kim, Shingo Takamatsu

Recent Large Language Model–based approaches for clarifying visual design largely focus on selecting questions that better uncover user intent, but often overlook the cognitive burden imposed on users—i.e., the effort required to interpret and answer these questions—which is crucial for effective human–agent interaction. We propose ***Agentic Model Predictive Questioning Control (A-MPQC)***, a test-time framework that reduces user interaction burden while improving visual design alignment by formulating multi-round clarification as trajectory optimization with receding-horizon replanning, allowing the agent to revise its questioning strategy as feedback arrives. We further introduce lookahead question plans to reduce ambiguity early, and a lightweight respond-or-reject surrogate reward to steer questions toward lower-burden formats (e.g., yes/no). Experiments on webpage and ad banner generation benchmarks show that A-MPQC not only produces better designs but also achieves lower user-interaction cost across diverse baselines—including fixed-format strategies (e.g., multiple-choice and open-ended) and a retrieval-augmented baseline—without retraining. Overall, our work explicitly formulates and optimizes human cognitive burden jointly with final design alignment, opening new opportunities for advancing human–agent interaction in design.

General Machine Learning · Causality

Haoxiang Wang, Haoxuan Li, Ziyan Wang, Zhiheng Zhang, Aoqi Zuo, Erdun Gao, Kun Zhang, Mingming Gong

Treatment responder classification seeks to learn a rule to classify individuals who will benefit from the treatment. This paper studies a new scenario in treatment responder classification when abstention is allowed, i.e., practitioners can opt out of making uncertain classification on some individuals for further investigation. By revealing the implicit relation between causal misclassification risk with abstention and Conditional Value at Risk (CVaR), we develop a doubly robust method named TRECA to learn the classification rule under loose convergence conditions on nuisance parameters, and further extend it to deal with possible violation on key assumptions such as monotonicity and unconfoundedness. Rigorous theories and extensive experiments on two real-world datasets demonstrate the theoretical and experimental guarantee on our methods in learning treatment responders classification rules with low regret at the cost of limited abstention.

General Machine Learning · Data

Yingze Li, dong wang, Yiming Guo, Yao Chen, Hongzhi Wang, Bingsheng He

LLM-augmented database analytics face a major bottleneck in the costly prefill phase. Although relational tables inherently contain repeated attribute values, standard row-by-row processing produces fragmented prompt layouts that obscure shared prefixes, thereby minimizing opportunities for prefix KV cache reuse and constraining system efficiency. Existing solutions typically employ heuristic or exhaustive search methods to reorder prompt layouts, but these approaches can be inefficient and may not leverage the structural properties of relational tables. We address this challenge by formulating prefix-cache-aware prompt layout optimization as a problem rooted in the isomorphism between prefix-cache reuse and the radix tree topology induced by the relational data distribution. Building on this perspective, we introduce a practical greedy tree-shaping algorithm that efficiently selects row and column orderings to maximize prefix overlap. Our approach, SOLO, improves prefill throughput by up to 90.3\% under fixed prefix-cache budget. Moreover, it reduces planning overhead by up to 242$\times$ compared to state-of-the-art baselines.

Deep Learning · Theory

Puyu Wang, Junyu Zhou, Philipp Liznerski, Marius Kloft

Kolmogorov--Arnold Networks (KANs) have recently emerged as a structured alternative to standard MLPs, yet a principled theory for their training dynamics, generalization, and privacy properties remains limited. In this paper, we analyze gradient descent (GD) for training two-layer KANs and derive general bounds that characterize their training dynamics, generalization, and utility under differential privacy (DP). As a concrete instantiation, we specialize our analysis to logistic loss under an NTK-separable assumption, where we show that polylogarithmic network width suffices for GD to achieve an optimization rate of order $1/T$ and a generalization rate of order $1/n$, with $T$ denoting the number of GD iterations and $n$ the sample size. In the private setting, we characterize the noise required for $(\epsilon,\delta)$-DP and obtain a utility bound of order $\sqrt{d}/(n\epsilon)$ (with $d$ the input dimension), matching the classical lower bound for general convex Lipschitz problems. Our results imply that polylogarithmic width is not only sufficient but also necessary under differential privacy, revealing a qualitative gap between non-private (sufficiency only) and private (necessity also emerges) training regimes. Experiments further illustrate how these theoretical insights can guide practical choices, including network width selection and early stopping.

Social Aspects · Security

zhixuan ma, Haichang Gao, Shangwen Li, Ping Wang, Han Yu

Federated learning (FL) is vulnerable to backdoor attacks. Yet sustaining backdoor effectiveness under repeated aggregation remains challenging. Existing methods often rely on heuristic trigger designs or indiscriminant parameter manipulation, leading to rapid decay or detectable anomalies. In this work, we view FL backdoor persistence through the lens of optimization dynamics, and argue that long-lasting attacks require alignment between trigger-induced representations and aggregation-stable parameter directions. Based on this insight, we propose the Coupled Trigger Optimization and Vulnerable Parameter Alignment (CTO-VPA) FL backdoor attack method. By constraining updates to this coupled subspace, backdoor behaviors can be embedded into optimization-stable directions while preserving benign performance. Experiments across multiple datasets and defense settings show that CTO-VPA achieves substantially improved persistence and robustness compared to prior attacks, highlighting the importance of trigger–parameter coupling in FL settings.

Sherry Yang

Large language model (LLM) agents trained using reinforcement learning has achieved superhuman performance in low-cost environments like games, mathematics, and coding. However, these successes have not translated to complex domains where the cost of interaction is high, such as the physical cost of running robots, the time cost of ML engineering, and the resource cost of scientific experiments. The true bottleneck for achieving the next level of agent performance for these complex and high-cost domains lies in the expense of executing actions to acquire reward signals. To address this gap, this paper argues that we should use world models as an intermediary between agents and the real world. We discuss how world models, viewed as models of dynamics, rewards, and task distributions, can overcome fundamental barriers of high-cost actions such as extreme off-policy learning and sample inefficiency in long-horizon tasks. Moreover, we demonstrate how world models can provide critical and rich learning signals to agents across a broad set of domains, including machine learning engineering, computer use, robotics, and AI for science. Lastly, we identify the challenges of building these world models and propose actionable items along dataset curation, architecture design, scaling, and evaluation of world models.

Sahil Sidheekh, Sriraam Natarajan

Probabilistic circuits (PCs) enable exact and tractable inference but employ data independent mixture weights that limit their ability to capture local geometry of the data manifold. We propose Voronoi tessellations (VT) as a natural way to incorporate geometric structure directly into the sum nodes of a PC. However, naïvely introducing such structure breaks tractability. We formalize this incompatibility and develop two complementary solutions: (1) an approximate inference framework that provides guaranteed lower and upper bounds for inference, and (2) a structural condition for VT under which exact tractable inference is recovered. Finally, we introduce a differentiable relaxation for VT that enables gradient-based learning and empirically validate the resulting approach on standard density estimation tasks.

General Machine Learning · Causality

Ambroise Heurtebise, Lemir Omar Chehab, Pierre Ablin, Alexandre Gramfort, Aapo Hyvarinen

Causal discovery is a difficult problem that typically relies on strong assumptions on the data-generating model, such as non-Gaussianity. In practice, many modern applications provide multiple related views of the same system, which has rarely been considered for causal discovery. Here, we leverage this multi-view structure to achieve causal discovery with weak assumptions. We propose a multi-view linear Structural Equation Model (SEM) that extends the well-known framework of non-Gaussian disturbances by alternatively leveraging correlation over views. We prove the identifiability of the model for acyclic SEMs. Subsequently, we propose several multi-view causal discovery algorithms, inspired by single-view algorithms (DirectLiNGAM, PairwiseLiNGAM, and ICA-LiNGAM). The new methods are validated through simulations and applications on neuroimaging data, where they enable the estimation of causal graphs between brain regions.

Deep Learning · Large Language Models

Arash Gholamidavoodi, Navid Rezazadeh, Seyed Davoudi, Pouya Pezeshkpour

Large language models (LLMs) must balance diversity and creativity against logical coherence in open-ended generation. Existing truncation-based samplers are effective but largely heuristic, relying mainly on probability mass and entropy while ignoring semantic geometry of the token space. We present Top-W , a geometry-aware truncation rule that uses Wasserstein distance—defined over token-embedding geometry—to keep the cropped distribution close to the original, while explicitly balancing retained probability mass against the entropy of the kept set. Our theory yields a simple closed-form structure for the fixed-potential subset update: depending on the mass–entropy trade-off, the optimal crop either collapses to a single token or takes the form of a one-dimensional prefix that can be found efficiently with a linear scan. We implement Top-W using efficient geometry-based potentials (nearest-set or k-NN) and pair it with an alternating decoding routine that keeps the standard truncation-and-sampling interface unchanged. Extensive experiments on four benchmarks (GSM8K, GPQA, AlpacaEval, and MT-Bench) across three instruction-tuned models show that Top-W consistently outperforms prior state-of-the-art decoding approaches achieving up to 33.7% improvement. Moreover, we find that Top-W not only improves accuracy-focused performance, but also boosts creativity under judge-based open-ended evaluation. We will release all code upon acceptance.

General Machine Learning · Causality

Rémi Khellaf, Aurélien Bellet, julie Josse

Causal inference typically assumes centralized access to individual-level data. Yet, in practice, data are often decentralized across multiple sites, making centralization infeasible due to privacy, logistical, or legal constraints. We address this problem by estimating the Average Treatment Effect (ATE) from decentralized observational data via a Federated Learning (FL) approach, allowing inference through the exchange of aggregate statistics rather than individual-level data. We propose a novel method to estimate propensity scores via a federated weighted average of local scores using Membership Weights (MW), defined as probabilities of site membership conditional on covariates. MW can be flexibly estimated with parametric or non-parametric classification models using standard FL algorithms. The resulting propensity scores are used to construct Federated Inverse Propensity Weighting (Fed-IPW) and Augmented IPW (Fed-AIPW) estimators. In contrast to meta-analysis methods, which fail when any site violates positivity, our approach exploits heterogeneity in treatment assignment across sites to improve overlap. We show that Fed-IPW and Fed-AIPW perform well under site-level heterogeneity in sample sizes, treatment mechanisms, and covariate distributions. Theoretical analysis and experiments on simulated and real-world data demonstrate clear advantages over meta-analysis and related approaches.