论文检索

General Machine Learning · Everything Else

Zheng He, Danica J Sutherland

Testing conditional independence is a fundamental yet inherently difficult challenge, as controlling Type I error is impossible in general. The recently popular "Model-X" paradigm offers a solution by relying on a perfectly known conditional distribution. In traditional "one-shot" testing regimes, slight deviations from perfect knowledge are sometimes allowable, but existing work in more realistic online settings has required exact adherence to Model-X. We propose a new approach for sequential testing of conditional independence that is far more robust to estimation errors in the conditional distribution. Our method, based on online optimization of the Kernel Conditional Independence statistic, introduces a novel normalization and "truncate-and-shift" calibration strategy to the testing-by-betting paradigm. This framework greatly improves validity with estimated conditionals while still providing high power across high-dimensional synthetic benchmarks and real-world fairness tasks.

Theory · Learning Theory

Andrea Della Vecchia, Damir Filipovic

This paper investigates theoretical and methodological foundations for stochastic optimal control (SOC) in discrete time. We start formulating the control problem in a general dynamic programming framework, introducing the mathematical structure needed for a detailed convergence analysis. The associate value function is estimated through a sequence of approximations combining nonparametric regression methods and Monte Carlo subsampling. The regression step is performed within reproducing kernel Hilbert spaces (RKHSs), exploiting the classical KRR algorithm, while Monte Carlo sampling methods are introduced to estimate the continuation value. To assess the accuracy of our value function estimator, we propose a natural error decomposition and rigorously control the resulting error terms at each time step. We then analyze how this error propagates backward in time-from maturity to the initial stage-a relatively underexplored aspect of the SOC literature. Finally, we illustrate how our analysis naturally applies to a key financial application: the pricing of American options.

Optimization · Zero-order and Black-box Optimization

Jingzhe Jing, Zheyi Fan, Szu Hui Ng, Qingpei Hu

Bayesian optimization (BO) for high-dimensional constrained problems remains a significant challenge due to the curse of dimensionality. We propose **L**ocal **C**onstrained **B**ayesian **O**ptimization (LCBO), a novel framework tailored for such settings. Unlike trust-region methods that are prone to premature shrinking when confronting tight or complex constraints, LCBO leverages the differentiable landscape of constraint-penalized surrogates to alternate between rapid local descent and uncertainty-driven exploration. Theoretically, we prove that LCBO achieves a convergence rate for the Karush-Kuhn-Tucker (KKT) residual that depends polynomially on the dimension $d$ for common kernels under mild assumptions, offering a rigorous alternative to global BO where regret bounds typically scale exponentially. Extensive evaluations on high-dimensional benchmarks (up to 100D) demonstrate that LCBO consistently outperforms state-of-the-art baselines.

Deep Learning · Large Language Models

Beibei Xiong, Hangyu Lv, Junqi Liu, Yisen Wang, Shaoshi Chen, Jianlin Wang, Zhengfeng Yang, Lihong Zhi

Automating formal proofs of combinatorial identities is challenging for LLM-based provers, as long-horizon proof planning is required and unconstrained search quickly explodes. Symbolic methods such as the Wilf--Zeilberger (WZ) method can achieve a mechanized proof of combinatorial identities by constructing special auxiliary functions and demonstrating that they satisfy specific recurrence relations. We propose WZ-LLM, a neuro-symbolic framework that turns WZ proof plans into executable proof sketches in Lean~4 and uses an LLM-based prover to discharge the resulting machine-checkable subgoals. We also train a dedicated WZ-Prover via a Lean-kernel-verified bootstrapping loop with expert-verified iteration, followed by DAPO-based refinement. Experiments show that WZ-LLM achieves a 34\% proof success rate on LCI-Test (100 classical combinatorial identities), outperforming strong baselines such as DeepSeek-V3 and Goedel-Prover-V2; moreover, on LCI-Test it proves 5 identities on which the symbolic-only baseline fails. WZ-LLM also improves performance on CombiBench and PutnamBench-Comb, suggesting the effectiveness of coupling symbolic proof sketches with learned formal reasoning. Experiments show that WZ-LLM achieves a 34\% proof success rate on LCI-Test (100 classic combinatorial identities), outperforming strong baselines such as DeepSeek-V3 and Goedel-Prover-V2, and delivering consistent gains on CombiBench and PutnamBench-Comb. These results indicate that our framework provides two complementary strengths: improved direct proving for identities beyond the scope of WZ, and substantially higher end-to-end success when WZ sketches guide a specialized prover.

Probabilistic Methods · Monte Carlo and Sampling Methods

Sanghyeok Choi, Sarthak Mittal, Víctor Elvira, Jinkyoo Park, Esmeralda S. Whitammer

This paper proposes a synergy of amortised and particle-based methods for sampling from distributions defined by unnormalised density functions. We state a connection between sequential Monte Carlo (SMC) and neural sequential samplers trained by maximum-entropy reinforcement learning (MaxEnt RL), wherein learnt sampling policies and value functions define proposal kernels and twist functions. Exploiting this connection, we introduce an off-policy RL training procedure for the sampler that uses samples from SMC -- using the learnt sampler as a proposal -- as a behaviour policy that better explores the target distribution. We describe techniques for stable joint training of proposals and twist functions and an adaptive weight tempering scheme to reduce training signal variance. Furthermore, building upon past attempts to use experience replay to guide the training of neural samplers, we derive a way to combine historical samples with annealed importance sampling weights within a replay buffer. On synthetic multi-modal targets (in both continuous and discrete spaces) and the Boltzmann distribution of alanine dipeptide conformations, we demonstrate improvements in approximating the true distribution as well as training stability compared to both amortised and Monte Carlo methods.

General Machine Learning · Kernel methods

Dan Pu, Tianyi Zhu, Yao Yan, Wei Lan

Two-sample comparison is a fundamental problem in machine learning, with broad applications such as generative modeling. Although the maximum mean discrepancy (MMD) is widely used, MMD-based tests often exhibit poor or even counterintuitive performance under covariance- and location-shift alternatives, partly due to cancellation effects induced by their sum-of-differences construction. To address this issue, we propose a kernel-based maximum-of-difference (MOD) test, which maximizes the squared discrepancy between within-sample and between-sample average distances, thereby improving sensitivity to subtle distributional differences. We further develop a fused MOD procedure to adaptively combine multiple kernels. Extensive experiments demonstrate clear performance gains over existing MMD-based methods.

General Machine Learning · Hardware and Software

Nina Wiedemann, Quentin Leboutet, Michael Paulitsch, Diana Wofk, Benjamin Ummenhofer

GPU kernel optimization challenges LLMs beyond standard coding tasks, as it requires an understanding of hardware architecture, parallel computing optimization strategies, and profiling outputs. However, most existing approaches leveraging LLMs for kernel generation apply standard prompting and feedback loops, considering hardware only through profiling feedback. We introduce KernelFoundry, an evolutionary framework that efficiently explores the space of GPU kernels through (1) MAP-Elites quality-diversity search with kernel-specific behavioral dimensions to sustain exploration; (2) meta-prompt evolution that co-evolves prompts with kernels to uncover task-specific optimization strategies, and (3) a template-based parameter optimization approach to tune kernels to inputs and hardware. We evaluate this framework on KernelBench, robust-kbench and custom tasks, generating SYCL kernels as a cross-platform GPU programming paradigm, and CUDA kernels for comparison to prior work. Our approach consistently outperforms the baseline methods and achieves an average speedup of 2.3 on KernelBench for SYCL. Moreover, KernelFoundry is implemented as a distributed framework with remote access to diverse hardware, allowing quick benchmarking and featuring a flexible user input layer to support kernel generation for a for a wide range of real use cases beyond benchmarking.

Theory · Probabilistic Methods

Binglin Li, Matthew Reed, Seong-Tae Kim

Compositional data analysis has gained increasing attention due to the widespread occurrence of simplex-valued data, including microbiome data. However, existing kernel or distance-based nonparametric two-sample tests are often designed for Euclidean data and rely on square-root or log-transformations, motivating the need for a unified framework for nonparametric two-sample testing applicable to both compositional and directional data. We propose a studentized spherical harmonic energy distance-based two-sample test over a fixed dimensional underlying space, incorporating U-statistics theory and recent developments of studentization in the context of compositional and directional data. We establish asymptotic normality of our studentized test statistics constructed via spherical harmonics theory, avoiding the need for permutation or bootstrap tests. Simulations demonstrate convergence to the limiting distribution, empirical size control, and improved power in certain scenarios. Our proposed framework paves a new direction for nonparametric testing in non-Euclidean data analysis.

Theory · Probabilistic Methods

Yang Yang, Duo Zheng, Sandeep Jain, Kai Zhang, Ping-Shou Zhong

This paper introduces a new family of adaptive, distribution-free independence tests for multivariate random vectors based on binary expansion coefficients, supported by a tractable asymptotic theory. Our first key contribution establishes a general equivalence between independence testing and testing cross-covariances among exponentially many binary expansion interaction coefficients, applicable to broad sample spaces and not limited to kernel-induced representations. While this exponential interaction structure makes naive construction and computation infeasible, we overcome this challenge by reformulating the proposed tests as a class of U-statistics and deriving an explicit kernel representation that enables scalable and efficient computation. Exploiting the multiscale nature of binary expansions, the proposed framework automatically adapts to unknown dependence structures by selectively truncating higher-order interactions, yielding both strong power and clear interpretability. To further enhance power and computational efficiency, we introduce an adaptive weighted aggregation procedure, termed wa-dCoBET, which combines a baseline Covariance Binary Expansion Test (CoBET) with a distance-measure–based CoBET. Extensive simulations and a real-data application demonstrate that wa-dCoBET consistently matches or outperforms HSIC and distance covariance, particularly in higher-dimensional and non-monotone settings, while maintaining accurate type I error control.

Deep Learning · Theory

Zixiong Yu, Guhan Chen, Jianfa Lai, Bohan Li, Songtao Tian

Scaling factors in residual branches have emerged as a prevalent method for boosting neural network performance, especially in normalization-free architectures. While prior work has primarily examined scaling effects from an optimization perspective, this paper investigates their role in residual architectures through the lens of generalization theory. Specifically, we establish that wide residual networks (ResNets) with constant scaling factors become asymptotically unlearnable as depth increases. In contrast, when the scaling factor exhibits rapid depth-wise decay combined with early stopping, over-parameterized ResNets achieve minimax-optimal generalization rates. To establish this, we demonstrate that the generalization capability of wide ResNets can be approximated by the kernel regression associated with a specific kernel. Our theoretical findings are validated through experiments on synthetic data and real-world classification tasks, including MNIST and CIFAR-100.

Applications · Computer Vision

Xian-Hua Han

Reconstructing hyperspectral images from compressive measurements is challenging due to a fundamental mismatch between locally reliable observations and globally entangled structures induced by spectral dispersion. This study formalizes this issue as a local–global dissonance in representation learning for CASSI systems. To resolve it, we propose a Hierarchical Scale-Reconciling Architecture (HSRA) that enforces local sufficiency and global consistency in a principled, scale-aware manner. HSRA combines multi-kernel token mixing, latent window interactions, and hierarchical multi-granularity spatially shifted attention to progressively reconcile physical constraints across scales. Embedded into a deep unfolding framework as a physically grounded learned prior, Extensive experiments on benchmarks demonstrate that HSRA achieves consistent and significant improvements over state-of-the-art methods.

General Machine Learning · Sequential, Network, and Time Series Modeling

Cheonwoo Lee, Dooho Lee, Doyun Choi, Jaemin Yoo

Multi-scale modeling has emerged as an effective design principle for time-series forecasting by capturing temporal dynamics at multiple resolutions. As no principled foundation has been established in the literature, we unify existing scaling methods into a scaling operator family, revealing a fundamental limitation of existing approaches: reliance on fixed and discrete scaling. To address this limitation, we propose SiGMA (Single Generalized Multi-scale Architecture), which enables position-wise scaling via the learnable discrete Gaussian (LDG) kernel grounded in scale-space theory. We evaluate SiGMA comprehensively on long- and short-term forecasting benchmarks against state-of-the-art multi-scale baselines. SiGMA outperforms all competitors on both tasks, especially achieving the best performance in 13 out of 16 long-term evaluation settings. Beyond accuracy, SiGMA significantly improves training speed by up to 5.3 times and reduces memory consumption by up to 3.8 times over the strongest competitors.

Deep Learning · Generative Models and Autoencoders

Tong Yang, Moonkyung Ryu, Chih-wei Hsu, Guy Tennenholtz, Yuejie Chi, Craig Boutilier, Bo Dai

Controllable diffusion generation often relies on various heuristics that are seemingly disconnected without a unified understanding. We bridge this gap with Diffusion Controller (DiffCon), a unified control-theoretic view that casts reverse diffusion sampling as state-only stochastic control within (generalized) linearly-solvable Markov Decision Processes (LS-MDPs). Under this framework, control acts by reweighting the pretrained reverse-time transition kernels, balancing terminal objectives against an $f$-divergence cost. From the resulting optimality conditions, we derive practical reinforcement learning methods for diffusion fine-tuning: (i) $f$-divergence-regularized policy-gradient updates, including a PPO-style rule, and (ii) a regularizer-determined reward-weighted regression objective with a minimizer-preservation guarantee under the Kullback–Leibler (KL) divergence. The LS-MDP framework further implies a principled model form: the optimal score decomposes into a fixed pretrained baseline plus a lightweight control correction, motivating a side-network parameterization conditioned on exposed intermediate denoising outputs, enabling effective \emph{gray-box adaptation} with a frozen backbone. Experiments on Stable Diffusion v1.4 across supervised and reward-driven finetuning show consistent gains in preference-alignment win rates and improved quality–efficiency trade-offs versus gray-box baselines and even the parameter-efficient white-box adapter LoRA.

General Machine Learning · Kernel methods

Rustem Takhanov, Zhenisbek Assylbekov

Conditionally positive definite (CPD) kernels are defined with respect to a function class $\mathcal{F}$. It is well known that such a kernel $K$ is associated with its native space (defined analogously to an RKHS), which in turn gives rise to a learning method --- called conditional kernel ridge regression (conditional KRR) due to its analogy with KRR --- where the estimated regression function is penalized by the square of its native space norm. This method is of interest because it can be viewed as classical linear regression, with features specified by $\mathcal{F}$, followed by the application of standard KRR to the residual (unexplained) component of the target variable. Methods of this type have recently attracted increasing attention. We study the statistical properties of this method by reducing its behavior to that of KRR with another fixed kernel, called the residual kernel. Our main theoretical result shows that such a reduction is indeed possible, at the cost of an additional term in the expected test risk, bounded by $\mathcal{O}(1/\sqrt{N})$, where $N$ is the sample size and the hidden constant depends on the class $\mathcal{F}$ and the input distribution. This reduction enables us to analyze conditional KRR in the case where $K$ is positive definite and $\mathcal{F}$ is given by the first $k$ principal eigenfunctions in the Mercer decomposition of $K$. We also consider the setting where $\mathcal{F}$ consists of $k$ random features from a random feature representation of $K$. It turns out that these two settings are closely related. Both our theoretical analysis and experiments confirm that conditional KRR outperforms standard KRR in these cases whenever the $\mathcal{F}$-component of the regression function is more pronounced than the residual part.

General Machine Learning · Kernel methods

Zonghao Chen, Aratrika Mustafi, Pierre Glaser, Anna Korba, Arthur Gretton, Bharath K. Sriperumbudur

We introduce a (de)-regularization of the Maximum Mean Discrepancy (DrMMD) and its Wasserstein gradient flow. Existing gradient flows that transport samples from source distribution to target distribution with only target samples, either lack tractable numerical implementation ($f$-divergence flows) or require strong assumptions and modifications, such as noise injection, to ensure convergence (Maximum Mean Discrepancy flows). In contrast, DrMMD flow can simultaneously (i) guarantee near-global convergence for a broad class of targets in both continuous and discrete time, and (ii) be implemented in closed form using only samples. The former is achieved by leveraging the connection between the DrMMD and the $\chi^2$-divergence, while the latter comes by treating DrMMD as MMD with a de-regularized kernel. Our numerical scheme employs an adaptive de-regularization schedule throughout the flow to optimally balance the trade-off between discretization errors and deviations from the $\chi^2$ regime. The potential application of the DrMMD flow is demonstrated across several numerical experiments, including a large-scale setting of training student/teacher networks.

General Machine Learning · Kernel methods

Yun Xin, Bangqi Pan, Jianfeng Lu, Shuqin Cao, Gang Li, Guanghui Wen

Federated Learning (FL) enables collaborative model training across decentralized clients, but its performance often degrades under non-IID data distributions, particularly in the presence of data skew. Existing approaches mitigate this issue by estimating client skew via kernel density estimation over neighboring model updates, which preserves privacy and reduces communication costs. However, such approaches suffer from two fundamental limitations: bias toward skewed majority clients due to fixed neighborhood structures, and vulnerability to noise-induced perturbation in kernel space. To address these challenges, we propose FedVeer, a skew-aware FL framework based on self-adaptive kernel density estimation with $k$-free neighborhoods. FedVeer dynamically determines the neighborhood size via max-margin learning to mitigate majority-client bias, and further incorporates Kalman filtering to stabilize margin estimation under noisy updates, with a high-probability theoretical guarantee on margin deviation. Extensive experiments on real-world datasets demonstrate that FedVeer consistently outperforms four baselines, achieving up to 6.36\% accuracy improvement and reducing noise-induced degradation by up to 6.01\%.

Deep Learning · Generative Models and Autoencoders

Ziang Gan, Qi Zhu, Libao Zhang

Diffusion models serve as generative priors for dataset distillation, yet existing pipelines rely on per-sample update rules that evolve each synthetic image independently, limiting their ability to optimize collective set-level objectives. We propose Set-Coupled Guidance (SCG), a plug-and-play auxiliary controller that shifts from per-image to group (IPC-at-once) sampling by injecting set-symmetric feedback at each diffusion step. SCG combines spectral set-point regulation, which aligns set-level statistics to real data via empirical characteristic function matching, with cooperative kernel coupling that stabilizes joint trajectories under noisy feedback. All computations operate on lightweight descriptors extracted from predicted clean latents, adding low overhead to the base method. We provide theoretical analysis including Lyapunov descent and input-to-state stability for distributional tracking. Experiments on ImageNette, ImageWoof, ImageNet-100 and ImageNet-1K show consistent accuracy gains across multiple diffusion-based baselines.

Social Aspects · Privacy

Quentin Sinh, Jan Ramon

We study the problem of computing a U-statistic with a kernel function $f$ of degree $k \geq 2$, i.e., the average of some function $f$ over all $k$-tuples of instances, in a federated learning setting. U-statistics of degree $2$ include several useful statistics such as Kendall's $\tau$ coefficient, the Area under the Receiver-Operator Curve and the Gini mean difference. Existing methods provide solutions only under the lower-utility local differential privacy model and/or scale poorly in the size of the domain discretization. In this work, we propose a protocol that securely computes U-statistics of degree $k \geq 2$ under central differential privacy by leveraging Multi Party Computation (MPC). Our method substantially improves accuracy when compared to prior solutions. We provide a detailed theoretical analysis of its accuracy, communication and computational properties. We evaluate its performance empirically, obtaining favorable results, e.g., for Kendall's $\tau$ coefficient, our approach reduces the Mean Squared Error by up to four orders of magnitude over existing baselines.

Optimization · Zero-order and Black-box Optimization

Zhe Li, Bicheng Ying, Zidong Liu, Haibo Yang

Classical optimization theory establishes that zeroth-order (ZO) algorithms suffer from a dimension-dependent slowdown, with convergence rates typically scaling with the model dimension compared to first-order methods. However, in contrast to these theoretical expectations, a growing body of recent work demonstrates the successful application of ZO methods to fine-tuning Large Language Models (LLMs) with billions of parameters. To explain this paradox, we derive the one-step learning dynamics of ZO SGD, where the empirical Neural Tangent Kernel (eNTK) naturally emerges as the key term governing the learning behavior. Inspection of the eNTK produced by ZO-SGD reveals that each element corresponds to the inner product of neural tangent vectors projected onto a random low-dimensional subspace. Thus, by invoking the Johnson-Lindenstrauss Lemma, our analysis shows that the fidelity of the ZO eNTK is governed primarily by the number of perturbations. Crucially, the approximation error depends on the model output size rather than the massive parameter dimension. This dimension-free property provides a theoretical justification for the scalability of ZO methods to LLMs finetuning tasks. We believe that this kernel-based framework offers a novel perspective for understanding ZO methods within the context of learning dynamics.

Applications · Chemistry, Physics, and Earth Sciences

jieyuan pei, Zhuoxuan Li, Wei Li, Haobo Zhang, jiawei jiang, Jianwei Zheng

Neural operators have emerged as powerful surrogates for solving PDEs, significantly accelerating scientific computation. While transformer-based architectures offer unmatched flexibility for irregular domains, they suffer from a fundamental efficiency gap: standard attention mechanisms assign uniform interaction budgets to all token pairs, neglecting the physical reality that far-field interactions are often compressible. To address this mismatch, we draw inspiration from classical fast solvers that exploit hierarchical near-far decompositions. We further observe that embedding such tree-structured hierarchies in Euclidean space incurs inherent distortion, whereas hyperbolic space naturally accommodates exponential branching. Consequently, we propose \textbf{Hyperbolic Neural Operator (HNO)}, which leverages intrinsic hyperbolic geometry to instantiate a continuous Gibbs kernel based on stabilized geodesic distances on the Lorentz hyperboloid. This design imposes a geometric inductive bias that naturally enjoys robust multi-scale routing akin to the Fast Multipole Method (FMM), yet within a unified, learnable attention mechanism. Empirically, HNO achieves state-of-the-art accuracy on six PDE benchmarks and two large-scale unstructured CFD tasks, reducing the mean relative $\ell_2$ error by up to 40\% compared to leading baselines. Codes are attached and will be available online.