论文检索

Reinforcement Learning · Deep RL

Mingfei Sun

Natural policy gradients improve optimization by accounting for the geometry of distribution space, but their practical use is limited by the cost of estimating and inverting the Fisher matrix. We present Randomized Advantage Transformation (RAT), a method for estimating Tikhonov-regularized natural policy gradients via direct backpropagation. By applying the Woodbury formula, we reformulate the regularized natural gradient as vanilla policy gradients with a transformed advantage. RAT computes this transformation efficiently via randomized block Kaczmarz iterations on on-policy mini-batches, avoiding explicit Fisher construction, conjugate-gradient solvers, and architecture-specific approximations. We provide convergence guarantees for RAT and demonstrate empirically that it matches or exceeds established natural-gradient methods across continuous and visual control benchmarks, while remaining simple to implement and compatible with various architectures.

Applications · Chemistry, Physics, and Earth Sciences

Jiyeon Kim, Byungju Lee, Won-Yong Shin

Unlike most static material properties widely studied in the machine learning literature, ionic transport properties are inherently dynamic, making their fast and accurate prediction from static atomic structures challenging. The current standard approach, molecular dynamics (MD) simulations, suffers from prohibitively high computational cost. Recent autoregressive learning-based MD acceleration methods requiring sequential inference remain slow and prone to error accumulation; in contrast, existing non-autoregressive material property prediction models are less accurate because they fail to exploit dynamics. Moreover, existing methods typically benefit from datasets either with or without atomic trajectories, but not both. To overcome these limitations, we propose a non-autoregressive learning framework based on modality reduction, which treats atomic trajectories as an auxiliary modality during training but does not require them at inference. This enables a predictor to learn dynamics without requiring sequential inference and to benefit from both types of datasets. As a result, our framework achieves over 200 times speedup compared to autoregressive models on the dataset with atomic trajectories while substantially reducing prediction error relative to non-autoregressive benchmarks across both types of datasets. Our code is available at https://anonymous.4open.science/r/2026.

Deep Learning · Large Language Models

Yebin Yang, Debing Zhang, Huaijin Wu, Jingtao Han, Lin Yao, Xiaohan Qin, Jingzhi Wang, Junchi Yan

LLMs have traditionally scaled along dense dimensions, where performance is coupled with near-linear increases in computational cost. While MoE decouples capacity from compute, it introduces large memory overhead and hardware efficiency challenges. To overcome these, we propose token-indexed parameters as a novel, orthogonal scaling axis that decouple model capacity from FLOPs. Specifically, we introduce ReToken and MoRT, which augment Transformer layers with modulation vectors retrieved from auxiliary embedding tables. These vectors modulate the backbone via lightweight, element-wise operations, incurring negligible FLOPs overhead. Extensive experiments on both dense and MoE backbones, spanning from 190M to 9.8B parameters, demonstrate that our approach consistently reduces validation loss and significantly improves downstream task performance (e.g., +7.3 on ARC-C, +6.3 on GSM8K). Rigorous isoFLOPs analysis further confirms that MoRT fundamentally shifts the quality–compute Pareto frontier, achieving comparable model quality with 35\% less compute relative to vanilla MoE architectures, and we validate that token-indexed parameters exhibit a predictable power-law scaling behavior. Moreover, our efficient implementation ensures that the overhead introduced by ReToken and MoRT remains marginal.

Deep Learning · Generative Models and Autoencoders

Elio Moreau, Florentin Coeurdoux, Grégoire Ferré, Eric Vanden-Eijnden

Understanding the geometry of learned distributions is fundamental to improving and interpreting diffusion models, yet systematic tools for exploring their landscape remain limited. Standard latent-space interpolations fail to respect the structure of the learned distribution, often traversing low-density regions. We introduce a framework based on the string method that computes continuous paths between samples by evolving curves under the learned score function. Operating on pretrained models without retraining, our approach interpolates between three regimes: pure generative transport, which yields continuous sample paths; gradient-dominated dynamics, which recover minimum energy paths (MEPs); and finite-temperature string dynamics, which compute principal curves---self-consistent paths that balance energy and entropy. We demonstrate that the choice of regime matters in practice. For image diffusion models, MEPs contain high-likelihood but unrealistic ``cartoon'' images, confirming prior observations that likelihood maxima appear unrealistic; principal curves instead yield realistic morphing sequences despite lower likelihood. For protein structure prediction, our method computes transition pathways between metastable conformers directly from models trained on static structures, yielding paths with physically plausible intermediates. Together, these results establish the string method as a principled tool for probing the modal structure of diffusion models---identifying modes, characterizing barriers, and mapping connectivity in complex learned distributions.

Jianqing Liang, Xinkai Wei, Zhiqiang Li

Graph Contrastive Learning (GCL) has significantly advanced self-supervised representation learning on graphs, yet its practical efficacy remains hindered by random augmentations that induce semantic distortion and rigid one-to-one sampling strategy that amplifies inter-class entanglement and intra-class dispersion. To address these limitations, we develop CL-GCL, a Comprehensive and Lightweight Graph Contrastive Learning framework. Specifically, we exploit graph coarsening to preserve structural semantics through community-level representations and manifold learning to capture local geometric relations without costly pairwise distance computations. This design naturally aligns with the neighborhood aggregation principle of Graph Convolutional Networks, enhancing structural consistency while eliminating negative sampling bias. We theoretically prove that CL-GCL approximates node-level contrastive loss under mild conditions. Extensive experiments demonstrate consistent superiority in both accuracy and efficiency over state-of-the-art GCL methods.

General Machine Learning · Sequential, Network, and Time Series Modeling

Chao Yang, Wenjie Shen, Shuang Li

We propose a non-autoregressive branching diffusion model for generating spatio-temporal point processes. Starting from a geometric principle---the Wasserstein-Fisher-Rao (WFR) gradient flow of a generalized KL divergence toward a simple reference intensity---we obtain a tractable forward noising mechanism with two interpretable components: (i) a Langevin-type \emph{drift-diffusion} step that perturbs event locations and times, and (ii) a \emph{birth-death branching} step that changes the event count via location-dependent thinning (deaths) and Poisson offspring replication (births). We learn the reverse-time dynamics using a permutation-equivariant denoiser that predicts a drift field and a net-growth field, and we train it using an entropic-regularized unbalanced optimal transport (UOT), which naturally handles count mismatch between noisy and clean samples. The resulting generator produces complete spatio-temporal event sets without autoregressive simulation or explicit intensity normalization.

Reinforcement Learning · Everything Else

Mingjie Hu, Enlu Zhou, Jianqiang Hu

This paper studies a constrained linear best arm identification problem with covariate selection in the fixed-confidence setting, where each arm is evaluated across multiple performance metrics. The mean performance of each metric depends linearly on the feature vectors of both arms and covariates. The goal is to identify the arm with the highest expected value of one targeted metric while ensuring that the means of the remaining metrics stay below specified thresholds for each covariate. We first establish an instance-dependent lower bound on the sample complexity, formulated as a multi-level optimization problem that captures both feasibility and optimality. We then prove that this bound is tight by designing an algorithm that asymptotically matches it. Since the original algorithm is computationally intensive, we develop a relaxed version of the bound through a surrogate optimization problem and derive its convex dual. Using this bound, we propose a duality-based decomposition algorithm that is computationally efficient, updating only two coordinates and performing a single gradient step per iteration. We further show that the algorithm achieves the relaxed bound in theory and demonstrates its practical effectiveness through numerical experiments.

Deep Learning · Generative Models and Autoencoders

Luhan Tang, Longxuan Yu, Shaorong Zhang, Greg Ver Steeg

Discrete diffusion language models (dLLMs) offer a fast and flexible alternative to autoregressive models (ARMs) for discrete sequence generation by performing iterative denoising with parallel updates. Despite these advantages, dLLMs are commonly evaluated using metrics developed for ARMs. Such evaluations rely on metrics computed from final generated samples and conflate model approximation error in the learned denoiser and sampler-induced error from the sampling dynamics. We introduce a sampler-centric oracle evaluation framework that replaces learned denoisers with an oracle Hidden Markov Model posterior derived from a ground-truth Markov chain, enabling isolation of sampler-induced error under controlled and method-consistent settings. We show that few-step discrete diffusion samplers are not distributionally correct, even under an exact oracle denoiser, with substantial distributional mismatch at the transition level persisting at small step counts and vanishing only when the number of diffusion steps approaches the sequence length. We also find that current metrics for evaluating dLLMs are insufficient: improvements in negative log-likelihood, generative perplexity, or MAUVE do not imply correct sampling.

Deep Learning · Robustness

Hesam Asadollahzadeh, Feng Liu, Christopher Leckie, Sarah Erfani

Finetuning pretrained multimodal models improves in-distribution performance but often degrades out-of-distribution (OOD) robustness, a phenomenon known as catastrophic forgetting. We develop a theoretical framework for multimodal contrastive finetuning by introducing a *contrastive target matrix* that reformulates the objective as a matrix least-squares problem, yielding closed-form solutions and a geometric decomposition of how different strategies manage pretrained knowledge. Our analysis reveals a largely overlooked limitation: standard Exponential Moving Average (EMA) teachers, widely used in robust finetuning, suffer from late-stage collapse where the teacher--student gap vanishes precisely when OOD robustness is most vulnerable. We prove that a Weighted Moving Average (WMA) teacher, which integrates the full optimization trajectory, maintains a persistent regularizing force over finite horizons, enabling bias-free convergence in the task subspace while preserving orthogonal knowledge. These insights motivate **TRACER** (**T**rajectory-**R**obust **A**nchoring for **C**ontrastive **E**ncoder **R**egularization), which combines contrastive learning with WMA-guided multi-perspective distillation. Extensive experiments on CLIP finetuning demonstrate consistent OOD accuracy and calibration gains across three backbone architectures. Comprehensive ablations across four axes (distillation components, regularization strength, update frequency, and kernel shape) confirm that TRACER is both principled and robust to hyperparameter choices.

Social Aspects · Robustness

Pengzhen Chen, Yanwei Liu, Xiaoyan Gu, Antonios Argyriou, Wu Liu, Weiping Wang

Reliable watermarking of panoramic imagery is fundamentally challenged by arbitrary 3D rotations. As panoramas are defined on the sphere, they naturally transform under the action of $SO(3)$, rendering conventional planar representations and augmentation-based robustness strategies inadequate and devoid of theoretical guarantees. To address this, we formulate panoramas as spherical signals and leverage $SO(3)$ representation theory to derive provably rotation-invariant descriptors. While spherical harmonic coefficients transform equivariantly under rotations, the natural invariant constructions are typically limited to zeroth-order statistics which eliminate directional information and severely constrain embedding capacity. In this work, we introduce a principled third-order invariant construction by coupling higher-order $SO(3)$ irreducible representations via tensor products and projecting onto the trivial representation. This yields a spherical invariant bispectrum that preserves phase information while remaining strictly rotation-invariant. Leveraging this property, we embed watermarks into higher-order spherical harmonic coefficients and recover them from invariant bispectral scalars, enabling reliable extraction under arbitrary 3D rotations. We provide a theoretical proof of $SO(3)$ invariance for it and demonstrate experimentally its near-perfect robustness to continuous rotations while maintaining high visual fidelity.

Optimization · Large Scale, Parallel and Distributed

Feihu Huang, Yuning Luo, Songcan Chen

Large models recently are widely applied in machine learning, so efficient training of large models has received widespread attention. More recently, a useful Muon optimizer is specifically designed for matrix-structured parameters of large models. Although some works have begun to studying the Muon optimizer, the existing Muon and its variants still suffer from high sample complexity or high memory for large models. To fill this gap, we propose a light and fast Muon (LiMuon) optimizer for training large models, which builds on the momentum-based variance reduced technique and randomized Singular Value Decomposition (SVD). In particular, our LiMuon optimizer simultaneously has a lower memory and lower sample complexity than the Muon. Moreover, we prove that our LiMuon has a lower sample complexity of $O(\epsilon^{-3})$ for finding an $\epsilon$-stationary solution of non-convex stochastic optimization under the generalized smooth condition. Numerical experimental results on training Mamba-130M, Qwen2.5-0.5B and ViT models demonstrate effectiveness of our LiMuon optimizer.

General Machine Learning · Online Learning, Active Learning and Bandits

Liangxiao Jiang, Liangxiao Jiang, Chaoqun Li, Shanshan Si

In crowdsourcing scenarios, to mitigate the impact of noisy labels assigned by non-expert workers, each instance is typically annotated multiple times by different workers. However, repeated annotation can introduce instance- or label-level redundancy, thereby inflating annotation costs. Despite its practical importance, research on repeated annotation strategies remains limited, and no existing strategy simultaneously avoids being offline, instance-unaware, and model-centric. In this paper, we propose a model-agnostic active annotation strategy, MA$^3$S, that addresses these limitations: (1) To reduce label redundancy caused by offline procedure, MA$^3$S estimates instance uncertainties with a general Beta distribution and updates them online as new labels arrive. (2) To prevent instance redundancy induced by instance-unaware designs, MA$^3$S constructs a nearest-neighbor graph to propagate instance uncertainties, reducing repeated annotations of similar instances. (3) To avoid being model-centric, MA$^3$S actively selects instances for annotation based solely on the estimated uncertainties, without relying on model feedback. Extensive experiments on synthetic and real-world datasets demonstrate that MA$^3$S consistently outperforms existing annotation strategies.

John Yang, Kilian Lieret, Joyce Yang, Carlos Jimenez, Muhtasham Oblokulov, Aryan Siddiqui, Ofir Press, Ludwig Schmidt, Diyi Yang

Existing coding benchmarks evaluate language models (LMs) on concrete, well-specified tasks such as fixing bugs or writing tests. However, human programmers do not spend all day addressing isolated GitHub issues. Instead, real-world software development is grounded in the pursuit of high-level goals. Evaluating whether LMs can iteratively develop code to accomplish open-ended objectives without explicit guidance remains an open challenge. We introduce CodeClash, a benchmark where LMs compete in multi-round tournaments to build the best codebase for achieving a competitive objective. Each round proceeds in two parts: agents edit their code, then their codebases compete head-to-head in a code arena that determines winners based on objectives like score maximization, resource acquisition, or survival. Models must decide for themselves how to improve their code both absolutely and against their opponents. We run 1680 tournaments to evaluate 8 LMs across 6 arenas, revealing how models exhibit diverse development styles and share fundamental limitations in strategic reasoning. Models also struggle with long-term codebase maintenance; repositories become progressively messy and redundant. Top models lose every round against expert human programmers. We open-source CodeClash to advance the study of autonomous, goal-oriented code development.

General Machine Learning · Unsupervised and Semi-supervised Learning

Hongyang He, Yan Zhong, Xinyuan Song, Daizong Liu, Xuanyu Liu, Victor Sanchez

Most semi-supervised learning frameworks rely on a single teacher that transfers zero-order supervision through pseudo-labels, constraining the student to imitate categorical outputs without perceiving the loss geometry. This design often leads to unstable optimization and limited generalization under scarce labels. We propose TTN (Two-Teachers Newton-guided Learning), a dual-teacher framework that integrates complementary supervision from MAE and DINOv3 and optimizes the student through a Newton step update. The two teachers provide multi-scale structural and semantic cues whose pseudo-labels and local Hessians are fused by confidence weighting, forming a unified second-order supervision signal. The student updates parameters preconditioned by the fused curvature, enabling stable convergence and geometry-consistent learning. TTN consistently improves over existing single-teacher and consistency-based semi-supervised learning methods on ImageNet, CIFAR-10, SVHN, and STL-10, demonstrating that combining multi-view self-supervised teachers with curvature-guided optimization yields robust and efficient semi-supervised learning.

Hoang Anh Just, Myeongseob Ko, Ruoxi Jia

Distilling long-form reasoning from teacher models into smaller students requires selecting which candidate solutions to train on. Recent work argues that one should select responses the student model assigns highest probability, i.e., favoring solutions ``natural'' to the student. However, we find that this approach works within a single teacher but fails when scaling to long reasoning traces from multiple diverse teachers. We identify a key cause: this approach scores entire solutions, but students generalize by recombining familiar reasoning steps, not by memorizing complete solutions. Full-trajectory scoring optimizes the wrong target; it rewards global fluency while the transferable signal lies in local step transitions. We propose Local Average Log Probability (LALP), which scores each reasoning step using only a small window of preceding context, measuring whether each step is justified by its immediate premises rather than whether the full response looks natural to the student. LALP enables two practical use cases: selecting the best teacher before fine-tuning and curating training data from diverse teacher pools. Across math, coding, and science reasoning tasks, LALP consistently improves accuracy when selecting the most natural solutions by a large margin.

General Machine Learning · Data

Hoang Anh Just, Myeongseob Ko, Ruoxi Jia

Distilling long-form reasoning from teacher models into smaller students requires selecting which candidate solutions to train on. Recent work argues that one should select responses the student model assigns highest probability, i.e., favoring solutions ``natural'' to the student. However, we find that this approach works within a single teacher but fails when scaling to long reasoning traces from multiple diverse teachers. We identify a key cause: this approach scores entire solutions, but students generalize by recombining familiar reasoning steps, not by memorizing complete solutions. Full-trajectory scoring optimizes the wrong target; it rewards global fluency while the transferable signal lies in local step transitions. We propose Local Average Log Probability (LALP), which scores each reasoning step using only a small window of preceding context, measuring whether each step is justified by its immediate premises rather than whether the full response looks natural to the student. LALP enables two practical use cases: selecting the best teacher before fine-tuning and curating training data from diverse teacher pools. Across math, coding, and science reasoning tasks, LALP consistently improves accuracy when selecting the most natural solutions by a large margin.

Deep Learning · Large Language Models

Janghyeon Kim, Minsoo Kim, Kyuhong Shim, Jungwook Choi

Large Reasoning Models (LRMs) achieve superior problem-solving through extended chain-of-thought generation, but the resulting key-value (KV) cache grows linearly with sequence length and creates severe memory bottlenecks—often exceeding GPU capacity for long reasoning traces. Existing KV cache compression methods rely on recent queries to estimate future token importance, implicitly assuming these serve as reliable proxies for future attention patterns. We demonstrate that this assumption fails in long-horizon reasoning: certain decoding steps generate Thought Revisiting Tokens (TRT) that re-attend to distant previous context, such as task-solving plans formulated early in the trace. Through systematic analysis, we discover that queries corresponding to the TRT cluster into a small number of similarity groups in the embedding space. Based on this insight, we propose BeaconKV, a training-free KV cache compression method that maintains beacon queries—compact representatives for each global query cluster—to anticipate which KV pairs will be revisited without storing the entire query history. We introduce Continual Farthest Point Sampling for memory-efficient beacon identification during inference. Across four open-source LRMs and diverse reasoning benchmarks, BeaconKV consistently outperforms state-of-the-art methods, achieving up to $5.8\times$ memory reduction while nearly preserving full cache accuracy and improving throughput by over $4.3\times$.

Reinforcement Learning · Online

Ryo Iwaki

Regularization is a core component of recent Reinforcement Learning (RL) algorithms. Mirror Descent Value Iteration (MDVI) uses both Kullback-Leibler divergence and entropy as regularizers in its value and policy updates. Despite its empirical success in discrete action domains and strong theoretical guarantees, the performance of KL-entropy-regularized methods does not surpass that of a strong entropy-only-regularized method in continuous action domains. In this study, we propose Mirror Descent Actor Critic (MDAC) as an actor-critic style instantiation of MDVI for continuous action domains, and show that its empirical performance is significantly boosted by bounding the actor's log-probability terms in the critic's loss function, compared to a non-bounded naive instantiation. Further, we relate MDAC to Advantage Learning by recalling that the actor's log-probability is equal to the regularized advantage function in tabular cases, and theoretically discuss when and why bounding the advantage terms is validated and beneficial. We also empirically explore effective choices for the bounding functions, and show that MDAC performs better than strong non-regularized and entropy-only-regularized methods with an appropriate choice of the bounding functions.

Social Aspects · Security

Qiushi Wu, Yue Xiao, Dhilung Kirat, Kevin Eykholt, Jiyong Jang, Douglas Schales

Recurring Pattern Bugs (RPBs) are defined as bugs where a single root cause appears repeatedly across multiple code segments. These bugs remain a persistent security threat even after individual instances are patched. Various static analyzers exist for finding specific bug patterns but require significant engineering effort and fail to generalize well beyond their predefined template, preventing them from detecting RPBs. To tackle RPBs, we introduce BugStone, a hybrid framework combining LLVM-based program analysis with Large Language Models to automate RPB detection. BugStone leverages a single patched instance to synthesize abstract error patterns and retrieves semantically similar bugs throughout the codebase. To evaluate BugStone, we create a ground truth dataset by analyzing over 1.9K security bugs reports, on which BugStone achieves 92.2% precision and 79.1% pairwise accuracy. We further validated BugStone through a large-scale real-world deployment. In the Linux kernel, BugStone identified over 22K potential issues; a manual audit of 400 samples confirmed 246 valid bugs, including invalid pointer dereferences, resource leaks, type errors, performance issues, and others. To evaluate the generalizability of BugStone, we further applied it to the top 100 Python projects, discovering multiple critical command injection vulnerabilities.

Applications · Robotics

Fengshuo Bai, Yufeng Li, Ruihai Wu, Peishuo Wang, Yuhan Wang, Bernie Zhu, Yuanfei Wang, Tawei Chou, Gao, Runchuan Zhu 等

Laboratory automation driven by scientific embodied agents represents a critical frontier in modern laboratories. Unlike conventional robotic domains, laboratory environments impose zero-tolerance constraints on manipulation precision and collision, as minor deviations can lead to irreversible chemical hazards or equipment damage. This naturally makes the automated laboratory an ideal testbed for advancing embodied safety. However, existing benchmarks predominantly feature high-tolerance manipulation tasks where intermediate failures are largely reversible. More critically, current Vision-Language-Action (VLA) models trained via static imitation learning cannot satisfy these strict constraints. Because they merely mimic successful demonstrations, they lack the ability to recover from execution drift, leading to catastrophic compounding errors in precision-critical domains. Overcoming this limitation requires transitioning from static datasets to interactive environments that support Reinforcement Learning (RL) for dynamic error recovery. To this end, we introduce SafeLab, a generative simulation benchmark designed for the full lifecycle of safe robot learning. Grounded in a high-fidelity chemistry lab, our framework integrates an {LLM engine} for procedural task synthesis, an {automated expert} for scalable demonstration collection, and an {interactive environment} for continuous RL refinement. Leveraging this infrastructure, we release a dataset of 6,000+ {complex trajectories} to evaluate state-of-the-art VLA models. Experiments reveal that current embodied agents fail significantly under these safety constraints. In contrast, our RL post-training pipeline enables agents to learn active error correction, mitigating hazardous failures and improving success rates by 37\%, thereby establishing SafeLab as a critical platform for developing reliable and safe generalist agents.