论文检索

Deep Learning · Large Language Models

Ziniu Li, Congliang Chen, Tianyun Yang, Tian Ding, Ruoyu Sun, Ge Zhang, Wenhao Huang, Zhiquan Luo

Large Language Models (LLMs) can improve via reinforcement learning by generating trajectories to discover better solutions. This exploration process represents an investment of finite GPU compute to obtain learning signals. However, current methods typically allocate a small, uniform budget to every task, which is inefficient and ineffective: easy tasks consistently succeed while difficult tasks consistently fail. For policy optimization algorithms such as Group Relative Policy Optimization (GRPO), both edge cases produce zero gradients, resulting in wasted computation. We address this by reframing exploration budget allocation as a resource optimization problem. Viewing each task's exploration as an `"item'' with a distinct "learning value'' and "computational cost'', we establish a connection to the classical knapsack problem and derive an optimal assignment rule. When applied to GRPO, our method increases the ratio of effective gradients by 40\%. As a computational "free lunch'', it enables substantially larger budgets (e.g., 93) for challenging tasks—allocations that would be expensive under a uniform allocation framework. These efficiency gains translate to meaningful improvements on mathematical reasoning benchmarks, with average gains of 2--4 points and peak gains of 9 points. Notably, achieving comparable performance with traditional homogeneous allocation would require approximately $2\times$ the computational resources.

Deep Learning · Other Representation Learning

Xiaolong Chen, Fuyuan Xiao, Xiaohong Zhang, Zehong Cao, Chin-teng Lin

Out-of-Distribution (OOD) detection requires accurately classifying In-Distribution (ID) samples while effectively distinguishing anomalous OOD data. However, existing methodologies predominantly rely on real-valued magnitude features, neglecting the semantic richness embedded in phase information, and often lack a systematic theoretical framework for quantitively modeling uncertainty. To address this dual limitation of incomplete feature representation and insufficient uncertainty modeling, the Trustworthy Quantum Evidence Neural Network (TrustworthyQENN) is proposed, a novel framework bridging complex-valued representation learning with Generalized Quantum Evidence Theory (GQET). Specifically, Supervised Complex-Valued Contrastive Learning (SCVCL) is proposed to synchronize amplitude distributions with phase correlations, thereby enforcing high intra-class compactness and inter-class separability for ID data. A quantum evidence generation mechanism based on GQET is subsequently devised, where the OOD state is formally grounded as the quantum empty set within a Hilbert space. Furthermore, the Generalized Quantum Evidential Combination Rule (GQECR) is leveraged to fuse multi-view evidence, thereby achieving trustworthy inference. Extensive experiments on the MSTAR, EuroSAT, and FUSAR-Ship benchmarks substantiate the superiority of TrustworthyQENN, which achieves a peak AUROC of 95.94% on the MSTAR dataset while consistently outperforming state-of-the-art methods across all evaluated scenarios.

Applications · Robotics

Guoxin Lian, Shuo Wang, Yucheng Wang, Yongcai Wang, Maiyue Chen, kaihui.wang, Bo Zhang, Zhizhong Su, Deying Li, Zhaoxin Fan

Vision-Language Navigation (VLN) requires agents to follow natural language instructions in partially observed 3D environments, motivating map representations that aggregate spatial context beyond local perception. However, most existing approaches rely on hand-crafted maps constructed independently of the navigation policy. We argue that maps should instead be learned representations shaped directly by navigation objectives rather than exhaustive reconstructions. Based on this insight, we propose MapDream, a map-in-the-loop framework that formulates map construction as autoregressive bird’s-eye-view (BEV) image synthesis. The framework jointly learns map generation and action prediction, distilling environmental context into a compact three-channel BEV map that preserves only navigation-critical affordances. Supervised pre-training bootstraps a reliable mapping-to-control interface, while the autoregressive design enables end-to-end joint optimization through reinforcement fine-tuning. Experiments on R2R-CE and RxR-CE achieve state-of-the-art monocular performance, validating task-driven generative map learning.

Deep Learning · Theory

Bolin Shen, Zhan Cheng, Neil Gong, Fan Yao, Yushun Dong

Machine Learning as a Service (MLaaS) has become a widely adopted method for delivering deep neural network (DNN) models, allowing users to conveniently access models via APIs. However, such services have been shown to be highly vulnerable to Model Extraction Attacks (MEAs). While numerous defense strategies have been proposed, verifying the ownership of a suspicious model with strict theoretical guarantees remains a challenging task. To address this gap, we introduce CREDIT a certified defense against MEAs. Specifically, we employ mutual information to quantify the similarity between DNN models, propose a practical verification threshold, and provide rigorous theoretical guarantees for ownership verification based on this threshold. We extensively evaluate our approach on several mainstream datasets and achieve state-of-the-art performance. Our implementation is publicly available at: \url{https://anonymous.4open.science/r/CREDIT}.

General Machine Learning · Evaluation

Xiangyu Jiang

Large language models increasingly rely on retrieval-augmented generation (RAG) to ground responses in external corpora. Yet, even with strong retrievers, generated statements can remain unsupported, and the resulting citations are often not reliable indicators of evidence. We introduce CiteGuard, a RAG decoding layer that treats sentence-level factuality as a multiple-testing problem and combines conformal calibration with false-discovery-rate control. CiteGuard converts claim–evidence scores into p-values for the null hypothesis "unsupported" and uses BH/BY procedures to decide which claims to keep (with citations) and which to abstain on. On FEVER and Natural Questions, CiteGuard reduces the false-discovery rate among accepted claims from 28–31% (vanilla RAG) to below 10% at α=0.10, while retaining 86–92% of supported claims. This yields a user-controlled risk budget: practitioners can trade off faithfulness and coverage via α, with finite-sample guarantees under standard exchangeability assumptions.

Deep Learning · Attention Mechanisms

Zhuoqing Song, Peng Sun, Huizhuo Yuan, Quanquan Gu

In standard causal attention, each token's query, key, and value (QKV) are static and encode only preceding context. We introduce CAuSal aTtention with Lookahead kEys (CASTLE), an attention mechanism that continually updates each token's keys as the context unfolds. We term these updated keys lookahead keys because they belong to earlier positions yet integrate information from tokens that appear later relative to those positions, while strictly preserving the autoregressive property. Although the mechanism appears sequential, we derive a mathematical equivalence that avoids explicitly materializing lookahead keys at each position and enables efficient parallel training. On language modeling benchmarks, CASTLE consistently outperforms standard causal attention across model scales, reducing validation perplexity and improving performance on a range of downstream tasks.

Deep Learning · Attention Mechanisms

Timon Klein, Jonas Kusch, Sebastian Sager, Stefan Schnake, Steffen Schotthöfer

The pursuit of reducing the memory footprint of the self-attention mechanism in multi-headed self attention (MHA) spawned a rich portfolio of methods, e.g., group-query attention (GQA) and multi-head latent attention (MLA). The methods leverage specialized low-rank factorizations across embedding dimensions or attention heads. From the point of view of classical low-rank approximation, these methods are unconventional and raise questions of which objects they really approximate and how to interpret the low-rank behavior of the resulting representations. To answer these questions, this work proposes a generalized view on the weight objects in the self-attention layer and a factorization strategy, which allows us to construct a parameter efficient scheme, called Tucker Attention. Tucker Attention requires an order of magnitude fewer parameters for comparable validation metrics, compared to GQA and MLA, as evaluated in LLM and ViT test cases. Additionally, Tucker Attention~encompasses GQA, MLA, MHA as special cases and is fully compatible with flash-attention and rotary position embeddings (RoPE). This generalization strategy yields insights of the actual ranks achieved by MHA, GQA, and MLA, and further enables simplifications for MLA.

Reinforcement Learning · Deep RL

Shaojun Xu, Xiaoling Zhou, Yihan Lin, Yapeng Meng, Xinglong Ji, Luping Shi, Rong Zhao

Model-Based Reinforcement Learning (MBRL) leverages latent imagination for sample efficiency, yet remains constrained by **Historical Tethering**: imagination is typically initialized from observed states. This creates a learning asymmetry, where the world model’s manifold discovery outpaces the policy's sparse-reward optimization. We propose **Mind Dreamer (MD)**, a framework that operationalizes **Active Counterfactual Reasoning (ACR)** to transcend Markovian continuity. MD reformulates discovery as the minimization of a global Relay Manifold Expected Free Energy (R-EFE); by invoking a latent-space $do$-operator, MD utilizes an adversarial generator to synthesize non-continuous **latent jumps** to epistemic blind spots that are physically plausible yet cognitively challenging. To resolve the credit assignment paradox across these spatial ruptures, we derive the **Relay Value Function (RVF)** and **Relay Uncertainty Function (RUF)**. These potentials treat synthesized anchors as latent bridges, propagating pragmatic and epistemic value through a principled Bellman-style formulation. Notably, we prove that the Uncertainty propagation across discontinuities necessitates a quadratic discount $\gamma^2$, establishing a formal epistemic horizon. Theoretically, MD acts as an optimal importance sampler that expands the manifold's spectral gap, reducing the hitting time to critical bottleneck states. Empirically, MD achieves a **1.67$\times$ average speedup** over DreamerV3 on DeepMind Control Suite, reaching **8.8$\times$** in sparse-reward tasks.

Probabilistic Methods · Monte Carlo and Sampling Methods

Benjamin Patrick Evans, Sumitra Ganesh, Leo Ardon

Large language models (LLMs) achieve remarkable generative performance, yet their output quality is dependent on the decoding strategy. While sampling-based methods (e.g., top-$k$, nucleus) and search-and-select based methods (e.g., beam search, best-of-$n$, majority voting) can improve upon greedy decoding, both approaches suffer from limitations: sampling commits to a single path, while search often expends excessive computation regardless of task complexity. We introduce **Entropy-informed DEcodiNg** (EDEN), a plug-and-play, model-agnostic decoding framework that adaptively allocates computation based on the model’s own uncertainty, approximating higher width beam search with *fewer generations required*. At each generation step, EDEN estimates the entropy of the output token distribution and adjusts the branching factor monotonically with the entropy, expanding more candidates in high-entropy regions and following a greedier path in low-entropy regions, improving sample efficiency. Experiments across complex tasks, including mathematical reasoning, code generation, and scientific questions, demonstrate that EDEN consistently improves output quality over existing decoding strategies, achieving better trade-offs between accuracy and token generations than fixed beam search approaches. By treating next token selection as a noisy maximisation problem, we prove that branching factors monotone in entropy are guaranteed to find better (i.e. more probable) continuations than any fixed branching factor within the same total computation budget, motivating the dynamic branching.

Social Aspects · Accountability, Transparency, and Interpretability

Charles Ye, Jasmine Cui, Dylan Hadfield-Menell

Language models remain vulnerable to prompt injection attacks despite extensive safety training. We trace this failure to role confusion: models assign roles based on how text sounds, not where it actually comes from. We design novel role probes to capture how models internally identify “who is speaking.” These reveal why prompt injection works: untrusted text that imitates a role inherits that role’s authority. We test this insight by injecting fabricated reasoning into user prompts and tool outputs, achieving average success rates of 60% on StrongREJECT and 61% on agent exfiltration, across multiple open- and closed-weight models with near-zero baselines. Strikingly, the degree of internal role confusion strongly predicts attack success before generation begins. Our findings reveal a fundamental gap: security is defined at the interface but authority is assigned in latent space. More broadly, we introduce a unifying, mechanistic framework for prompt injection, demonstrating that diverse prompt-injection attacks exploit the same underlying role-confusion mechanism. Code available at: https://anonymous.4open.science/r/role-science-B522.

Deep Learning · Foundation Models

Sujung Hong, Chanyong Yoon, Seong Jae Hwang

Large diffusion vision–language models (LDVLMs) have recently demonstrated competitive performance on multimodal tasks, emerging as a promising alternative to autoregressive models. They enable parallel decoding for efficient inference and leverage bidirectional attention to capture global context. Despite these advances, their behavior under long-form generation remains underexplored. In this work, we show that existing LDVLMs suffer from repetitive generation and lead to degraded visual grounding. Through analysis, we identify two underlying causes of these failures. First, repetitive generation originates from a mask token prior. Because generation tokens are initialized as mask tokens, their hidden representations progressively drift toward a shared prior direction over generation steps. Second, a fundamental misalignment exists between the positional attention bias and the iterative unmasking process. This discrepancy suppresses the model's attention toward informative visual tokens, leading to degradation in visual grounding. Based on these insights, we propose a training-free approach that mitigates both issues. Specifically, we introduce Mask Prior Suppression and Monotonic RoPE Scaling, which mitigate mask prior drift and positional attention collapse during decoding. Experiments on general multimodal benchmarks and visual grounding tasks demonstrate improvements over baseline LDVLMs, with robust gains on long-form description benchmarks. Overall, our results show that these failures can be effectively addressed with a lightweight, plug-and-play strategy that requires no additional training and generalizes across diverse LDVLM architectures.

Reinforcement Learning · Everything Else

Fang YUAN, Quanjun Yin, Siqi Shen, Yuxiang Xie, Junqiang Yang, Long Qin, Junjie Zeng, Qinglun Li

Unsupervised Environment Design (UED) offers a promising paradigm for improving reinforcement learning generalization by adaptively shaping training environments, but it requires reliable environment evaluation to remain effective. However, existing UED methods evaluate environments using indirect proxy signals such as regret, value-based errors, or Monte Carlo, which suffer from bias, high variance, or substantial computational overhead and fail to reflect agent realized learning progress. To address these limitations, we propose Parameter Change Environment Design (PACE), which evaluates an environment through the policy parameter change induced by training on that environment, directly grounding environment selection in realized learning progress. Specifically, PACE assigns environment value using a first-order approximation of the policy optimization objective, where the improvement induced by an environment is proportional to the squared $\ell_2$ norm of the corresponding parameter update, enabling low-variance and computation-efficient evaluation without additional rollouts. Experiments on MiniGrid and Craftax show that PACE consistently outperforms established UED baselines, achieving higher IQM and smaller Optimality Gap on OOD evaluations, including an IQM of 96.4% and an Optimality Gap of 17.2% on MiniGrid.

Social Aspects · Safety

Nivya Talokar, Ayush Kumar Tarun, Murari Mandal, Maksym Andriushchenko, Antoine Bosselut

LLM-based agents increasingly execute real-world workflows via tools and memory. Granting LLMs such powers enables ill-intended adversaries to likewise use these agents to carry out complex misuse scenarios. Existing agent-misuse benchmarks largely test single-prompt instructions, leaving a gap in measuring how agents end up helping with harmful or illegal tasks over multiple turns. We introduce **STING** (*Sequential Testing of Illicit N-step Goal execution*), an automated red-teaming framework that constructs a step-by-step illicit plan grounded in a benign persona and iteratively probes a target agent with adaptive follow-ups, using judge agents to track phase completion. We further introduce an analysis framework that models multi-turn red-teaming as a time-to-first-jailbreak random variable, enabling analysis tools like discovery curves, hazard-ratio attribution by attack language, and a new metric: Restricted Mean Jailbreak Discovery. Across AgentHarm scenarios, STING yields substantially higher illicit-task completion than single-turn prompting and chat-oriented multi-turn baselines adapted to tool-using agents. In multilingual evaluations across six non-English settings, we find that attack success and illicit-task completion do *not* consistently increase in lower-resource languages, diverging from common chatbot findings. Overall, STING provides a practical way to evaluate and stress-test agent misuse in realistic deployment settings, where interactions are inherently multi-turn and often multilingual.

Deep Learning · Generative Models and Autoencoders

Hao Fu, Tianyu Su, Meng Liu, Chenfang Yang, Tian Gan

Diffusion models have shown impressive capabilities in text-to-image synthesis, but multi-concept personalized generation remains challenging, particularly in aligning multiple reference concepts while preserving fidelity. In this work, we propose a novel framework that addresses this challenge with a two-stage Sketch-to-Rendering process, utilizing Dual Optimal Transport (OT) for structural alignment and texture injection. Our approach consists of two key components: Structural Guidance via OT: Ensures shape alignment by using mass-preserving OT for spatial consistency, and Texture Injection via Geometry-Guided OT: Leverages low-frequency structure alignment to inject high-frequency texture details via OT-based residual transfer, preserving texture fidelity without distorting structure. Extensive experiments demonstrate that our method significantly enhances both conceptual fidelity and visual quality in multi-concept generation. Ablation studies further confirm the effectiveness of the proposed optimal transport guidance and the decoupling of structure and texture during the generation process.

Deep Learning · Theory

Konstantin Nikolaou, Jonas Scheunemann, Sven Krippendorf, Samuel Tovey, Christian Holm

Neural scaling laws describe predictable power-law relationships between model size, dataset size, compute cost, and performance. While these laws are applied to improve the performance of modern foundation models, the mechanisms underpinning them are less understood, in part due to the absence of scalable analysis tools. To this end, we introduce a framework for efficiently measuring the alignment between the empirical neural tangent kernel (eNTK) and loss residuals. Applying this framework to scaling experiments reveals a consistent pattern: larger and better-performing models exhibit *lower* kernel alignment throughout training. We interpret this *unalignment* through the lens of *spectral reach*: the capacity of a model to learn from progressively weaker spectral modes in its eNTK. This interpretation allows us to explain why larger models achieve lower losses: they sustain learning on weaker signals that smaller models cannot access. We further demonstrate that feature learning improves spectral reach and provide a mechanistic explanation of how this occurs, suggesting practical avenues for performance improvement.

Deep Learning · Generative Models and Autoencoders

Tong Yang, Moonkyung Ryu, Chih-wei Hsu, Guy Tennenholtz, Yuejie Chi, Craig Boutilier, Bo Dai

Controllable diffusion generation often relies on various heuristics that are seemingly disconnected without a unified understanding. We bridge this gap with Diffusion Controller (DiffCon), a unified control-theoretic view that casts reverse diffusion sampling as state-only stochastic control within (generalized) linearly-solvable Markov Decision Processes (LS-MDPs). Under this framework, control acts by reweighting the pretrained reverse-time transition kernels, balancing terminal objectives against an $f$-divergence cost. From the resulting optimality conditions, we derive practical reinforcement learning methods for diffusion fine-tuning: (i) $f$-divergence-regularized policy-gradient updates, including a PPO-style rule, and (ii) a regularizer-determined reward-weighted regression objective with a minimizer-preservation guarantee under the Kullback–Leibler (KL) divergence. The LS-MDP framework further implies a principled model form: the optimal score decomposes into a fixed pretrained baseline plus a lightweight control correction, motivating a side-network parameterization conditioned on exposed intermediate denoising outputs, enabling effective \emph{gray-box adaptation} with a frozen backbone. Experiments on Stable Diffusion v1.4 across supervised and reward-driven finetuning show consistent gains in preference-alignment win rates and improved quality–efficiency trade-offs versus gray-box baselines and even the parameter-efficient white-box adapter LoRA.

Deep Learning · Generative Models and Autoencoders

Jaeyeon Kim, Seunggeun Kim, Taekyun Lee, David Pan, Hyeji Kim, Sham Kakade, Sitan Chen

A natural desideratum for generative models is \emph{self-correction}--detecting and revising low-quality tokens at inference. While Masked Diffusion Models (MDMs) have emerged as a promising approach for generative modeling in discrete spaces, their capacity for self-correction remains poorly understood. Prior attempts to incorporate self-correction into MDMs either require overhauling MDM architectures/training or rely on imprecise proxies for token quality, limiting their applicability. Motivated by this, we introduce PRISM--Plug-in Remasking for Inference-time Self-correction of Masked Diffusions--a lightweight, model-agnostic approach that applies to any pretrained MDM. Theoretically, PRISM defines a self-correction loss that provably learns per-token quality scores, without RL or a verifier. These quality scores are computed in the same forward pass with MDM and used to detect low-quality tokens. Empirically, PRISM advances MDM inference across domains and scales: Sudoku; unconditional text (170M); and code with LLaDA (8B).

Reinforcement Learning · Deep RL

Haitong Ma, Ofir Nabati, Aviv Rosenberg, Bo Dai, Oran Lang, Craig Boutilier, Na Li, Shie Mannor, Lior Shani, Guy Tennenholtz

Reinforcement learning (RL) struggles to scale to large, combinatorial action spaces common in many real-world problems. This paper introduces a novel framework for training discrete diffusion models as highly effective policies in these complex settings. Our key innovation is an efficient online training process that ensures stable and effective policy improvement. By leveraging policy mirror descent (PMD) to define an ideal, regularized target policy distribution, we frame the policy update as a distributional matching problem, training the expressive diffusion model to replicate this stable target. This decoupled approach stabilizes learning and significantly enhances training performance. Our method achieves state-of-the-art results and superior sample efficiency across a diverse set of challenging combinatorial benchmarks, including DNA sequence generation, RL with macro-actions, and multi-agent systems. Experiments demonstrate that our diffusion policies attain superior performance compared to other baselines. Crucially, our extensive empirical analysis reveals a key trade-off: FKL demonstrates superior sample efficiency and faster initial convergence, whereas RKL ensures greater training stability and higher asymptotic performance.

Optimization · Non-Convex

Shutong Ding, Yimiao Zhou, Ke Hu, Xi Yao, Junchi Yan, Xiaoying Tang, Ye Shi

Recent advances in diffusion models show promising potential to accelerate nonconvex problem solving by leveraging their multimodality. However, most existing diffusion-based optimization approaches rely on supervised learning and lack a mechanism to enforce constraint satisfaction, which is required in real-world applications. In that case, we investigate and theoretically analyze the inherent problem of supervised diffusion solvers and identify the distributional misalignment problem, i.e., the generated solution distribution often exhibits low probability mass on the feasible region. To resolve this issue, we propose DiOpt, a new diffusion-based learning framework for constrained nonconvex optimization, which effectively learns the mapping from noise to the constraint region. Specifically, this framework operates in two distinct phases: an initial warm-start phase, implemented via supervised learning, followed by a bootstrapping training phase. This dual-phase architecture is designed to iteratively refine solutions, thereby improving the objective function with high constraint satisfaction. Finally, we also employ a solution selection technique in inference for better optimality. Notably, DiOpt is the first successful integration of the diffusion solver in constrained nonconvex optimization. Evaluations on diverse nonconvex tasks demonstrate the superiority of DiOpt in both optimality and constraint satisfaction.

Reinforcement Learning · Deep RL

Youssef Mroueh

Group Relative Policy Optimization (GRPO) was introduced recently and used to train DeepSeek\textendash R1 for promoting reasoning in LLMs under verifiable (binary) rewards. We show that the mean{+}variance calibration of these rewards induces a contrastive loss in which the contrastive samples are synthetic data drawn from the previous policy. While GRPO was originally paired with clipping to keep updates near the old policy, we analyze variants that differ in reward normalization (mean-only vs.\ mean{+}variance) and in how they regularize updates using KL divergence: either penalizing divergence from the previous model (\emph{mirror}), penalizing divergence from a fixed reference model $\pi_{\mathrm{ref}}$, or combining both forms of regularization. For each, the optimal policy $\pi_n$ admits an explicit form in terms of the binary reward and the first and second order statistics of the reward under $\pi_{n-1}$, as well as the policies $\pi_{n-1}$ and $\pi_{\mathrm{ref}}$. Iterating results in a sequence $\{\pi_n\}$ whose \emph{probability of success (PoS)} obeys a simple recurrence that converges to a fixed point determined by the reference PoS and the regularization strength. We further show that this fixed point exceeds the reference, demonstrating that GRPO amplifies the policy's probability of success.