论文检索

Reinforcement Learning · Deep RL

Jing-Cheng Pang, Sun Liu, Chang Zhou, Xian Tang, Haichuan Ma, KUN JIANG, Jianlong Wang, Kai Zhang, Sijie Wu, Haoran Cai 等

Domain-specific large language models (LLMs), typically developed by fine-tuning a pre-trained general-purpose LLM on specialized datasets, represent a significant advancement in applied AI. A common strategy in LLM fine-tuning is curriculum learning, which pre-orders training samples based on metrics like difficulty to improve learning efficiency compared to a random sampling strategy. However, most existing methods for LLM fine-tuning rely on a static curriculum, designed prior to training, which lacks adaptability to the model's evolving needs during fine-tuning. To address this, we propose EDCO, a novel framework based on two key concepts: \textit{inference entropy} and \textit{dynamic curriculum orchestration}. Inspired by recent findings that maintaining high answer entropy benefits long-term reasoning gains, EDCO prioritizes samples with high inference entropy in a continuously adapted curriculum. EDCO integrates three core components: an efficient entropy estimator that uses prefix tokens to approximate full-sequence entropy, an entropy-based curriculum generator that selects data points with the highest inference entropy, and an LLM trainer that optimizes the model on the selected curriculum. Comprehensive experiments in communication, medicine and law domains, EDCO outperforms traditional curriculum strategies for fine-tuning Qwen3-4B and Llama3.2-3B models under supervised and reinforcement learning settings. Furthermore, the proposed efficient entropy estimation reduces computational time by 83.5\% while maintaining high accuracy.

Deep Learning · Large Language Models

Wei Zhang, Lintong Du, yuanhe zhang, Zhenhong Zhou, Kun Wang, Li Sun, Sen Su

Despite the strong performance of Large Language Models (LLMs) on complex instruction-following tasks, precise control of output length remains a persistent challenge. Existing methods primarily attempt to enforce length constraints by externally imposing length signals or optimization objectives, while largely overlooking the underlying limitation: the model's intrinsic deficit in length cognition. To address this, we propose \textbf{LARFT} (\textbf{L}ength-\textbf{A}ware \textbf{R}einforcement \textbf{F}ine-\textbf{T}uning), a training framework that aligns the model's length cognition with its action. Specifically, LARFT integrates length-oriented reinforcement learning with a hindsight length awareness. By transforming on-policy data into hindsight self-awareness tasks where the model learns to identify the actual length of its own generation, LARFT jointly optimizes the model’s internal representation of length information and refines its policy to satisfy length constraints, thereby achieving precise and reliable length instruction following. Extensive experiments across four base models demonstrate that LARFT outperforms existing baselines, achieving an average improvement of \textbf{+20.92} points across three length instruction following benchmarks with only a marginal decline of \textbf{-1.45} points on four general capability benchmarks.

Social Aspects · Security

Zhifang Zhang, Qiqi Tao, JIAQI LYU, Na Zhao, Lei Feng, Joey Tianyi Zhou

Large vision-language models (LVLMs) excel at vision-language tasks but remain vulnerable to backdoor attacks. Most existing backdoor attacks on LVLMs force the model to generate predefined target patterns. However, these fixed-pattern attacks are easy to detect, as the model tends to memorize frequent patterns and exhibits overconfidence on targets given poisoned inputs. To address these limitations, we introduce TokenSwap, a more evasive and stealthy backdoor attack that focuses on the \emph{compositional understanding} capabilities of LVLMs. Instead of enforcing a fixed targeted content, TokenSwap subtly disrupts the understanding of object relationships in text. Specifically, it causes the backdoored model to generate outputs that mention the correct objects in the image but misrepresent their relationships (i.e., bags-of-words behavior). During training, TokenSwap injects a visual trigger into selected samples while swapping the grammatical roles of key tokens in the textual answers. Since the poisoned samples differ only subtly from clean ones, an adaptive token-weighted loss is employed to emphasize learning on swapped tokens, strengthening the association between visual triggers and the bags-of-words behavior. Extensive experiments demonstrate that TokenSwap achieves high attack success rates while maintaining evasiveness and stealthiness across multiple benchmarks and LVLM architectures.

Probabilistic Methods · Bayesian Models and Methods

Shaorong Zhang, Rob Brekelmans, Greg Ver Steeg

Diffusion Posterior Sampling (DPS) provides a principled Bayesian approach to inverse problems by sampling from $p(x_0 \mid y)$. While posterior sampling is valuable for capturing uncertainty and multi-modality, many classical and practical inverse problem settings ultimately prioritize accurate point estimation—most notably the MAP estimator, which has long served as a standard reconstruction objective in imaging and scientific applications. We introduce Local MAP Sampling (LMAPS), a new inference framework that iteratively solving local MAP subproblems along the diffusion trajectory. This perspective clarifies their connection to global MAP and DPS, offering a unified probabilistic interpretation for optimization-based methods. Building on this foundation, we develop practical algorithms with a covariance approximation motivated by Gaussian prior assumption, a reformulated objective for stability and interpretability. Across a broad set of image restoration and scientific tasks, LMAPS achieves state-of-the-art performance.

Theory · Probabilistic Methods

Beatrix M. G. Nielsen, Emanuele Marconato, Luigi Gresele, Andrea Dittadi, Simon Buchholz

For a broad family of discriminative models that includes autoregressive language models, identifiability results imply that if two models induce the same conditional distributions, then their internal representations agree up to an invertible linear transformation. We ask whether an analogous conclusion holds approximately when the distributions are close instead of equal. Building on the observation of Nielsen et al. (2025) that closeness in KL divergence need not imply high linear representational similarity, we study a distributional distance based on logit differences and show that closeness in this distance does yield linear similarity guarantees. Specifically, we define a representational dissimilarity measure based on the models’ identifiability class and prove that it is bounded by the logit distance. We further show that, when model probabilities are bounded away from zero, KL divergence upperbounds logit distance; yet the resulting bound fails to provide nontrivial control in practice. As a consequence, KL-based distillation can match a teacher’s predictions while failing to preserve linear representational properties, such as linear-probe recoverability of human-interpretable concepts. In distillation experiments on synthetic and image datasets, logit-distance distillation yields students with higher linear representational similarity and better preservation of the teacher’s linearly recoverable concepts.

Applications · Robotics

Yihong Guo, Dongqiangzi Ye, Sijia Chen, Anqi Liu, Xianming Liu

Autonomous driving requires safe planning, but most learning-based planners lack explicit self-correction ability: once an unsafe action is proposed, there is no mechanism to correct it. Thus, we propose CorrectionPlanner, an autoregressive planner with self-correction that models planning as motion-token generation within a propose, evaluate, and correct loop. At each planning step, the policy proposes an action, namely a motion token, and a learned collision critic predicts whether it will induce a collision within a short horizon. If the critic predicts a collision, we retain the sequence of historical unsafe motion tokens as a self-correction trace, generate the next motion token conditioned on it, and repeat this process until the safe motion token is proposed or the safety criterion is met. This self-correction trace, consisting of all the unsafe motion tokens, represents the planner’s correction process in motion-token space (analogous to reasoning trace in language models). We train the planner with imitation learning followed by model-based reinforcement learning using rollouts from a pretrained world model that realistically models agents' reactive behaviors. Closed-loop evaluations show that CorrectionPlanner reduces the collision rate by over 20% on Waymax and obtains state-of-the-art planning scores on nuPlan.

Applications · Computer Vision

Jihyun Lee, Changmin Lee, Donghwan Kim, Tae-Kyun Kim

While existing methods for reconstructing hand–object interactions have made impressive progress, they either focus on rigid or part-wise rigid objects—limiting their ability to model real-world objects (e.g., cloth, stuffed animals) that exhibit highly non-rigid deformations—or model deformable objects without full 3D hand reconstruction. To bridge this gap, we present PhysHandi (Physics-based Reconstruction of Hand and Deformable Object Interactions), a framework that enables full 3D reconstruction of both interacting hands and non-rigid objects. Our key idea is to physically simulate object deformations driven by forces induced from densely reconstructed 3D hand motions, ensuring that the reconstructed object dynamics are both physically plausible and coherent with the interacting hand movements. Furthermore, we demonstrate that such simulation of object deformations can, in turn, refine and improve hand reconstruction via inverse physics. In experiments, PhysHandi outperforms the state-of-the-art baseline across reconstruction and future prediction.

Applications · Robotics

Ruiheng Liu, Haihong Hao, Mingfei Han, Xin Gu, Kecheng Zhang, Changlin Li, Xiaojun Chang

Advancing towards artificial superintelligence requires rich and intelligent perceptual capabilities. A critical frontier in this pursuit is overcoming the limited spatial understanding of Multimodal Large Language Models (MLLMs), where geometry information is essential. Existing methods often address this by rigidly injecting geometric signals into every input, while ignoring their necessity and adding computation overhead. Contrary to this paradigm, our framework endows the model with an awareness of perceptual insufficiency, empowering it to autonomously engage geometric features in reasoning when 2D cues are deemed insufficient. To achieve this, we first introduce an independent geometry input channel to the model architecture and conduct alignment training, enabling the effective utilization of geometric features. Subsequently, to endow the model with perceptual awareness, we curate a dedicated spatial-aware supervised fine-tuning dataset. This serves to activate the model’s latent internal cues, empowering it to autonomously determine the necessity of geometric information. Experiments across multiple spatial reasoning benchmarks validate this approach, demonstrating significant spatial gains without compromising 2D visual reasoning capabilities, offering a path toward more robust, efficient and self-aware multi-modal intelligence.

Deep Learning · Generative Models and Autoencoders

Yuanbin Man, Ying Huang, Zhile Ren, Miao Yin

4D generation, or dynamic 3D content generation, integrates spatial, temporal, and view dimensions to model realistic dynamic scenes, playing a foundational role in advancing world models and physical AI. However, maintaining long-chain consistency across both frames and viewpoints through the unique spatio-camera-motion (SCM) attention mechanism introduces substantial computational and memory overhead, often leading to out-of-memory (OOM) failures and prohibitive generation times. To address these challenges, we propose Turbo4DGen, an ultra-fast acceleration framework for diffusion-based multi-view 4D content generation. Turbo4DGen introduces a spatiotemporal cache mechanism that persistently reuses intermediate attention across denoising steps, combined with dynamically semantic-aware attention pruning and an adaptive SCM chain bypass scheduler, to drastically reduce redundant SCM attention computation. Our experimental results show that Turbo4DGen achieves an average 9.7$\times$ speedup without quality degradation on the ObjaverseDy and Consistent4D datasets. To the best of our knowledge, Turbo4DGen is the first dedicated acceleration framework for 4D generation.

Social Aspects · Accountability, Transparency, and Interpretability

Joshua Cooper, Christopher Nassif

Distinguishing Large Language Model (LLM) generated text from human writing is a critical and difficult challenge. While LLMs are trained to write like humans, we hypothesize that this training leaves an indelible mark. LLMs develop a particularly strong aversion to token repetition very early in training. This bias persists as a ``Vestigial Heuristic'' (a developmental artifact) that is activated in LLM-generated text, separating LLM from human writing. To probe this phenomenon, we introduce Telescope Perplexity, a metric that evaluates the token repetition of the model, $P(s_i | s_{1:i})$. Our empirical investigation reveals that the Telescope Perplexity signature emerges early in pre-training, and Telescope Perplexity empirically enables highly effective zero-shot LLM detection. We show state-of-the-art or competitive performance across diverse datasets (including modern evaluation sets we introduce), reference models, and perturbation schemes with greater efficiency than other methods.

Social Aspects · Alignment

Florian Dietz, William Wale, Oscar Gilg, Robert McCarthy, Felix Michalak, Gustavo Danon, Miguelito De Guzman, Dietrich Klakow

Detecting misalignment in large language models is challenging because models may learn to conceal misbehavior during training. Standard auditing techniques fall short: black-box methods often cannot distinguish misaligned outputs from benign ones, and mechanistic interpretability does not scale with model capabilities. We introduce Split Personality Training (SPT), which fine-tunes a second ``honest persona'' into LoRA parameters that remain inactive during normal operation. After the main model responds, we activate the LoRA adapter and insert a trigger string, enabling the honest persona to review the response while accessing the main model's latent states. We test our method on the Anthropic Auditing Game Model Organism, a benchmark where Llama-3.3-70B is trained to exploit reward hacks while concealing this behavior. SPT achieves 96% overall accuracy, whereas Anthropic reports near 0% accuracy. The honest persona reveals latent knowledge inaccessible to external observers, such as the fictional biases the compromised model was trained on.

Deep Learning · Theory

Hoyeon Chang, Bálint Mucsányi, Seong Joon Oh

Neural networks adapt through first-order parameter updates, yet it remains unclear whether such updates preserve logical coherence. We investigate the geometric limits of the Linear Propagation Assumption (LPA), the premise that local updates coherently propagate to logical consequences. To formalize this, we adopt relation algebra and study three core operations on relations: negation flips truth values, converse swaps argument order, and composition chains relations. For negation and converse, we prove that guaranteeing direction-agnostic first-order propagation necessitates a tensor factorization separating entity-pair context from relation content. However, for composition, we identify a fundamental obstruction. We show that composition reduces to conjunction, and prove that any conjunction well-defined on linear features must be bilinear. Since bilinearity is incompatible with negation, this forces the feature map to collapse. These results suggest that failures in knowledge editing, the reversal curse, and multi-hop reasoning may stem from common structural limitations inherent to the LPA.

Social Aspects · Everything Else

Yucen Gao, Zhemeng Yu, Zhuoran Li, Jianxiong Guo, Xiaofeng Gao

Social recruitment offers a solution to worker scarcity in Spatial Crowdsourcing (SC) but faces challenges that are often ignored in traditional Influence Maximization. First, task heterogeneity arising from offline execution constraints breaks the ``interest-implies-participation'' assumption, as social influence often fails to translate into physical presence. Second, finite task demand creates a ``saturation trap'', a non-submodular setting in which utility drops sharply to zero once demand is met. To bridge these gaps, we propose GKD-Recruiter, a Task-Aware framework designed to maximize Effective Task Satisfaction (ETS). We explicitly model the complex worker-task affinity via a heterogeneous graph and capture directional social influence using a novel Influential GAT. To robustly fuse these distinct signals, we introduce a Graph Knowledge Distillation mechanism. Furthermore, we employ Rainbow DQN to navigate the non-submodular combinatorial search space, avoiding the local optima that trap greedy heuristics. Extensive experiments on real-world datasets demonstrate that GKD-Recruiter significantly outperforms state-of-the-art baselines in both solution quality and inference efficiency. The code is available at \url{https://anonymous.4open.science/r/GKD-Recruiter-3A4B}.

General Machine Learning · Transfer, Multitask and Meta-learning

Feiyang Deng, Lingfeng Luo, Di Wang, Qinmengge Li, Lingxuan Kong, Kevin He

Accurate prediction in survival analysis with competing risks is challenged by rare event rates and limited effective sample sizes. Knowledge distillation offers a promising way to transfer information from an external teacher to improve a local student, but existing methods are overwhelmingly developed for uncensored outcomes and do not directly extend to survival analysis, where censored observations provide only partial information. Moreover, prior work often assumes that teacher and student share identical outcome definitions, whereas in competing risks settings, they may differ in outcome granularity and event definitions, further complicating knowledge transfer. To address these gaps, we propose DiSKD (Discrete Survival Knowledge Distillation), a deep learning framework for discrete-time competing risks that integrates teacher predictions via a cause-specific, time-dependent Kullback--Leibler divergence. DiSKD enables flexible and privacy-preserving transfer without requiring raw data sharing, remains robust to model misspecification or outcome-definition heterogeneity, and adaptively weights teacher guidance by emphasizing compatible teachers while down-weighting less relevant ones. Simulation studies and real-world applications demonstrate improved discrimination and calibration.

Deep Learning · Large Language Models

Sunghwan Kim, Junhee Cho, Beong-woo Kwak, Taeyoon Kwon, Liang Wang, Nan Yang, Xingxing Zhang, Furu Wei, Jinyoung Yeo

Large language models (LLMs) have shown promise as interactive agents that solve tasks through extended sequences of environment interactions. While prior work has primarily focused on system-level optimizations or algorithmic improvements, the role of task horizon length in shaping training dynamics remains poorly understood. In this work, we present a systematic empirical study that examines horizon length through controlled task constructions. Specifically, we construct controlled tasks in which agents face identical decision rules and reasoning structures, but differ only in the length of action sequences required for successful completion. Our results reveal that increasing horizon length alone constitutes a training bottleneck, inducing severe training instability driven by exploration difficulties and credit assignment challenges. We demonstrate that horizon reduction is a key principle to address this limitation, stabilizing training and achieving better performance in long-horizon tasks. Moreover, we find that horizon reduction is related to stronger generalization across horizon lengths: models trained under reduced horizons generalize more effectively to longer-horizon variants at inference time, a phenomenon we refer to as horizon generalization.

Zongsheng Cao, Anran Liu, Jun Xie, Jing Li, Lang Chen, Feng Chen, zhepeng Wang, Zigan Wang

Long-video temporal reasoning remains challenging for Large Video Language Models (LVLMs). Recent reasoning-enhanced models apply reinforcement learning with outcome supervision to improve temporal understanding. However, outcome-only rewards cannot distinguish whether a model arrived at the correct answer through valid temporal reasoning or fabricated claims, a fundamental limitation that undermines trustworthiness. We observe a key structural correspondence: in videos, events form \emph{temporal traces}, ordered sequences of how entities interact over time; in model reasoning, \emph{reasoning traces} capture step-by-step temporal claims. Correct temporal reasoning requires the latter to mirror the former. This correspondence enables us to \emph{verify} reasoning traces against the video's temporal structure. We introduce \textbf{Temporal Reasoning Traces (TRT)}, a unified representation that indexes ordered event chains from videos and serves as a verification oracle for model reasoning. Building on TRT, we propose \textbf{trace-grounded process supervision}: during reinforcement learning, each temporal claim in the model's reasoning trace is programmatically verified against TRT,rewarding grounded reasoning and penalizing fabrications. Unlike neural reward models that may themselves err, our verification is fully deterministic. Extensive experiments show the effectiveness of our model, achieving state-of-the-art performance.

Deep Learning · Generative Models and Autoencoders

Mikhail Persiianov, Arip Asadulaev, Nikita Andreev, Nikita Starodubcev, Dmitry Baranchuk, Anastasis Kratsios, Evgeny Burnaev, Aleksandr Korotin

Learning conditional distributions $\pi^\star(\cdot|x)$ is a central problem in machine learning, which is typically approached via supervised methods with paired data $(x,y) \sim \pi^\star$. However, acquiring paired data samples is often challenging, especially in problems such as domain translation. This necessitates the development of *semi-supervised* models that utilize both limited paired data and additional unpaired i.i.d. samples $x \sim \pi^\star_x$ and $y \sim \pi^\star_y$ from the marginal distributions. The usage of such combined data is complex and often relies on heuristic approaches. To tackle this issue, we propose a new learning paradigm that integrates both paired and unpaired data seamlessly using data likelihood maximization techniques. We demonstrate that our approach also connects intriguingly with inverse entropic optimal transport (OT). This finding allows us to apply recent advances in computational OT to establish an *end-to-end* learning algorithm to get $\pi^\star(\cdot|x)$. In addition, we derive the universal approximation property, demonstrating that our approach can theoretically recover true conditional distributions with arbitrarily small error. Finally, we demonstrate through empirical tests that our method effectively learns conditional distributions using paired and unpaired data simultaneously.

Reinforcement Learning · Deep RL

Arip Asadulaev, Maksim Bobrin, Salem Lahlou, Dmitry V. Dylov, Fakhri Karray, Martin Takac

Off-policy learning methods seek to derive an optimal policy directly from a fixed dataset of prior interactions. This objective presents significant challenges, primarily due to the inherent distributional shift and value function overestimation bias. These issues become even more noticeable in *zero-shot* reinforcement learning, where an agent trained on reward-free data must adapt to new tasks at test time without additional training. In this work, we address the off-policy problem in a zero-shot setting by discovering a theoretical connection of successor measures to stationary density ratios. Using this insight, our algorithm can infer optimal importance sampling ratios, effectively performing a stationary distribution correction with an optimal policy *for any task on the fly*. We benchmark our method in motion tracking tasks on SMPL Humanoid, continuous control on ExoRL, and for the long-horizon OGBench tasks. Our technique seamlessly integrates into forward-backward representation frameworks and enables *fast-adaptation* to new tasks in a *raining-free* regime. More broadly, this work bridges off-policy learning and zero-shot adaptation, offering benefits to both research areas.

Xu Wang, Bingqing Jiang, Yu Wan, Baosong Yang, Lingpeng Kong, Difan Zou

Sparse autoencoders (SAEs) have become a standard tool for mechanistic interpretability in autoregressive large language models (LLMs), enabling researchers to extract sparse, human-interpretable features and intervene on model behavior. Recently, as diffusion language models (DLMs) have become an increasingly powerful and promising alternative to autoregressive LLMs, it is essential to develop tailored mechanistic interpretability tools for this emerging class of models. In this work, we present **DLM-Scope**, the first SAE-based interpretability framework for DLMs, and demonstrate that trained Top-K SAEs can faithfully extract sparse, interpretable features. Notably, we find that inserting SAEs affects DLMs differently from autoregressive LLMs: while SAE insertion in LLMs typically incurs a loss penalty, in DLMs it can reduce cross-entropy loss when applied to early layers, a phenomenon absent or markedly weaker in LLMs. Additionally, SAE features in DLMs enable more effective diffusion-time interventions, often outperforming LLM steering. Moreover, we pioneer new SAE-based research directions for DLMs: we show that SAEs provide useful signals for DLM decoding order, and that SAE features remain stable during DLM post-training. Overall, our work establishes a foundation for mechanistic interpretability in DLMs and highlights the potential of applying SAEs to DLM-related tasks and algorithms.

Applications · Time Series

Maximilian Mauel, Johannes Hübers, David Berghaus, Patrick Seifner, Ramses J Sanchez

Ordinary differential equations (ODEs) are central to scientific modelling, but inferring their vector fields from noisy trajectories remains challenging. Current approaches such as symbolic regression, Gaussian process (GP) regression, and Neural ODEs often require complex training pipelines and substantial machine learning expertise, or they depend strongly on system-specific prior knowledge. We propose FIM-ODE, a pretrained Foundation Inference Model that *amortises* ODE inference by predicting the vector field directly from noisy trajectory data *in a single forward pass*. We pretrain FIM-ODE on a prior distribution over ODEs with low-degree polynomial vector fields and represent the target field with neural operators. FIM-ODE achieves strong *zero-shot* performance, matching and often improving upon ODEFormer, a recent pretrained symbolic baseline, across a range of regimes despite using a simpler training prior. Pretraining also provides a strong initialisation for finetuning, enabling fast and stable adaptation that outperforms modern neural and GP baselines without requiring machine learning expertise. We release pretrained weights with the supplementary material.