论文检索

Social Aspects · Everything Else

Tom Zahavy

How do we fundamentally discover new things? In a letter to Maurice Solovine, Albert Einstein conceptualized discovery as a cyclical process involving an intuitive 'jump' from sensory experience to axioms, followed by logical deduction. While Generative AI has mastered Induction (statistical pattern matching) and is rapidly conquering Deduction (formal proof), we argue it lacks the mechanism for Abduction—the generation of novel explanatory hypotheses. Using Einstein’s formulation of General Relativity as a computational case study, we demonstrate that the prevailing theory of "creativity as data compression" (induction) fails to account for discoveries where observational data is scarce. This position paper argues that while a modern Large Language Model could plausibly execute the deductive phase of proving theorems from established premises, it is structurally incapable of the abductive 'Jump' required to formulate those premises. We identify the translation of simulation into formal axioms as the critical bottleneck in artificial scientific invention, and propose that physically consistent, multimodal world models offer the necessary sensory grounding to bridge this divide.

Optimization · Discrete and Combinatorial Optimization

Tae-Hoon Lee, Min-Soo Kim

Primal heuristics play a crucial role in quickly finding feasible solutions for NP-hard integer linear programming (ILP). Although $\textit{end-to-end learning}$-based primal heuristics (E2EPH) have recently been proposed, they are typically unable to independently generate feasible solutions. To address this challenge, we propose RL-SPH, a novel reinforcement learning-based start primal heuristic capable of independently generating feasible solutions, even for ILP involving non-binary integers. Empirically, RL-SPH rapidly obtains high-quality feasible solutions with a 100% feasibility rate, achieving on average a 39× lower primal gap and a 2.3× lower primal integral compared to existing start primal heuristics.

Javier Rando, Jie Zhang, Nicholas Carlini, Florian Tramer

In the past decade, considerable research effort has been devoted to securing machine learning (ML) models that operate in adversarial settings. Yet, progress has been slow even for simple "toy" problems (e.g., robustness to small adversarial perturbations) and is often hindered by non-rigorous evaluations. Today, adversarial ML research has shifted towards studying larger, general-purpose language models. In this position paper, we argue that the situation is now even worse: in the era of LLMs, the field of adversarial ML studies problems that are (1) less clearly defined, (2) harder to solve, and (3) even more challenging to evaluate. As a result, we caution that yet another decade of work on adversarial ML may be failing to produce meaningful progress.

Applications · Everything Else

Yukun Chen, Jiaming Li, Longze Chen, Ze Gong, Jingpeng Li, Zhen Qin, Hengyu Chang, Lei Zhang, Ancheng Xu, Zhihao Yang 等

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a prevailing paradigm for enhancing reasoning in Multimodal Large Language Models (MLLMs). However, relying solely on outcome supervision risks reward hacking, where models learn spurious reasoning patterns to satisfy final answer checks. While recent rubric-based approaches offer fine-grained supervision signals, they suffer from high computational costs of instance-level generation and inefficient training dynamics caused by treating all rubrics as equally learnable. In this paper, we propose **Stratified Rubric-based Curriculum Learning (RuCL)**, a novel framework that reformulates curriculum learning by shifting the focus from data selection to reward design. RuCL generates generalized rubrics for broad applicability and stratifies them based on the model's competence. By dynamically adjusting rubric weights during training, RuCL guides the model from mastering foundational perception to tackling advanced logical reasoning. Extensive experiments on various visual reasoning benchmarks show that RuCL yields a remarkable **+7.83%** average improvement over the Qwen2.5-VL-7B model, achieving a state-of-the-art accuracy of **60.06%**.

Reinforcement Learning · Everything Else

Yiran Zeng, Da Chen, Hangyu Mao, Yuanxing Zhang, Pengfei Wan, Mengchen Zhao

Recent advances in group-based reinforcement learning (RL) greatly improve LLMs' ability in text reasoning. Yet, these methods lack sufficient modeling of multimodal information, leading to significant reasoning hallucination. In this work, we propose POLIA, a novel group-based RL method with visual-object-level intrinsic advantage for multimodal reasoning. POLIA introduces two advantage computation stages over candidate answers and visual objects, respectively. The answer-level extrinsic advantages are computed based on the extrinsic rewards of a group of candidate answers. Moreover, we compute an intrinsic advantage for each visual object based on its confidence score and reference relations with final answers. Intuitively, the intrinsic advantage of an object reflects its potential contribution to the correct answer. This two-stage advantage computation ensures an accurate credit assignment mechanism over multimodal reasoning sequences with multiple visual objects. Experimental results on diverse multimodal reasoning benchmarks show that POLIA significantly outperforms open MLLMs and strong baselines. Our code is available at https://anonymous.4open.science/r/POLIA-AB1D.

Reinforcement Learning · Planning

Zuyuan Zhang, Zeyu Fang, Tian Lan

Geometric properties can be leveraged to stabilize and speed reinforcement learning. Existing examples include encoding symmetry structure, geometry-aware data augmentation, and enforcing structural restrictions. In this paper, we take a novel view of RL through the lens of order theory and recast value function estimates into learning a desired poset (partially ordered set). We propose \emph{GCR-RL} (Geometric Coherence Regularized Reinforcement Learning) that computes a sequence of super-poset refinements -- by refining posets in previous steps and learning additional order relationships from temporal difference signals -- thus ensuring geometric coherence across the sequence of posets underpinning the learned value functions. Two novel algorithms by Q-learning and by actor--critic are developed to efficiently realize these super-poset refinements. Their theoretical properties and convergence rates are analyzed. We empirically evaluate GCR-RL in a range of tasks and demonstrate significant improvements in sample efficiency and stable performance over strong baselines.

Jing Zhang, Chao Luo

Cross multi-domain log anomaly detection aims to train a unified model applying in multiple heterogeneous systems, alleviating the annotation cost and scalability bottlenecks of traditional cross single-domain approaches. However, existing methods face two fundamental challenges: (i) geometric proximity alone is insufficient to certify normality, and (ii) forcibly aligning distributions across domains can induce negative transfer. To address these issues, we propose MPFM (Cross Multi-Domain Prototype Flow Matching for Log Anomaly Detection), grounded in the principle that anomalies are samples that cannot be stably generated by the normal data-generating mechanism. Specifically, MPFM employs a shared–private prototype system to disentangle cross-domain commonalities from domain-specific patterns, introduces domain-conditioned flow matching to perform anomaly detection by integrating structural and dynamical evidence, and further leverages prototype-drift-driven hard example mining to improve robustness near decision boundaries. Experiments on HDFS, BGL, Thunderbird, and Spirit demonstrate that MPFM delivers substantial gains under multi-domain joint training.

Reinforcement Learning · Online

Yao Luan, Ni Mu, Hanfei Ge, Yiqin Yang, Bo XU, Qing-Shan Jia

Unsupervised skill discovery (USD) aims to learn diverse behaviors without reward functions, but often results in task-irrelevant or hazardous behaviors due to uniform exploration. Guided skill discovery (GSD) addresses this issue by incorporating human intent to focus exploration on meaningful regions. However, existing GSD methods typically require training additional guidance models from scratch, which can be ineffective with sparse human feedback. To tackle this, we propose COLLIE, a GSD framework that utilizes sparse human feedback effectively by constructing a semantically coherent skill latent space. The semantical coherence property enables a training-free guidance signal construction, eliminating the need for additional model training beyond skill learning. Furthermore, as this property is derived from dense unsupervised data, the latent space is well-structured, ensuring reliability even with sparse human feedback. Theoretical analysis justifies the effectiveness of our training-free guidance signal, while experiments across diverse state-based and pixel-based tasks show that COLLIE learns diverse, human-aligned skills, avoids hazardous behaviors, and achieves superior downstream performance with minimal human feedback.

Reinforcement Learning · Multi-agent

Grégoire Lambrecht, Mathieu Lauriere

Mean Field Games (MFGs) provide a powerful framework for modeling the collective behavior of large populations of interacting agents. In this paper, we address the problem of Imitation Learning (IL) in MFGs subject to common noise, where the population distribution evolves stochastically. This stochasticity compels agents to adopt population-aware policies to respond to aggregate shocks. We formulate two distinct learning objectives: recovering a Nash equilibrium and maximizing performance against an expert population. We investigate two imitation proxies: Behavioral Cloning (BC) and Adversarial (ADV) divergence. We then establish finite-sample error bounds showing that minimizing these proxies effectively controls both the policy’s exploitability and its performance gap relative to the expert. Furthermore, we propose a numerical framework using generalized Fictitious Play and Deep Learning to compute expert population-aware policies. Through experiments on three environments we demonstrate that standard population-unaware policies fail to capture the equilibrium dynamics. Our results highlight that learning population-aware policies is crucial to avoid being misled by the randomness inherent in common noise.

Yaorui Shi, Shugui Liu, Yu Yang, Wenyu Mao, Yuxin Chen, Qi GU, Hui Su, Xunliang Cai, Xiang Wang, An Zhang

Long-horizon agentic reasoning necessitates effectively compressing growing interaction histories into a limited context window. Most existing memory systems serialize history as text, where token-level cost is uniform and scales linearly with length, often spending scarce budget on low-value details. To this end, we introduce **MemOCR**, a multimodal memory agent that improves long-horizon reasoning under tight context budgets by allocating memory space with adaptive information density through visual layout. Concretely, MemOCR maintains a structured rich-text memory (e.g., headings, highlights) and renders it into an image that the agent consults for memory access, visually prioritizing crucial evidence while aggressively compressing auxiliary details. To ensure robustness across varying memory budgets, we train MemOCR with reinforcement learning under budget-aware objectives that expose the agent to diverse compression levels. Across long-context multi-hop and single-hop question-answering benchmarks, MemOCR outperforms strong text-based baselines and achieves more effective context utilization under extreme budgets.

Reinforcement Learning · Inverse

Lionel Blondé, Joao A. Candido Ramos, Alexandros Kalousis

We consider imitation learning in the low-data regime, where only a limited number of expert demonstrations are available. In this setting, methods that rely on large-scale pretraining or high-capacity architectures can be difficult to apply, and efficiency with respect to demonstration data becomes critical. We introduce Noise-Guided Transport (NGT), a lightweight off-policy method that casts imitation as an optimal transport problem solved via adversarial training. NGT requires no pretraining or specialized architectures, incorporates uncertainty estimation by design, and is easy to implement and tune. Despite its simplicity, NGT achieves strong performance on challenging continuous control tasks, including high-dimensional Humanoid tasks, under ultra-low data regimes with as few as 20 transitions.

General Machine Learning · Evaluation

Mina Remeli, Moritz Hardt

Pairwise comparisons by humans or judge models, combined with aggregation methods such as Elo or Bradley-Terry, have become a central part of evaluating generative models. However, there has been significant debate whether they measure what they intend to measure. Some argue, pairwise comparisons from judges may reward superficial stylistic cues or display judge biases. In a more positive turn, we show that model rankings from pairwise comparisons strongly agree with ground-truth-based accuracy rankings when such ground truth is available for comparison. To make this observation, we convert well-known benchmarks into free-form generative evaluations scored with Elo rankings from pairwise comparisons. We find that Elo rankings show Spearman correlation above 0.9 with accuracy rankings across five established benchmarks. In addition, Elo rankings have significantly more agreement with accuracy than direct evaluation when the judge is weak. Finally, we show that style and judge bias have only minor effects on model rankings. Although style and bias impede absolute measurement, our work demonstrates that model rankings from pairwise comparisons nevertheless reflect accuracy.

Reinforcement Learning · Inverse

Tian Xu, Zexuan Chen, Zhilong Zhang, Yi-Chen Li, Chenyang Wang, lei yuan, Yang Yu

Adversarial imitation learning (AIL) demonstrates superior expert sample efficiency compared to behavioral cloning (BC), yet requires substantial online environment interaction. While recent empirical work has explored initializing AIL algorithms with BC-pretrained policies to address this limitation, a rigorous theoretical understanding of pretraining's role in AIL remains lacking. This paper provides a systematic theoretical analysis and develops principled pretraining algorithms for accelerating AIL. We first analyze AIL with policy pretraining alone, identifying reward error as the dominant error source and thereby uncovering a critical yet previously unexplored gap: the omission of reward pretraining. Leveraging this insight, we introduce a principled policy-reward co-pretraining mechanism through reward-shaping analysis. Our analysis reveals a fundamental connection between expert policies and shaping rewards, naturally motivating CoPT-AIL, an approach that jointly pretrains both policies and rewards through a single BC procedure. We prove that CoPT-AIL achieves an improved imitation gap bound compared to standard AIL, establishing the first theoretical guarantee for pretraining benefits in AIL. Experimental results validate CoPT-AIL's superior performance over existing AIL methods.

Applications · Neuroscience, Cognitive Science

Shujun Zhou, Junrong Qi, Guozhang Chen

The hippocampus encodes space through a striking gradient of place field sizes along its dorsal-ventral axis, yet the principles generating this continuous gradient from discrete grid cell inputs remain unclear. We propose a unified theoretical framework establishing how multiscale hippocampal place fields arise from the frequency-dependent decay of grid cell projections. Functionally, this organization establishes an inductive bias in the population code, managing a fundamental trade-off between spatial precision and sample efficiency. Translating this insight to artificial neural networks, we incorporate a hippocampus-inspired positional embedding (HIPE) into the Transformer architecture to induce multi-scale representation. Experimental results confirm that this mechanism effectively improves data efficiency. Our work establishes a link between neural connectivity, activity patterns, and learning, suggesting a principled approach to utilizing multi-scale representations for sample-efficiency learning.

Reinforcement Learning · Inverse

Chaitanya Kharyal, Calarina Muslimani, Matthew Taylor

Reward design remains a significant bottleneck in applying reinforcement learning (RL) to real-world problems. A popular alternative is reward learning, where reward functions are inferred from human feedback rather than manually specified. Recent work has proposed learning reward functions from human ratings rather than traditional binary preferences, enabling richer and potentially less cognitively demanding supervision. Building on this paradigm, we introduce a new rating-based RL method, Ranked Return Regression for RL (R4). At its core, R4 uses a novel ranking mean squared error loss that learns from a dataset of trajectory–rating pairs, treating the human-provided discrete ratings (e.g., "bad," "neutral," "good") as ordinal targets. Unlike prior rating-based approaches, R4 offers formal guarantees: its solution set is provably minimal and complete under mild assumptions. Empirically, using both human-provided and simulated ratings, we demonstrate that R4 consistently matches or outperforms existing rating and preference-based RL methods on robotic benchmarks from OpenAI Gym and the DeepMind Control Suite.

Applications · Time Series

Yihang Lu, Xianwei Meng, Enhong Chen

Neural Forecasters (NFs) have become a cornerstone of Long-term Time Series Forecasting (LTSF). However, recent progress has been hampered by an overemphasis on architectural complexity at the expense of fundamental forecasting principles. In this work, we revisit the principles of LTSF. We begin by formulating a Variance Reduction Hypothesis (VRH), positing that generating and combining multiple forecasts is essential to reducing the inherent uncertainty of NFs. Guided by this, we propose Boosted Direct Output (BDO), a streamlined paradigm that synergistically hybridizes the causal structure of Auto-Regressive (AR) with the stability of Direct Output (DO), while implicitly realizing the principle of forecast combination within a single network. Furthermore, we address the critical validation-test generalization gap by employing parameter smoothing to stabilize optimization. Extensive experiments demonstrate that these trivial yet principled improvements enable a direct temporal MLP to outperform recent, complex state-of-the-art models in nearly all benchmarks, without relying on intricate inductive biases. Finally, we empirically verify our hypothesis, establishing a dynamic performance bound that highlights promising directions for future research. The code for review is available at: \url{https://anonymous.4open.science/r/ReNF-A151}.

Reinforcement Learning · Deep RL

Tuan Nguyen, Naseem Khan, Khang Tran, Hai Phan, Issa Khalil

The rapid rise of synthetic media has made deepfake detection a critical challenge for online safety and trust. Progress remains constrained by the scarcity of large, high-quality datasets. Although multimodal large language models (LLMs) exhibit strong reasoning capabilities, their performance on deepfake detection is poor, often producing explanations that are misaligned with visual evidence or hallucinatory. To address this limitation, we introduce a reasoning-annotated dataset for deepfake detection and propose Paragraph-level Relative Policy Optimization (PRPO), a reinforcement learning algorithm that aligns LLM reasoning with image content at the paragraph level. Experiments show that PRPO improves detection accuracy by a wide margin and achieves the highest reasoning score of 4.55/5.0. Ablation studies further demonstrate that PRPO significantly outperforms GRPO under test-time conditions. These results underscore the importance of grounding multimodal reasoning in visual evidence to enable more reliable and interpretable deepfake detection.

Applications · Everything Else

Zhengyang Tang, Zihan Ye, Chenyu Huang, Xuhan Huang, Chengpeng Li, Sihang Li, Guanhua CHEN, Ming Yan, Zizhuo Wang, Hongyuan Zha 等

Large Reasoning Models (LRMs) have demonstrated strong capabilities in complex multi-step reasoning, opening new opportunities for automating optimization modeling. However, existing domain adaptation methods, originally designed for earlier instruction-tuned models, often fail to exploit the advanced reasoning patterns of modern LRMs --- In particular, we show that direct fine-tuning on traditional *non-reflective* datasets leads to limited gains. To fully leverage LRMs' inherent reasoning abilities, we propose **CALM** (*Corrective Adaptation with Lightweight Modification*), a framework that progressively refines LRMs within their native reasoning modes for optimization modeling tasks. In CALM, an expert intervener identifies reasoning flaws and provides concise corrective hints, which the LRM incorporates to produce improved reasoning trajectories. These interventions modify fewer than 2.6\% of generated tokens, but generate high-quality data for soft adaptation through supervised fine-tuning. The adapted model is then further improved through reinforcement learning. Building on CALM, we develop **STORM** (*Smart Thinking Optimization Reasoning Model*), a 4B-parameter LRM that achieves a new state-of-the-art average accuracy of 68.9\% across five popular optimization modeling benchmarks, matching the performance of a 671B LRM. These results demonstrate that dynamic, hint-based data synthesis both preserves and amplifies the native reasoning patterns of modern LRMs, offering a more effective and scalable path towards expert-level performance on challenging optimization modeling tasks.

Deep Learning · Large Language Models

Ruijun Huang, Fang DONG(董方), Xin Zhang, Hengjie Cao, Zhendong Huang, Anrui Chen, Jixian Zhou, Mengyi Chen, Yifeng Yang, Mingzhi Dong 等

Mixture-of-Experts (MoE) architectures scale Large Language Models via expert specialization induced by conditional computation. In practice, however, expert specialization often fails: some experts become functionally similar, while others functioning as de facto shared experts, limiting the effective capacity and model performance. In this work, we analysis from a spectral perspective on parameter and gradient spaces, uncover that (1) experts share highly overlapping dominant spectral components in their parameters, (2) dominant gradient subspaces are strongly aligned across experts, driven by ubiquitous low-rank structure in human corpus, and (3) gating mechanisms preferentially route inputs along these dominant directions, further limiting specialization. To address this, we propose Spectral-Decoupled MoE (SD-MoE), which decomposes both parameter and gradient in the spectral space. SD-MoE improves performance across downstream tasks, enables effective expert specialization, incurring minimal additional computation, and can be seamlessly integrated into a wide range of existing MoE architectures, including Qwen and DeepSeek.

Applications · Computer Vision

Guofeng Zhang, Angtian Wang, Jacob Fang, Liming Jiang, Haotian Yang, Alan Yuille, Chongyang Ma

Real-world videos naturally portray complex interactions among distinct physical objects, effectively forming dynamic compositions of visual elements. However, most current video generation models synthesize scenes holistically and therefore lack mechanisms for explicit compositional manipulation. To address this limitation, we propose HECTOR, a generative pipeline that enables fine-grained compositional control. In contrast to prior methods, HECTOR supports hybrid reference conditioning, allowing generation to be simultaneously guided by static images and/or dynamic videos. Moreover, users can explicitly specify the trajectory of each referenced element, precisely controlling its location, scale, and speed (see Figure1). This design allows the model to synthesize coherent videos that satisfy complex spatiotemporal constraints while preserving high-fidelity adherence to references. Extensive experiments demonstrate that HECTOR achieves superior visual quality, stronger reference preservation, and improved motion controllability compared with existing approaches.