论文检索

Reinforcement Learning · Deep RL

Daniel Ebi, Gaspard Lambrechts, Damien Ernst, Klemens Böhm

Asymmetric actor-critic methods are widely used in partially observable reinforcement learning, but typically assume full state observability to condition the critic during training, which is often unrealistic in practice. We introduce the informed asymmetric actor-critic framework, allowing the critic to be conditioned on arbitrary state-dependent privileged signals without requiring access to the full state. We show that any such privileged signal yields unbiased policy gradient estimates, substantially expanding the set of admissible privileged information. This raises the problem of selecting the most adequate privileged information in order to improve learning. For this purpose, we propose two novel informativeness criteria: a dependence-based test that can be applied prior to training, and a criterion based on improvements in value prediction accuracy that can be applied post-hoc. Empirical results on partially observable benchmark tasks and synthetic environments demonstrate that carefully selected privileged signals can match or outperform full-state asymmetric baselines while relying on strictly less state information.

Reinforcement Learning · Batch/Offline

Xiaocong Chen, Siyu Wang, Lina Yao

Flow-matching policies have recently emerged as a powerful class of generative models for offline reinforcement learning (RL), capable of capturing complex, multi-modal action distributions from static datasets. However, standard training objectives are largely agnostic to the global properties of the generative path, permitting learned vector fields that are irregular and unstable, which can hinder performance. In this work, we introduce PDE-regularized Q-Learning (PQL), a novel algorithm that addresses this limitation by imposing a principled structure on the entire probability flow. PQL makes two synergistic contributions: first, a partial differential equation based regularizer derived from the continuity equation enforces global smoothness and stability on the flow. Second, to solve the complex optimization problem introduced by this regularizer, we propose a Beta-distributed timestep sampling strategy that focuses learning on the critical trajectory segments where the trade-off between imitation and smoothness is most acute. Through extensive experiments, we demonstrate that by structuring the generative journey and not just its destination, PQL achieves state-of-the-art performance on a wide range of challenging offline RL tasks.

Social Aspects · Robustness

Tim Beyer, Sophie Xhonneux, Simon Geisler, Gauthier Gidel, Leo Schwinn, Stephan Günnemann

In this position paper, we argue that current safety alignment research efforts for large language models are hindered by many intertwined sources of noise, such as small datasets, methodological inconsistencies, and unreliable evaluation setups. This can, at times, make it impossible to evaluate and compare attacks and defenses fairly, thereby slowing research progress. We systematically analyze the LLM safety evaluation pipeline, covering dataset curation, optimization strategies for automated red-teaming, response generation, and response evaluation using LLM judges. At each stage, we identify key issues and highlight their practical impact. We also propose a set of guidelines for reducing noise and bias in evaluations of future attack and defense papers. Lastly, we offer an opposing perspective, highlighting practical reasons for existing limitations. We believe that addressing the outlined problems in future research will improve the field’s ability to generate easily comparable results and make measurable progress.

Theory · Everything Else

Chansophea Wathanak In, Yi Li, Wai Ming Tai, Xuan Wu

This paper studies active regression for single-index models under general $\ell_p$-loss with an unknown $1$-Lipschitz link function $f$, formulated as $\min_{f,x} \Vert f(Ax)-b\Vert_p^p$ with full access to $A$ but coordinate-query access to $b$. Prior work established upper bounds for known link functions for all $p\geq 1$ and for unknown link functions only in the $p=2$ case, together with lower bounds for $p\leq 2$. This work addresses the more challenging setting of unknown link functions and general $p \geq 1$. A non-adaptive sampling algorithm is presented that achieves a $(1+\epsilon)$-approximation using $O(d^{p/2\vee 1}/\epsilon^{p\vee 2}\text{poly}\log(n/\epsilon))$ queries. Nearly tight lower bounds are also established for non-adaptive queries when $p>2$. These results close much of the remaining gap in active $\ell_p$ regression for single-index models.

Deep Learning · Algorithms

BINGZHENG GAN, Tianyi Zhang, LI YUSU, Jing Huang, Wei Shi, Yangkai Ding, Tao Yu

The scalability of Large Language Models to long sequences is hindered by the quadratic cost of self-attention and the limitations of positional encodings. To address these, we introduce **Caracal**, a novel architecture that replaces self-attention with a parameter-efficient, $\mathcal{O}(L \log L)$ Multi-Head Fourier (MHF) module. Our contributions are threefold: (1) We leverage the Fast Fourier Transform (FFT) for sequence mixing, inherently addressing both bottlenecks mentioned above. (2) We apply a frequency-domain causal masking technique that enforces autoregressive capabilities via asymmetric padding and truncation, overcoming a critical barrier for Fourier-based generative models. (3) Unlike efficient models relying on hardware-specific implementations (e.g., Mamba), **Caracal** uses standard library operators. This ensures robust portability, eliminating common deployment barriers. Evaluations demonstrate that **Caracal** performs competitively with Transformer and SSM baselines, offering a scalable and simple pathway for efficient long-sequence modeling. Code is available in the supplementary materials.

Deep Learning · Generative Models and Autoencoders

Xiaoyan Xing, Xiao Zhang, Sezer Karaoglu, Theo Gevers, Anand Bhattad

Image-to-image relighting requires representations that disentangle scene properties from illumination. Recent methods rely on latent intrinsic representations but remain under-constrained and often fail on challenging materials such as metal and glass. A natural hypothesis is that stronger pretrained visual priors should resolve these failures. We find the opposite: features from top-performing semantic encoders often degrade relighting quality, revealing a fundamental trade-off between semantic abstraction and photometric fidelity. We study this trade-off and introduce Augmented Latent Intrinsics (ALI), which balances semantic context and dense photometric structure by fusing features from a pixel-aligned visual encoder into a latent-intrinsic framework, together with a self-supervised refinement strategy to mitigate the scarcity of paired real-world data. Trained only on unlabeled real-world image pairs and paired with a dense, pixel-aligned visual prior, ALI achieves strong relighting improvements, with the largest gains on complex, specular materials.

Deep Learning · Attention Mechanisms

Shaoyi Zheng, Wenbo Lu, Yuxuan Xia, Shenji Wan

Designing sparse attention for 2-dimensional image data in diffusion models and vision-language models requires reconciling spatial locality with hardware-efficient execution, a fundamental trade-off that existing methods struggle to resolve. Prior approaches preserve 2D structure through handcrafted sparsity patterns, but often incur uncoalesced memory access, limiting practical speedups on modern GPUs. We present HilbertA, a 2D-aware and GPU-efficient sparse attention mechanism, and show that Hilbert curves provide a hardware-aligned inductive bias for sparse attention over 2D data. By reordering image tokens along Hilbert curves, HilbertA preserves local spatial neighborhoods while inducing a contiguous memory layout aligned with efficient GPU execution. To enable global information flow without uncoalesced access, HilbertA further employs a layer-wise sliding schedule, allowing long-range interactions to emerge progressively across depth. In addition, a small central shared region facilitates cross-tile communication and enhances positional awareness. Implemented in Triton, HilbertA achieves substantial acceleration while maintaining or improving model quality across both diffusion models and VLM. On Flux.1-dev, HilbertA delivers up to $4.17\times$ at $2048\times2048$, with image quality comparable to baselines. On Qwen3-VL-8B, HilbertA achieves over $2.08\times$ attention acceleration with $1.55\times$ improvements in Time-To-First-Token, while maintaining competitive model performance.

General Machine Learning · Supervised Learning

Zikun Jin, Yuhua Qian, Xinyan Liang, Jiaqian Zhang, Haijun Geng

Robust signal enhancement at extremely low SNR is fundamentally challenging because noise becomes strongly entangled with the signal and corrupts local time–frequency (TF) evidence. In this regime, fixed resolution short-time Fourier transform (STFT) enhancement with purely data driven convolutional biases can become overconfident in unreliable TF regions, causing unstable suppression or residual artifacts. We propose FracKGMF, which couples Fractional Distance Decay Convolution (FracConv) with Knowledge Guided Multi-view Fusion (KGMF) for expressive TF modeling and reliability aware decisions under heavy corruption. FracConv introduces a lightweight fractional distance decay family that reshapes local interactions into long tailed receptive patterns, enabling aggregation of weak but globally consistent cues when per-bin observations are ambiguous. KGMF further injects a Wiener inspired reliability view derived from noise statistics to calibrate multi-view fusion, avoiding over-suppression in uncertain regions while exploiting confident structure for effective denoising. Across speech and electromagnetic (EM) benchmarks, FracKGMF improves perceptual quality and intelligibility over state-of-the-art baselines, with especially robust gains at extremely low SNR; on EM dataset at -20 dB, it achieves an average improvement of 33 dB where conventional TF methods performed poorly. The code will be released upon acceptance.

General Machine Learning · Representation Learning

Vinh Le, Nguyen Dang, Tu Vu, Linh Van, Duc Nguyen, Trung Le

Knowledge distillation is crucial for compressing large Vision–Language Models (VLMs) into efficient architectures. While prior VLM research has primarily focused on reasoning tasks like visual question answering, multimodal embedding learning, a key component for large-scale retrieval, has received comparatively less attention. Existing distillation methods typically align static global representations, overlooking hierarchical feature structure and fine-grained cross-modal interactions. This leads to a structural gap where student models fail to inherit object-level semantics and spatial relationships from teachers. To address this limitation, we propose **HieRD**, a Hierarchical Representation Distillation framework that preserves hierarchical structure within and across modalities throughout the distillation process by leveraging clustered visual tokens and multi-granular alignment with phrase-level text. Experimental results on multimodal embedding and downstream tasks show that HieRD consistently outperforms strong baselines, reflecting the effectiveness of its fine-grained semantic and spatial modeling, while enabling compact and efficient embedding models.

Reinforcement Learning · Batch/Offline

Gwanwoo Song, Kwanyoung Park, Youngwoon Lee

In offline reinforcement learning (RL), single-step temporal-difference (TD) learning can suffer from bootstrapping error accumulation over long horizons. Action-chunked TD methods mitigate this by backing up over multiple steps, but can introduce suboptimality by restricting the policy class to open-loop action sequences. To resolve this trade-off, we present Chunk-Guided Q-Learning (CGQ), a single-step TD algorithm that guides a fine-grained single-step critic by regularizing it toward a chunk-based critic trained using temporally extended backups. This reduces compounding error while preserving fine-grained value propagation. We theoretically show that CGQ attains tighter critic optimality bounds than either single-step or action-chunked TD learning alone. Empirically, CGQ achieves strong performance on challenging long-horizon OGBench tasks, often outperforming both single-step and action-chunked methods.

Applications · Health / Medicine

Xuanning Hu, Hao Tuo, Jinglong Ji, Anchen Li, Qianli Xing, Bo Yang

Structure-based drug design (SBDD) can be effectively realized through an iterative refinement via the Design-Make-Test-Analyze (DMTA) cycle, which is a common workflow used by human experts. However, most LLMs function as one-shot generators that lack feedback mechanisms, leaving the DMTA loop disconnected. In this work, we propose K-BTS, a Knowledge-Driven Bi-level Thompson Sampling framework that formalizes iterative SBDD as a Dynamic Hierarchical Multi-Armed Bandit problem. K-BTS closes the DMTA loop by decoupling decisions into two levels: an upper-level policy that prioritizes high-potential molecular lineages and a lower-level mechanism that retrieves explicit chemical rules to guide LLM generation. By integrating a dual-level Bayesian update, the framework transforms sparse docking scores into reusable experience. On the CrossDocked2020 benchmark, K-BTS achieves a state-of-the-art Top-1 average docking score. The results from diverse dimensions show that K-BTS ensures search determinism through a smooth, monotonic convergence that synchronizes structural drift with affinity improvement.

Applications · Computer Vision

Chenhao Qiu, Yechao Zhang, Xin Luo, Shien Song, Xusheng Liu

Long video question answering requires locating sparse, time-scattered visual evidence within highly redundant content. Although current MLLMs perform well on short videos, long videos introduce long-horizon search and verification, which often necessitates multi-turn, agentic interaction. We show that existing LVU agents can exhibit evidence misalignment: they produce correct answers that are not supported by the retrieved or inspected evidence. To characterize this failure, we introduce two diagnostics (temporal groundedness and semantic groundedness) and use them to reveal two pressures that amplify misalignment: prompt pressure from shared-context saturation at inference time and reward pressure from outcome-only optimization during training. These findings point to a structural root cause: the coupled agent paradigm conflates long-horizon planning with answer authority. We therefore propose the decoupled planner--inspector framework, which separates planning from answer authority and gates final answering on pixel-level verification. Across four long-video benchmarks, our framework improves both answer accuracy and evidence alignment, achieving 55.1\% on LVBench and 62.0\% on LongVideoBench while producing interpretable search trajectories. Moreover, the decoupled architecture scales consistently with increased search budgets and supports plug-and-play upgrades of the MLLM backbone without retraining the planner. Code and models are available at \url{https://anonymous.4open.science/r/VideoSeal-8C8D}.

Applications · Computer Vision

Ghadi Nehme, Eamon Whalen, Faez Ahmed

Despite recent progress, recovering parametric CAD construction sequences from geometric input, such as meshes or point clouds, is a key challenge for design and manufacturing, as existing CAD reconstruction and generation methods are largely restricted to difficult-to-edit formats like meshes or Breps or editable simple sketch-and-extrude pipelines and low-complexity datasets. We introduce CADFit, a hybrid optimization-based CAD reconstruction framework that recovers complex, editable CAD construction sequences from meshes by incrementally fitting and validating parametric operations using geometric feedback. Our approach is distinguished by formulating reconstruction as an IoU-driven optimization over structured CAD programs and supporting a rich set of operations, including extrusions, revolutions, fillets, and chamfers. Experiments on multiple CAD benchmarks show that CADFit outperforms state-of-the-art mesh-to-CAD methods in volumetric Intersection-over-Union and Chamfer Distance, while substantially reducing the Invalid Ratio of reconstructed CAD programs, particularly for complex designs. We further present a multimodal pipeline that enables end-to-end reconstruction of CAD construction sequences from images by combining image-based geometry reconstruction with CADFit. By enabling accurate reconstruction of higher-complexity CAD models, CADFit provides a practical foundation for generating richer datasets and advancing future learning-based approaches to CAD reverse engineering.

Social Aspects · Alignment

Aakash Sen Sharma, Debdeep Sanyal, Manodeep Ray, Vivek Srivastava, Shirish Karande, Murari Mandal

Post-training alignment of large language models (LLMs) relies on large-scale human annotations guided by policy specifications that change over time. Cultural shifts, value reinterpretations, and regulatory or industrial updates make static alignment increasingly brittle. As policies evolve, deployed models can diverge from current alignment objectives, creating an Alignment–Reality Gap that is difficult to audit or correct. Existing remediation typically requires re-annotation under revised guidelines, which introduces systematic challenges, including guideline ambiguity, annotator interpretation drift, and reduced consistency at scale. We introduce TRACE (Triage and Re-align by Alignment Conflict Evaluation), a framework that transforms re-alignment into a structured optimization problem over existing data without requiring fresh human annotation. Leveraging a stronger model as a proxy judge, TRACE operates via a three-stage pipeline: (1) triaging preference pairs into inversion, suppression, or retention categories based on alignment conflicts; (2) computing an alignment impact score via bi-level optimization to prioritize high-leverage samples; and (3) executing updates using a hybrid objective that combines relational losses (e.g., IPO) for preference inversion and punitive losses (e.g., NPO) for response suppression. Experiments on Qwen2.5-7B, Gemma-2-9B, and Llama-3.1-8B demonstrate robust re-alignment on synthetic benchmarks and the PKU-SafeRLHF dataset without degrading general utility. This work provides a scalable approach for LLM realignment under evolving data annotation policies and alignment guidelines.

Deep Learning · Large Language Models

Lingjing Kong, Xin Liu, Guangyi Chen, Martin Q. Ma, Xiangchen Song, Yuekai Sun, Mikhail Yurochkin, Taylor W. Killian, Russ Salakhutdinov, Kun Zhang 等

Reinforcement learning (RL) has emerged as a key mechanism for transforming LLMs into robust reasoners. While supervised fine-tuning (SFT) often limits models to the distribution of observed reasoning traces, RL post-training significantly improves performance on out-of-distribution (OOD) tasks that require unfamiliar recombinations of familiar steps. We argue that this improvement is driven by **compositional generalization**, which we formalize through a **Hierarchical Latent Selection Model**. In this framework, reasoning traces are generated by a cascade of discrete latent selection variables corresponding to reusable atomic modules, including both skills (local operations) and routing mechanisms (how intermediate information is selected, reused, and composed). We theoretically show that RL’s exploratory nature provides sufficient coverage to identify latent structure and enable compositional generalization. We design controlled experiments to validate this theory. Our results demonstrate that RL can extract atomic modules from compound traces and recombine them to solve new configurations. Moreover, we find that training on compound traces can yield stronger generalization than training on isolated atomic modules. Finally, we investigate relations between SFT and RL and identify an effective protocol in which SFT ensures coverage of all atomic modules, while RL focuses on novel compositions beyond the SFT support to encourage exploration.

Theory · Reinforcement Learning and Planning

Yike Zhao, Onno Eberhard, Malek khammassi, Ali Sayed, Michael Muehlebach

The family of linear recurrent neural networks has shown strong performance as recurrent memory units in partially observable reinforcement learning. We provide a theoretical justification for their empirical effectiveness by constructing and studying two linear filters: (i) the first exactly reproduces the pre–softmax logits of the belief vector in a hidden Markov model (HMM) under a deterministic transition matrix, thereby serving as a sufficient statistic for optimal policy learning, (ii) the second achieves vanishing state-decoding error under a nearly deterministic transition matrix, thus reducing state ambiguity to near zero. The results extend to action-controlled HMMs, where the corresponding linear filters become time-varying with action-dependent dynamics. We illustrate our main results through numerical experiments and further show that the constructed linear filter serves as a strong feature extractor in a small reinforcement learning game.

Applications · Computer Vision

Yinuo Zhang, Zian Cheng, Ziya Zhao, Zongyu Li, Bingshuo Liu, Qingbin Liu, Junxian Cai, chen, Zhiying Tu, Dianhui Chu 等

Photoshop is a professional image editing software whose complex multi-level menus, fine-grained operations, and layer-based non-destructive editing pose substantial challenges for automated agents. Existing GUI benchmarks and methods primarily target web interfaces and short-horizon, low-complexity tasks, falling short in modeling the multi-step decision-making and semantic understanding required by professional graphic software. We introduce PSBench, the first benchmark specifically designed for image editing in Adobe Photoshop, consisting of 600 human-annotated tasks across three difficulty levels, with tasks drawn from official tutorials and popular real-world workflows. PSBench covers core functionalities such as canvas adjustment, layer manipulation, and filter application, and provides fine-grained evaluation metrics tailored to each task category. Our experiments show that even the state-of-the-art system, Agent S3, achieves a success rate of only 18.09\% on difficult tasks, indicating that GUI agents still face considerable challenges in operating complex professional software. Furthermore, human-in-the-loop evaluations reveal that MLLMs, when serving as interactive assistants, can significantly improve novice users’ task completion rates and reduce operation time.

Applications · Neuroscience, Cognitive Science

Raymond Khazoum, Daniela Fernandes, Aleksandr Krylov, Qin Li, Stephane Deny

Mental rotation—the ability to compare objects seen from different viewpoints—is a fundamental example of mental simulation and spatial world modeling in humans. Here we propose a mechanistic model of human mental rotation, leveraging recent advances in deep, equivariant, and neuro-symbolic learning. Our model consists of three stacked components: (1) an equivariant neural encoder, producing 3D spatial representations of objects from images, (2) a neuro-symbolic object encoder, deriving symbolic objects descriptions from these spatial representations, and (3) a neural decision agent, comparing these symbolic descriptions to prescribe rotation simulations in 3D latent space via a recurrent pathway. Our model design is guided by the abundant experimental literature on mental rotation, which we complemented with experiments in VR where participants could at times manipulate the objects to compare. Our model captures well the performance, response times and behavior of participants in our and others' experiments, and through ablation studies we demonstrate the necessity of each component. Our work adds to a recent collection of deep neural models of human spatial reasoning, further demonstrating the potency of integrating deep, equivariant, and symbolic representations to model the human mind.

Deep Learning · Large Language Models

Shvat Messica, Jiawen Zhang, Kevin Li, Theodoros Tsiligkaridis, Marinka Zitnik

Time series reasoning tasks increasingly start from a natural language question and require targeted analysis of time series. Relevant evidence may be global or confined to a few short segments, so the model must decide what to inspect. Most existing methods compress the full series into a fixed representation before inference, preventing question-adaptive analysis. We introduce ARTIST, an approach that formulates time-series reasoning as a sequential decision problem and trains models to interleave reasoning with adaptive temporal segment selection. ARTIST uses a controller-reasoner architecture and reinforcement learning to optimize segment selection based on answer correctness, allowing the model to actively acquire task-relevant information during inference. We evaluate ARTIST on six time-series reasoning benchmarks against large language models, vision-language models, and prior time series reasoning systems. ARTIST improves average accuracy by 6.46 percentage points over the strongest model, with the largest gains on rare event localization and multi-segment evidence accumulation. Supervised fine-tuning improves performance, and reinforcement learning yields further gains by optimizing question-adaptive segment selection. Across datasets, ARTIST achieves higher accuracy while using a smaller fraction of the input time series, highlighting the importance of learned, selective data utilization for time series reasoning.

Theory · Learning Theory

Hubert Chan, Elaine Shi, Mengshi Zhao, Mingxun Zhou

Modern federated and streaming learning systems often release intermediate models, so privacy must hold for the full trajectory under adaptive interaction. Motivated by participation privacy, we study single-edit neighboring user streams, where one insertion/deletion shifts all subsequent updates and defeats standard Hamming-neighbor continual-release analyses. We give an auditable modular recipe. A randomized buffering wrapper emits bins of size $[U,2U]$, reducing single-edit streams to a Hamming-style per-bin update stream with explicit backlog/delay guarantees, where $U$ is calibrated by the privacy parameters $(\varepsilon,\delta)$. We then prove a certification theorem for independently decomposable (prefix-causal, fresh-noise) continual mechanisms: any non-adaptive Hamming-neighbor DP proof lifts to adaptive inputs. Together, these ingredients yield trajectory-level $(\varepsilon,\delta)$-DP for single-edit streams using standard primitives (e.g., tree prefix sums), with an explicit privacy--latency link via $U$. Streaming DP-SGD experiments validate the privacy-utility-latency tradeoffs and the induced delay distributions.