论文检索

Social Aspects · Accountability, Transparency, and Interpretability

David Chanin, Adrià Garriga-Alonso

Sparse Autoencoders (SAEs) extract features from LLM internal activations, meant to correspond to interpretable concepts. A core SAE training hyperparameter is L0: how many SAE features should fire per token on average. Existing work compares SAE algorithms using sparsity-reconstruction tradeoff plots, implying L0 is a free parameter with no single correct value aside from its effect on reconstruction. In this work we study the effect of L0 on SAEs, and show that if L0 is not set correctly, the SAE fails to disentangle the underlying features of the LLM. If L0 is too low, the SAE will mix correlated features to improve reconstruction. If L0 is too high, the SAE finds degenerate solutions that also mix features. Further, we present a proxy metric that can help guide the search for the correct L0 for an SAE on a given training distribution. We show that our method finds the correct L0 in toy models and coincides with peak sparse probing performance in LLM SAEs. We find that most commonly used SAEs have an L0 that is too low. Our work shows that L0 must be set correctly to train SAEs with correct features.

Social Aspects · Robustness

Lucas D. Konrad, Nikolas Kuschnig

Identifying *most influential sets* (MIS) – size-$k$ subsets whose removal maximally changes a target estimand – is typically infeasible because it requires searching over $\binom{n}{k}$ subsets. We show that, for a broad class of estimands whose leave-set-out effect admits a linear-fractional form, the MIS problem reduces to a one-parameter sequence of top-$k$ selections. Using Dinkelbach's method, we obtain an efficient algorithm that runs in $O(n)$ per iteration and terminates in finitely many steps. We show that our approach returns globally optimal sets for univariate settings, such as average treatment effect estimation in randomized experiments. For partial linear models, we establish selection consistency under Neyman orthogonality and mild first-stage stability. We validate our method through simulations and real-world applications – recovering MIS that were previously computationally inaccessible.

Deep Learning · Generative Models and Autoencoders

Michael Hersche, Nicolas Menet, Ronan Tanios, Abbas Rahimi

Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) models, offering sub-linear generation latency and bidirectional capabilities that are particularly appealing for code generation and editing. Achieving sub-linear latency in discrete DLMs requires predicting multiple tokens in parallel. However, standard DLMs sample tokens independently from conditional marginal distributions, failing to capture the joint dependencies among concurrently generated tokens. As a result, they often lead to syntactic inconsistencies and break multi-token structures. In this work, we introduce CoDiLA (Coherent Diffusion with Local Autoregression), a method that reconciles parallel sampling with local dependency modeling. Rather than forcing the DLM to resolve fine-grained syntax, CoDiLA delegates local decoding to a small, auxiliary AR model operating on the diffusion latents. This design allows for parallel block generation while ensuring sequential validity within each block and maintaining core DLM capabilities, including bidirectional modeling across blocks. We demonstrate that using a highly compact auxiliary AR model (e.g., 0.6B parameters) effectively eliminates coherence artifacts, establishing a new Pareto frontier for accuracy and speed in code generation benchmarks.

Applications · Health / Medicine

Zhengqiu Yu, Yueping Ding, Xiangrong Liu

Patient-level sepsis prediction in the ICU requires models that track how a patient’s condition evolves over time and integrate heterogeneous structured evidence from electronic health records. We present PathwayLLM, a trajectory-based framework that grounds prediction on temporal signals together with graph-structured and pathway-level clinical information derived from statistical dependency discovery. PathwayLLM follows a three-stage design. First, each observation window is encoded from multiple structured views, including physiological measurements, temporal dynamics, a heterogeneous patient–diagnosis–medication graph, and pathway signals constructed from discovered conditional independence structures among clinical variables. Second, these representations are provided to a pre-trained language model as auxiliary contextual embeddings so that risk prediction and evidence-conditioned text explanations can be learned jointly. Third, a Clinical Trajectory LSTM with Deterioration Attention aggregates window-level representations to highlight critical deterioration points and produce a patient-level risk score. On MIMIC-IV (15,410 ICU patients; 8.45% sepsis prevalence), PathwayLLM achieves AUROC 0.891 and AUPRC 0.724, outperforming strong time-series and pre-trained baselines. Ablation studies indicate that trajectory aggregation and structured clinical signals are key contributors, and clinician review suggests that the generated explanations are coherent, interpretable, and clinically relevant.

Theory · Game Theory

Shuang Cui, He Huang, Yu-e Sun, Chen Xue

Budget-feasible procurement auctions play a pivotal role in various AI-driven marketplaces, such as data acquisition and crowdsourcing, where a buyer with a limited budget seeks to procure services from strategic sellers with private costs. While numerous budget-feasible mechanisms have been proposed for the classic objective of maximizing the buyer's valuation, the more challenging and economically significant objective of social welfare maximization has only recently been studied, and existing approaches still sacrifice budget feasibility, thereby limiting their practical applicability. In this paper, we bridge this gap by proposing BFM-SWM, the first budget-feasible mechanism with provable approximation guarantees for submodular welfare maximization in procurement auctions. Our mechanism satisfies standard economic properties, including truthfulness, individual rationality, and non-negative auctioneer surplus. As a by-product, we develop BFM-VM, a variant tailored for valuation maximization, which achieves a deterministic approximation ratio of $1/(12+4\sqrt{3})$ for general submodular functions, substantially improving upon the best-known deterministic ratio of $1/64$ established by [Balkanski et al., SODA 2022], while reducing the running time from $\mathcal{O}(n^2\log n)$ to $\mathcal{O}(n\log n)$. Extensive experiments demonstrate the efficiency and effectiveness of our mechanisms.

Applications · Robotics

Xiaoji Zheng, Ziyuan Yang, Yuhang PENG, Yanhao Chen, Yuanrong Tang, Gengyuan Liu, Bokui Chen, Jiangtao Gong

End-to-end autonomous driving models trained with imitation learning (IL) often generalize poorly, particularly in long-tail scenarios where expert demonstrations are sparse. Reinforcement learning (RL) can provide complementary reward signals, but applying RL in real-world autonomous driving is challenging in offline settings without simulators, where datasets consist almost exclusively of expert actions and lack behavioral diversity. We propose CoIRL-AD, a competitive dual-policy framework that integrates IL and RL under a unified offline training regime. CoIRL-AD decouples IL and RL into separate actors to alleviate objective conflicts between imitation and reward maximization, and introduces a competition-based mechanism that stabilizes learning and enables effective exploration while remaining anchored to expert behavior. Experiments on the nuScenes benchmark show a 27\% relative reduction in collision rate weighted by L2 error compared to strong baselines, with substantially larger gains on cross-city generalization (up to 77\%) and long-tail scenarios (up to 85\%), demonstrating that competitive integration of IL and RL significantly improves robustness in offline end-to-end autonomous driving. Code is available at: \url{https://anonymous.4open.science/r/drive-with-two-minds}.

Optimization · Large Scale, Parallel and Distributed

Ling Chen, Houming Wu, Wenjie Yu

Pipeline parallelism is essential for large-scale model training, but existing asynchronous approaches often degrade convergence due to parameter mismatch between forward and backward passes. We propose Asynchronous Multi-Directional Pipeline parallelism (AMDP) to mitigate this issue while sustaining high utilization. AMDP limits the first stage of each pipeline to process at most two minibatches before backpropagation, bounding the number of parameter updates between forward and backward passes. To alleviate the resulting pipeline bubbles, AMDP launches multiple concurrent pipelines and adapts their number according to pipeline depth. In addition, AMDP accumulates gradients across minibatches and applies them in a single update, ensuring that only a bounded number of minibatches experience parameter mismatch, limited to within one optimization step. Experiments on GPT- and BERT-style models demonstrate that AMDP significantly accelerates training while preserving convergence. The source code based on Megatron-LM is available at https://anonymous.4open.science/r/Megatron-AMDP-59A7.

Reinforcement Learning · Batch/Offline

Mahmoud Selim, Cristina Cipriani, Karl Johansson

Diffusion policies offer a powerful and expressive parameterization for continuous control. Yet, their integration with reinforcement learning remains conceptually and algorithmically challenging. In this work, we address this gap by introducing a noisy-space action-value (Q-)function that assigns values to diffusion latents through the distribution of executed actions induced by the denoising process. We show that this construction admits a precise semantic interpretation and derive a noisy-space policy gradient (NSPG) in which value estimates for noisy latents are computed exclusively using clean action-space values. Building on this result, we formulate a KL-regularized policy improvement over noisy latents and show that the resulting objective admits a diffusion-compatible regression form, avoiding backpropagation through the denoising process. Experiments on the D4RL benchmark demonstrate that semantically grounded value gradients provide a principled, effective foundation for training diffusion policies in offline reinforcement learning.

Reinforcement Learning · Everything Else

Yunhai Hu, Zining Liu, Xiangyang Yin, Tianhua Xia, BO BAO, Eric Sather, Vithursan Thangarasa, Sai Qian Zhang

Speculative reasoning has recently been proposed as a means to accelerate reasoning-intensive generation in large multimodal models, but its effectiveness is often constrained by misalignment between speculative drafts and target-verified reasoning. In this work, we introduce \textit{DREAM-R}, a framework that substantially improves the performance of speculative reasoning. At its core, DREAM-R employs \textit{Speculative Alignment Policy Optimization} (SAPO), a reinforcement-learning objective that trains draft models to generate reasoning steps that are both faithful to target trajectories and concise. We further propose a \textit{Threshold-based Verification Mechanism} (TBVM) that uses a ratio-based criterion to provide stable and interpretable acceptance of speculative steps only when positive evidence clearly dominates, thereby preventing error propagation. Building on these components, we develop a \textit{Fully Parallel Speculative Reasoning} (FPSR) framework that parallelizes draft generation, target-side reasoning, and verification across multi-step reasoning, enabling early stopping and clean fallback. Experiments on reasoning-heavy benchmarks demonstrate up to $2.49\times$ speedup while preserving target-model accuracy, yielding substantial efficiency gains without compromising reasoning quality.

Applications · Time Series

Manrui Jiang, Jingru Huang, Yong Chen, Chen Zhang

Forecasting multivariate hidden Markov processes is challenging due to nonlinear and nonstationary observations, latent state transitions, and cross-sequence dependencies. While deep learning methods achieve strong predictive accuracy, they typically lack explicit state modeling, whereas Hidden Markov Models (HMMs) provide interpretable latent states but struggle with complex nonlinear emissions and scalability. To address these limitations, we propose DRL-STAF, a Deep Reinforcement Learning based STate-Aware Forecasting framework that jointly predicts next-step observations and estimates the corresponding hidden states for complex multivariate hidden Markov processes. Specifically, DRL-STAF models complex nonlinear emissions using deep neural networks and estimates hidden state transitions via reinforcement learning, avoiding predefined transition structures and enabling flexible adaptation to diverse and high-order dynamics. In particular, DRL-STAF remains effective when typical HMM-based methods suffer from state-space explosion. Extensive experiments demonstrate that DRL-STAF consistently outperforms HMM variants, standalone deep learning models, and existing DL–HMM hybrids in both forecasting accuracy and hidden state estimation.

Social Aspects · Accountability, Transparency, and Interpretability

Guancheng Zhou, Yisi Luo, Zhengfu He, Zhenyu Jin, Xuyang Ge, Wentao Shu, Deyu Meng, Xipeng Qiu

Most current paradigms in visual mechanistic interpretability (MI) remain confined to interpreting internal units of the vision model via heuristic methods (e.g., top-$K$ activation retrieval or optimization with regularization). In this work, we establish a theoretical distributional view for visual MI, which models the influence of a feature activation on the natural image distribution, thereby formulating a Kullback-Leibler (KL)-minimal optimization problem to model the MI task. Under this framework, statistical biases are identified within previous MI paradigms, which reveal that they may either be perceptually uninterpretable to humans (i.e., deviate from the natural image distribution), or mechanistically unfaithful to the vision models (i.e., unable to activate model features). To resolve the biases under the distributional view, we propose a model with a KL-minimal soft-constraint principle for visual MI that theoretically balances interpretability and faithfulness. We realize this principle via energy-guided diffusion posterior sampling. Extensive experiments validate the theoretical soundness of the proposed distributional view and demonstrate the practical effectiveness of our paradigm on the DINOv3 vision model.

Social Aspects · Privacy

Zhenlong Liu, Wenyu Jiang, Feng Zhou, Hongxin Wei

Membership Inference Attacks (MIAs) aim to estimate whether a specific data point was used in the training of a given model. Existing state-of-the-art attacks typically rely on training multiple reference models to approximate the conditional score distribution for individual data points, which leads to significant computational overhead and limits their practical applicability. In this work, we propose a novel approach -- Bayesian Membership Inference Attack (BMIA), which performs conditional attack through Bayesian sampling. Specifically, we apply Laplace approximation to a single reference model to obtain a posterior over model parameters, enabling direct estimation of the conditional score distribution. Theoretically, we demonstrate that Bayesian sampling reduces intra-model variance, thereby improving attack power. This insight naturally motivates the multi-reference variant that further enhances performance when additional reference models are available. Extensive experiments across image, text, and tabular datasets indicate that our method achieves state-of-the-art performance in both effectiveness and efficiency.

Deep Learning · Large Language Models

Xin Qiu, Yulu Gan, Conor Hayes, Qiyao Liang, Yinggan XU, Roberto Dailey, Elliot Meyerson, Babak Hodjat, Risto Miikkulainen

Fine-tuning large language models (LLMs) for downstream tasks is an essential stage of modern AI deployment. Reinforcement learning (RL) has emerged as the dominant fine-tuning paradigm, underpinning many state-of-the-art LLMs. In contrast, evolution strategies (ES) has largely been overlooked due to the widespread belief that it does not scale to modern model sizes. This paper overturns this assumption by demonstrating the first successful application of ES to full-parameter fine-tuning of LLMs at the billion-parameter scale, without dimensionality reduction. ES can indeed search over extremely high-dimensional parameter spaces and outperform established RL implementations across multiple axes, including improved tolerance to long-horizon and delayed rewards, robustness across diverse base LLMs, reduced susceptibility to reward hacking, and improved training stability. These findings suggest that ES is not merely a viable alternative to RL, but a fundamentally different and powerful backpropagation-free post-training paradigm that opens a new direction for LLM fine-tuning beyond current RL-based approaches.

Theory · Active Learning and Interactive Learning

Vincent Cohen-Addad, Sasidhar Kunapuli, Vahab Mirrokni, Mahdi Nikdan, David Woodruff, Samson Zhou

In the data selection problem, the objective is to choose a small, representative subset of data that can be used to efficiently train a machine learning model. Sener and Savarese [ICLR 2018] showed that, given an embedding representation of the data and suitable geometric assumptions, heuristics based on $k$-center clustering can be used to perform data selection. This perspective was further explored by Axiotis et. al. [ICML 2024], who proposed a data selection approach based on $k$-means clustering and sensitivity sampling. However, these methods rely on the assumption that the dataset exhibits intrinsic geometric structure that can be effectively captured by clustering, whereas many modern datasets instead possess global algebraic structure that is better exploited by low-rank approximation or principal component analysis. In this paper, we introduce a new data selection framework based on low-rank approximation and residual-based sampling, formulated through the lens of row subset selection and loss-preserving coreset construction. Given an embedding representation of the data satisfying mild regularity conditions, which can be interpreted as algebraic or angular notions of Lipschitz continuity, we show that it is possible to select a weighted subset of $\tilde{O}\left(k + \frac{1}{\varepsilon^2}\right)$ data points whose average loss approximates the average loss over the full dataset within a $(1+\varepsilon)$ relative error, up to an additive $\varepsilon \Phi_k$ term, where $\Phi_k$ denotes the optimal rank-$k$ approximation cost of the embedding matrix. We complement these theoretical guarantees with empirical evaluations, demonstrating that on a range of real-world datasets, our data selection approach achieves improved performance over prior strategies based on uniform sampling or clustering-based sensitivity sampling.

Social Aspects · Security

ZHIXIANG ZHANG, Zesen Liu, Yuchong Xie, Quanfeng Huang, Dongdong She

Semantic caching has emerged as a pivotal technique for scaling LLM applications, widely adopted by major providers including AWS and Microsoft. By utilizing semantic embedding vectors as cache keys, this mechanism effectively minimizes latency and redundant computation for semantically similar queries. In this work, we conceptualize semantic cache keys as a form of fuzzy hashes. We demonstrate that the locality required to maximize cache hit rates fundamentally conflicts with the cryptographic avalanche effect necessary for collision resistance. Our conceptual analysis formalizes this inherent trade-off between performance (locality) and security (collision resilience), revealing that semantic caching is naturally vulnerable to key collision attacks. While prior research has focused on side-channel and privacy risks, we present the first systematic study of integrity risks arising from cache collisions. We introduce CacheAttack, an automated framework for launching black-box collision attacks. We evaluate CacheAttack in security-critical tasks and agentic workflows. It achieves a hit rate of 86% in LLM response hijacking and can induce malicious behaviors in LLM agent, while preserving strong transferability across different embedding models. A case study on a financial agent further illustrates the real-world impact of these vulnerabilities. Finally, we discuss mitigation strategies, highlighting a persistent trade-off between cache efficiency and robustness.

Deep Learning · Algorithms

Yuanxu Sun, Yuezhou Ma, Haixu Wu, Guanyang Zeng, Muye Chen, Jianmin Wang, Mingsheng Long

Boundary representation (B-rep) is the industry standard for computer-aided design (CAD). While deep learning shows promise in processing B-rep models, existing methods suffer from a representation gap: continuous approaches offer analytical precision but are visually abstract, whereas discrete methods provide intuitive clarity at the expense of geometric precision. To bridge this gap, we introduce Brep2Shape, a novel self-supervised pre-training framework designed to align abstract boundary representations with intuitive shape representations. Our framework employs a geometry-aware task where the model learns to predict dense spatial points from parametric Bézier control points, enabling the network to better understand physical manifolds derived from abstract coefficients. To enhance this alignment, we propose a dual transformer backbone with parallel streams that independently encode surface and curve tokens to capture their distinct geometric properties. Additionally, the topology attention is integrated to model the inherent interdependencies between surfaces and curves, thereby maintaining topological consistency. Experimental results demonstrate that Brep2Shape offers significant scalability, achieving state-of-the-art accuracy and faster convergence across various downstream tasks.

Deep Learning · Large Language Models

Juntong Shi, Brian Trippe, Jure Leskovec, Stefano Ermon, Minkai Xu

Diffusion language models (DLMs) offer substantial speed advantages through parallel decoding, but the lack of token dependencies limits generation quality compared to autoregressive (AR) models. Recent progress attempts to bridge the gap via importance sampling, with DLM being the proposal and AR being the target. However, due to the huge gap between their probability space, the sampling requires a large number of particles and thus expensive computation. In this paper, we introduce PoE-Bridge, a novel decoding framework that drastically improves generation speed and accuracy by introducing an intermediate distribution to bridge the gap. The distribution is constructed as a Product-of-Experts (PoE) of the DLM proposal and the AR target. With the intermediate distribution, we first conduct multi-token sampling with the DLM and then apply rejection sampling using the PoE to retain only the verified tokens. The generated chunks are then evaluated by the AR target via importance sampling to produce the final faithful generation. We further propose several improved techniques, including mixed-temperature sampling for enhanced diversity and elastic rejection windows for reducing wasted verification. Empirically, PoE-Bridge achieves significantly improved accuracy with $5\times$ speedup over the standard DLM decoding approach, and recovering at least 95% of the target AR model's performance, efficiently advancing most of the quality gap on challenging mathematical reasoning and coding tasks.

Changshuo Wang, Shuting He, Xiang Fang, Weijun Li, Yixian Shen, Mingkun Xu, Zhongtian Sun, Prayag Tiwari

Point cloud data, with its inherent geometric and topological structures, plays a critical role in 3D vision tasks. However, existing parameter-efficient fine-tuning (PEFT) methods predominantly focus on input token prompting, overlooking the intrinsic geometric information. To address this limitation, we propose TopAdapter, a novel PEFT framework that enhances geometric perception by injecting local topological information into pre-trained 3D vision models. TopAdapter leverages 0D, 1D, and 2D simplices from algebraic topology as fundamental building blocks, introducing two core modules: the Topology Injection module (ToInjection) and the Topology Transfer module (ToTransfer). ToInjection constructs multi-scale topological features using a simplex generator and dynamically fuses them with semantic features via a geometric controller, thereby enhancing geometric adaptability. ToTransfer propagates these topological primitives across Transformer layers, ensuring efficient transmission of geometric information. Extensive experiments demonstrate that TopAdapter outperforms existing PEFT methods, achieving performance comparable to full fine-tuning across various benchmarks.

Social Aspects · Accountability, Transparency, and Interpretability

Min Xue, Artur Andrzejak

Mechanistic interpretability of Transformer models commonly relies on training auxiliary proxy models, such as Sparse Autoencoders or Cross-Layer Transcoders. While effective, these post-hoc approaches introduce approximation bias and incur substantial computational overhead. We propose an alternative, training-free interpretability framework that directly exploits the Singular Value Decomposition (SVD) of weight matrices in Transformer MLP sublayers. By operating natively on model parameters, our method improves scalability while preserving fidelity to the original weights. We show that the projection matrices of MLP sublayers admit a natural decomposition into orthogonal, interpretable rank-1 subspaces, which we term **Detector-Effector Units** (DEUs). Within each unit, a singular vector functions as a detector of input patterns and modulates a coupled effector vector that encodes output semantics. Building on this structure, we introduce **Subspace Contribution Analysis** (SCA), a diagnostic method that quantifies the direct causal contribution of individual native subspaces to model predictions. Experiments across the GPT-2 family demonstrate that our framework, **Native Network Anatomy** (NaNA), identifies dominant functional pathways with orders-of-magnitude efficiency gains over training-based interpretability baselines, while maintaining weight fidelity. Our results suggest that SVD-based analyses provide a scalable and faithful alternative to learned proxy approaches for mechanistic interpretability.

Changshuo Wang, Shuting He, Xiang Fang, Weijun Li, Xingyu Gao, Zhonghang Liu, Prayag Tiwari, Dimitrios Kanoulas

Few-shot point cloud semantic segmentation (FS-PCSS) aims to achieve precise segmentation of novel categories using only limited labeled samples. Existing prototype-based methods typically rely on shallow feature fusion strategies, failing to adequately model the feature distribution shift between support and query sets, resulting in insufficient prototype adaptation. To address this, we propose the Deep Prototype Refinement Network (DPR-Net), which systematically achieves progressive adaptation by constructing a coarse-to-fine prototype evolution trajectory. Our core Dynamic Prototype Refinement (DPR) module explicitly decomposes features into common and distinctive subspaces based on channel activation, enabling targeted adjustment of domain-sensitive features while preserving class-shared semantics. By cascading multiple refinement modules, we construct a prototype trajectory transitioning from support-biased to query-adapted representations, mitigating both under- and over-adaptation. Furthermore, our Mixture of Prototype Experts (MoPE) mechanism treats prototypes at different stages as complementary experts and adaptively ensembles their predictions through confidence-driven weighting. Extensive experiments demonstrate that DPR-Net achieves state-of-the-art performance with high efficiency. Notably, with only 0.28M parameters, DPR-Net achieves 80.76% mIoU on S3DIS (2-way 1-shot), representing a 15.92% improvement over the baseline.