论文检索

Social Aspects · Safety

Xiao Wang, Yifei Zhang, Yongkang Liu, Xiaocui Yang, Zihan Wang, Shi Feng, Daling Wang

Safety alignment of Large Language Models (LLMs) is extremely fragile, fine-tuning on small number of benign samples can erase safety behaviors learned from millions of preference examples. Existing studies attempt to explain this phenomenon by comparing parameters and hidden states before and after fine-tuning, but overlook their dynamic evolution during fine-tuning. In this work, we analyze parameter dynamics and uncover a critical mechanism underlying safety degradation, where benign fine-tuning causes parameters cumulatively drift toward danger-aligned directions, progressively undermining the model's safety. Inspired by these findings, we propose Sample-Level Quantification of Safety Degradation (SQSD), a method that quantifies each training sample's influence on safety degradation. Specifically, SQSD assigns continuous risk scores to individual samples by measuring their induced parameter updates along safety and danger directions. Extensive experiments across three models and two datasets show that SQSD outperforms baselines in better separating high-risk and low-risk samples, with risk scores that consistently predict the severity of safety degradation. In particular, SQSD exhibits strong transferability across architectures, parameter scales, and parameter-efficient methods.

Deep Learning · Generative Models and Autoencoders

Junyu Zhang, Daochang Liu, Younghyun Kim, Jong Hwan Ko, Shichao Zhang, Chang Xu, Eunbyung Park

Flow map matching (FMM) enables one- and few-step sampling for diffusion-style generation, yet its performance is often hindered by the mismatch between ground-truth training transitions and model-induced flow maps. We propose \textbf{Contrastive Flow Map Matching (CFMM)}, a principled framework that explicitly aligns FMM training with practical sampling. Our approach is grounded in a theoretical upper bound on the reverse KL divergence, which decomposes the distributional gap into a marginal mismatch over intermediate states and a conditional mismatch in endpoint reconstruction. This analysis motivates two complementary objectives: average-velocity regression for marginal alignment and a sampling-aligned InfoNCE contrastive loss for conditional refinement. CFMM is a training-only plug-in for pre-trained FMMs, incurs no inference-time overhead, and supports training FMMs from scratch. Experiments on CIFAR-10, ImageNet, and LSUN across multiple FMM baselines demonstrate consistent improvements in fidelity and perceptual quality with only modest additional training cost.

Social Aspects · Privacy

Fengyu Gao, Jing Yang

Preference alignment is a crucial post-training step for large language models (LLMs) to ensure their outputs align with human values. However, post-training on real human preference data raises privacy concerns, as these datasets often contain sensitive user prompts and human judgments. To address this, we propose **DPPrefSyn**, a novel algorithm for generating differentially private (DP) synthetic preference data to enable privacy-preserving preference alignment. DPPrefSyn is a principled framework grounded in the Bradley–Terry preference model and the intrinsic geometric structure of pairwise human preference data. It first learns an underlying preference model from private data with formal differential privacy guarantees, and then leverages the learned model together with public prompts to synthesize high-quality preference data. It exploits the shared linear structure of per-cluster reward models to effectively capture heterogeneous human preferences in private datasets, and leverages DP Principal Component Analysis (DP-PCA) to improve learning accuracy. Extensive experimental results demonstrate that DPPrefSyn achieves competitive alignment performance under strong DP guarantees. These findings highlight the potential of synthetic preference data as a practical alternative for privacy-preserving preference alignment across a broad range of applications. To the best of our knowledge, this is the first work to generate DP synthetic preference data for LLM alignment.

Deep Learning · Large Language Models

Chenzhi Hu, Qinzhe Hu, Yuhang Xu, Junyi Chen, Ruijie Wang, Shengzhong Liu, Jianxin Li, Fan Wu, Guihai Chen

Large reasoning models (LRMs) like OpenAI o1 and DeepSeek-R1 achieve high accuracy on complex tasks by adopting long chain-of-thought (CoT) reasoning paths. However, the inherent verbosity of these processes frequently results in redundancy and overthinking. To address this issue, existing works leverage Group Relative Policy Optimization (GRPO) to reduce LRM output length, but their static length reward design cannot dynamically adapt according to the relative problem difficulty and response length distribution, causing over-compression and compromised accuracy. In this paper, we propose *SmartThinker*, a novel GRPO-based efficient reasoning method with progressive CoT length calibration. *SmartThinker* makes a two-fold contribution: First, it dynamically estimates the optimal length with peak accuracy during training, and further guides the overlong responses to approach the optimal length, in order to achieve length reduction while sustaining high accuracy. Second, it dynamically modulates the length reward coefficient to avoid the unwarranted penalization of correct reasoning paths. Extensive experiment results show that *SmartThinker* achieves up to 52.5\% average length compression with improved accuracy, and achieves up to 16.6\% accuracy improvement on challenging benchmarks like AIME25.

Deep Learning · Large Language Models

Yuhang Xu, Kaibin Tian, Yang Tian, Zhice Yang, Yifeng Yu, Yan Li, Shengzhong Liu, Fan Wu, Guihai Chen

Reinforcement Learning (RL) has become a cornerstone for improving the performance of Large Language Models (LLMs). However, its rollout phase constitutes a significant efficiency bottleneck, mainly arising from the long-tail bubbles across data parallel ranks, particularly in long-context scenarios where faster GPUs remain idle while waiting for stragglers. Existing solutions, such as partial rollout or asynchronous RL, mitigate these bubbles by compromising the algorithm's strict synchronous nature. Instead, we propose **BubbleSpec**, a novel framework that accelerates RL rollouts while strictly keeping the mathematical exactness. Instead of attempting to eliminate bubbles, BubbleSpec exploits them. We exploit the idle time windows of faster ranks to pre-generate rollout results for subsequent steps, serving as drafts for speculative decoding. Unlike prior speculative methods that rely on historical epoch similarity and warm-ups, BubbleSpec is agnostic to dataset size and provides immediate acceleration from the onset of training. Extensive evaluations demonstrate that BubbleSpec reduces decoding steps by **$\sim$50\%** and increases rollout throughput by up to **1.8$\times$**. Critically, BubbleSpec is seamlessly compatible with various RL frameworks and strategies as it sustains the strict synchronous property of RL algorithms.

Social Aspects · Alignment

Kaizhao Liu, Qi Long, Zhekun Shi, Weijie Su, Jiancong Xiao

Aligning large language models (LLMs) with diverse human preferences is critical for ensuring fairness and informed outcomes when deploying these models for decision-making. In this paper, we seek to uncover fundamental statistical limits concerning aligning LLMs with human preferences, with a focus on the probabilistic representation of human preferences and the preservation of diverse preferences in aligned LLMs. We first show that human preferences can be represented by a reward model if and only if the preference among LLM-generated responses is free of any Condorcet cycle. Moreover, we prove that Condorcet cycles exist with probability converging to one exponentially fast under a general probabilistic preference model called the Luce model, thereby demonstrating the impossibility of fully aligning human preferences using reward-based approaches such as reinforcement learning from human feedback. Next, we explore the conditions under which LLMs would employ mixed strategies -- meaning they do not collapse to a single response -- when aligned in the limit using a non-reward-based approach, such as Nash learning from human feedback. We identify a necessary and sufficient condition for mixed strategies: the absence of a response that is preferred over all others by a majority. As a blessing, we prove that this condition holds with high probability under the Luce model, thereby highlighting the statistical possibility of preserving minority preferences without explicit regularization in aligning LLMs.

Social Aspects · Accountability, Transparency, and Interpretability

Gengrui (Edward) Zhang

GenAI systems, particularly LLMs, rely heavily on vast amounts of publicly available digital content as training data. A significant portion of this content is protected by copyright. While large-scale data scraping may be lawful under certain jurisdictions, the use of copyrighted works to generate outputs that compete with or replicate original creations raises unresolved legal, economic, and ethical concerns. In this position paper, we argue that data providers should be fairly compensated based on their measurable contribution to inference-time outcomes, rather than through coarse, one-time licensing or blanket agreements. We examine alternative perspectives on data ownership, fair use, and model training, and discuss why existing approaches fail to align incentives between GenAI developers and content creators. We then outline concrete roadmaps for developing decentralized systems that enable contribution-aware revenue sharing, including mechanisms for attribution, accounting, and payout at scale. We argue that fair revenue distribution for data providers will not only help resolve ongoing legal disputes surrounding GenAI systems, but also foster a new era of collaboration, rather than competition, between model developers and data creators. By incentivizing the production and sharing of high-quality datasets, such mechanisms can ultimately accelerate the development of more robust, trustworthy, and socially sustainable GenAI systems.

Social Aspects · Accountability, Transparency, and Interpretability

Jared Fernandez, Clara Na, Yonatan Bisk, Constantine Samaras, Emma Strubell

Proper accounting of the energy requirements and environmental impact of artificial intelligence (AI) systems is necessary for researchers, developers, policy makers, and users to assess the barriers to building systems at scale. With the growing complexity of pipelines and underlying infrastructure needed to develop and deploy AI systems, previous approaches for evaluating AI efficiency which focus on the costs of a single training run or an individual inference prediction are no longer sufficient. In this position paper, we enunciate the need for applying life cycle assessment to evaluate the costs of the machine learning model development and deployment pipeline to properly account for the required resources and downstream impact. Life cycle assessments enable the incorporation of costs across the full life cycle of an AI system and its underlying infrastructure, from the embodied costs associated with the physical computing hardware through the operational costs in training and inference.

Deep Learning · Large Language Models

Chenye Ke, Yan Zhuang, Zirui Liu, Qi Liu

The advancements of Large Language Models (LLMs) are primarily attributed to massive pretraining data, which also introduces risks like privacy leakage and data contamination. Therefore, it is crucial to determine whether an LLM has been trained on a given target text. Existing detection methods primarily rely on local statistics of isolated tokens (e.g., those with the lowest probabilities), neglecting the probability dynamics during the token generation process. In this paper, we shift the detection paradigm from a local token to a global sequence perspective, grounded in the core intuition that memorized sequences exhibit volatility patterns distinct from those generated via inference. We propose Adaptive Entropic Convolutional Analysis (AECA), a framework that conceptualizes the probability sequence as a dynamic signal, integrating calibration with convolutional filtering to effectively capture memorization signals. Extensive experiments demonstrate that AECA surpasses state-of-the-art baseline methods, achieving an average AUC improvement of up to 1.5\%, with its advantage being particularly pronounced in long-text scenarios.

Applications · Computer Vision

Masanari Oi, Koki Maeda, Ryuto Koike, Daisuke Oba, Nakamasa Inoue, Naoaki Okazaki

While multimodal large language models (MLLMs) have made substantial progress in single-image spatial reasoning, multi-image spatial reasoning, which requires integration of information from multiple viewpoints, remains challenging. Cognitive studies suggest that humans address such tasks through two mechanisms: *cross-view correspondence*, which identifies regions across different views that correspond to the same physical locations, and *stepwise viewpoint transformation*, which composes relative viewpoint changes sequentially. However, existing studies incorporate these mechanisms only partially and often implicitly, without explicit supervision for both. We propose Human-Aware Training for Cross-view correspondence and viewpoint cHange (HATCH), a training framework with two complementary objectives: (1) Patch-Level Spatial Alignment, which encourages patch representations to align across views for spatially corresponding regions, and (2) Action-then-Answer Reasoning, which requires the model to generate explicit viewpoint transition actions before predicting the final answer. Experiments on three benchmarks demonstrate that \method consistently outperforms baselines of comparable size by a clear margin and achieves competitive results against much larger models, while preserving single-image reasoning capabilities.

Applications · Social Sciences

Yan Zhuang, Junhao Yu, Bohou Zhang, Zachary Pardos, Jinze Wu, Daoqiang Zhang

Adaptive testing is widely adopted in AI-driven educational assessment systems (e.g., GRE), where the goal is to select an optimal subset of questions from a large question pool to accurately estimate an examinee's ability. A fundamental challenge is that: optimal question subsets are inherently personalized, and solving for them is NP-hard. Recently, it has been framed as a gradient matching problem: aligning gradients between selected subsets and the full question set across the entire ability parameter space. However, such global alignment on entire space is computationally expensive and difficult to scale. In this work, we propose GPM (Gradient Path Matching), a novel framework that instead aligns gradients along possible optimization paths toward the final estimate. By leveraging intermediate gradients as supervision, GPM learns an explicit and generalizable selection algorithm from large-scale data. We provide theoretical analysis on its convergence and scalability. Experiments on both real-world and synthetic datasets demonstrate that it achieves the same estimation accuracy using, on average, 20% fewer questions.

General Machine Learning · Data

Hossein Mohebbi, Oliver Schulte, Ke Li, Pascal Poupart

Data-driven modeling in real-world regression tasks often suffers from limited training samples, high collection costs, and noisy observations. Inspired by the impact of data augmentation in vision and language, we propose a novel Counterfactual Residual Data Augmentation (CRDA) technique for tabular regression. Our key insight is that once a regressor has modeled the systematic component of the data, the remaining noise can be viewed as an invariant residual that remains stable under small perturbations of carefully selected features. We exploit this residual invariance to generate new, yet realistic, training samples, effectively expanding the dataset without requiring additional real data. Our method is model-agnostic and readily applicable to various types of regressors. In experiments across datasets from a variety of benchmark repositories, on average, CRDA reduces an MLP Regressor's MSE by 22.9% and an XGBoost Regressor's MSE by 6.4%. When compared to existing state-of-the-art data generators and augmentation techniques, CRDA consistently outperforms in MSE reduction. By adding principled counterfactual variations to the training data, our method offers a simple and efficient remedy for noise-prone, small-sample regression settings.

Deep Learning · Large Language Models

Ye Mo, Chuan Zhou, Fengxiang Cheng, Jialin Yu, Fenrong Liu, Liangming Pan, Sheng Zhou, Haoxuan Li, Zhouchen Lin, Phil Torr

Large Language Models (LLMs) demonstrate remarkable performance across various natural language processing tasks but struggle with complex logical reasoning, particularly in real-world settings. Existing research is largely confined to the closed-world assumption, which posits that all premises required for reasoning are explicitly provided. However, real-world tasks frequently exhibit open-world characteristics, where the provided information is insufficient to infer a conclusion due to missing premises or implicit commonsense knowledge. To address this limitation, we propose OpenIKLR, an Open-world Incomplete-Knowledge-aware Logical Reasoning framework that integrates symbolic logic solvers with LLMs. OpenIKLR first translates natural language into symbolic representations to precisely pinpoint reasoning gaps via a logical solver. It then iteratively generate a minimal set of necessary missing premises using LLMs. To ensure these additional premises are both logically sound and factually accurate, we introduce a dual-verification process: logic verification via the solver and fact verification via the LLMs. Extensive experiments demonstrate that OpenIKLR consistently outperforms existing logical reasoning and RAG baselines across multiple backbones and real-world datasets, highlighting its efficacy in handling incomplete information. The code is available at https://anonymous.4open.science/r/ICML26_22398-B5BF/.

Deep Learning · Large Language Models

Samuele Marro, Jialin Yu, Emanuele La Malfa, Oishi Deb, Jiawei Li, Yibo Yang, Ebey Abraham, Sunando Sengupta, Eric Sommerlade, Michael Wooldridge 等

As frontier Large Language Models (LLMs) increasingly saturate new benchmarks shortly after they are published, benchmarking itself is at a juncture: if frontier models keep improving, it will become increasingly hard for humans to generate discriminative tasks, provide accurate ground-truth answers, or evaluate complex solutions. If benchmarking becomes infeasible, our ability to measure any progress in AI is at stake. We refer to this scenario as the *post-comprehension regime*. In this work, we propose Critique-Resilient Benchmarking, an adversarial framework designed to compare models even when full human understanding is infeasible. Our technique relies on the notion of *critique-resilient correctness*: an answer is deemed correct if no adversary has convincingly proved otherwise. Unlike standard benchmarking, humans serve as bounded verifiers and focus on localized claims, which preserves evaluation integrity beyond full comprehension of the task. Using an itemized bipartite Bradley-Terry model, we jointly rank LLMs by their ability to solve challenging tasks and to generate difficult yet solvable questions. We showcase the effectiveness of our method in the mathematical domain across eight frontier LLMs, showing that the resulting scores are stable and correlate with external capability measures. Our framework reformulates benchmarking as an adversarial generation-evaluation game in which humans serve as final adjudicators.

Samuele Marro, Jialin Yu, Emanuele La Malfa, Oishi Deb, Jiawei Li, Yibo Yang, Ebey Abraham, Sunando Sengupta, Eric Sommerlade, Michael Wooldridge 等

As frontier Large Language Models (LLMs) increasingly saturate new benchmarks shortly after they are published, benchmarking itself is at a juncture: if frontier models keep improving, it will become increasingly hard for humans to generate discriminative tasks, provide accurate ground-truth answers, or evaluate complex solutions. If benchmarking becomes infeasible, our ability to measure any progress in AI is at stake. We refer to this scenario as the *post-comprehension regime*. In this work, we propose Critique-Resilient Benchmarking, an adversarial framework designed to compare models even when full human understanding is infeasible. Our technique relies on the notion of *critique-resilient correctness*: an answer is deemed correct if no adversary has convincingly proved otherwise. Unlike standard benchmarking, humans serve as bounded verifiers and focus on localized claims, which preserves evaluation integrity beyond full comprehension of the task. Using an itemized bipartite Bradley-Terry model, we jointly rank LLMs by their ability to solve challenging tasks and to generate difficult yet solvable questions. We showcase the effectiveness of our method in the mathematical domain across eight frontier LLMs, showing that the resulting scores are stable and correlate with external capability measures. Our framework reformulates benchmarking as an adversarial generation-evaluation game in which humans serve as final adjudicators.

Reinforcement Learning · Deep RL

Masanari Oi, Mahiro Ukai, Masahiro Kaneko, Naoaki Okazaki, Nakamasa Inoue

Direct preference optimization (DPO) has emerged as a promising approach for aligning large language models (LLMs) with human preferences. However, the widespread reliance on the response-level Bradley-Terry (BT) model may limit its full potential, as the reference and learnable models are assumed to be autoregressive only after deriving the objective function. Motivated by this limitation, we revisit the theoretical foundations of DPO and propose a novel formulation that explicitly introduces the autoregressive assumption prior to applying the BT model. By reformulating and extending DPO, we derive a novel variant, termed \textbf{Autoregressive DPO (ADPO)}, that explicitly integrates autoregressive modeling into the preference optimization framework. Without violating the theoretical foundations, the derived loss takes an elegant form: it shifts the summation operation in the DPO objective outside the log-sigmoid function. Furthermore, through theoretical analysis of ADPO, we show that there exist two length measures to be considered when designing DPO-based algorithms: the token length $\mu$ and the feedback length $\mu'$. To the best of our knowledge, we are the first to explicitly distinguish these two measures and analyze their implications for preference optimization in LLMs.

Probabilistic Methods · Variational Inference

Yuhang Xi, Yu-Feng Yu, Chuan-Xian Ren, Zhao-Rong Lai

Parameter-Efficient Fine-Tuning (PEFT) is essential for adapting Large Language Models, yet existing methods typically struggle to balance model capacity with computational efficiency. Standard approaches often enforce rigid low-rank constraints, while dynamic alternatives incur significant memory overheads. To resolve this dilemma, we propose Spectral Bridge Variational Inference (SBVI), a geometric framework that reformulates LoRA not as static parameter optimization, but as a continuous Wasserstein gradient flow on the manifold of Gaussian measures. Rather than fixing the rank at initialization, SBVI governs the singular value evolution via a stochastic differential equation driven by a thermodynamic competition between task gradients and adaptive entropic friction. This mechanism induces a spectral bifurcation that automatically prunes redundant noise modes while amplifying signal-rich components, naturally discovering a layer-wise optimal rank distribution. We derive a scalable algorithm with linear complexity using factorized Riemannian retractions and an Empirical Bayes friction update. Experiments on reasoning and coding benchmarks demonstrate that SBVI achieves state-of-the-art performance, offering superior accuracy and memory efficiency compared to existing static and dynamic adaptation methods.

Deep Learning · Foundation Models

Jialin Yu, Yuxiang Zhou, Haoxuan Li, Junchi Yu, Mengyue Yang, Yulan He, Nevin Zhang, Phil Torr, Ricardo Silva

Adapting to latent confounded shift remains a core challenge in modern AI. This setting is driven by hidden variables that induce spurious correlations between inputs and outputs during training, leading models to rely on non-causal shortcuts. For example, a model may learn to treat metadata (e.g., data source like "Amazon") as a proxy for positive sentiment, causing failure when the source becomes predominantly negative during deployment. To address this *latent confounded shift*, we introduce Causal Fine-Tuning (CFT). Using a structural causal model as an inductive bias, we derive sufficient conditions under which the causal effect of inputs is identifiable (despite latent confounding), and translate these insights into a fine-tuning objective that decomposes representations into high-level causal and low-level spurious components. Instantiating this framework in BERT, we show that learning such causal/spurious representations and adjusting them accordingly yield a more robust predictor. Experiments on spurious correlation injection attacks in text demonstrate that our method outperforms black-box domain generalization baselines, highlighting the benefits of explicitly modeling causal structure.

Probabilistic Methods · Monte Carlo and Sampling Methods

Aaron Havens, Brian Karrer, Neta Shaul

Sampling from unnormalized densities is analogous to the generative modeling problem, but the target distribution is defined by a known energy function instead of data samples. Evaluating the energy function is often costly, and thus a primary challenge is to learn an efficient sampler. We introduce *Flow Sampling*, a framework built on diffusion models and flow matching for the data-free setting. Our training objective is conditioned on a noise sample and regresses onto a *denoising* diffusion drift constructed from the energy function. In contrast, diffusion models' objective is conditioned on a data sample and regresses onto a *noising* diffusion drift. We utilize the interpolant process to minimize the number of energy function evaluations during training, resulting in an efficient and scalable method for sampling unnormalized densities. Furthermore, our formulation naturally extends to Riemannian manifolds, enabling diffusion-based sampling in geometries beyond the Euclidean space. We derive a closed-form formula for the conditional drift on constant curvature manifolds, including hyperspheres and hyperbolic spaces. We evaluate Flow Sampling on synthetic energy benchmarks, large-scale amortized molecular conformer generation, and distributions supported on the sphere, demonstrating strong empirical performance.

Probabilistic Methods · Monte Carlo and Sampling Methods

Robert Gruhlke, Julius Berner, David Sommer, Lorenz Richter

Diffusion models offer a powerful framework for sampling from complex probability densities by learning to reverse a noising process. A common approach involves solving for the time-reversed stochastic differential equation (SDE), which requires the score function of the evolving sample distribution. The logarithm of this distribution's density is governed by a Hamilton-Jacobi-Bellman (HJB) type partial differential equation (PDE). However, current methods for solving this PDE, such as PINNs or trajectory-based techniques, often suffer from long training times and significant sensitivity to hyperparameter tuning. In this work, we introduce a novel and efficient solver for the underlying HJB equation based on the functional tensor train (FTT) format. The FTT representation leverages latent low-rank structures to efficiently approximate high-dimensional functions, enabling both model compression and rapid computation. By integrating this efficient representation with a backward-in-time iterative scheme derived from backward stochastic differential equations (BSDEs), we develop a fast, robust and accurate sampling method. Our approach overcomes primary bottlenecks of existing techniques, enabling high-fidelity sampling from challenging target distributions with improved efficiency.