论文检索

Reinforcement Learning · Deep RL

Calvin Luo, Chen Sun, shuran song

A promising recipe towards intelligent robotic decision-making is the finetuning of pretrained generative control policies, which can summarize offline experience effectively through behavior cloning, with reinforcement learning techniques to adapt them to online experience. In this work we present Diffusion Filtered Exploration via Ensembles (DF-ExpEnse), an exploration technique that meaningfully improves the quality of online experience collection, thus increasing the sample efficiency of the finetuning procedure. DF-ExpEnse first leverages the multimodal modeling capability of the generative control policy to create an expressive and tractably evaluatable candidate set. Then, it utilizes an ensemble of critics to identify an action with high exploration interest that best balances quality with uncertainty. When instantiated in a parallelized fleet, DF-ExpEnse further utilizes cross-agent communication to facilitate collaborative exploration as a group. As it is only used for online experience collection, DF-ExpEnse can be seamlessly integrated on top of existing techniques that seek to finetune pretrained generative control policies via reinforcement learning. We experimentally validate consistent sample-efficiency benefits when using DF-ExpEnse for exploration over both manipulation and locomotion tasks, compared to default finetuning and alternative action selection schemes.

Social Aspects · Accountability, Transparency, and Interpretability

Varun Babbar, Zachery Boner, Margo Seltzer, Cynthia Rudin

Many real-world decisions require prioritizing high-risk cases, such as clinicians prioritizing high-risk patients before lower-risk ones. Falling rule lists (FRLs), which are ordered if--then rules with monotonically decreasing risks, provide an interpretable framework for such tasks; however, their single-path structure yields a highly restricted model class. We introduce falling trees, a new family of interpretable models that enforces the same monotonic risk constraint while permitting tree-structured branching. We present GraviTree, a novel dynamic-programming-with-bounds algorithm for learning the Rashomon set of falling trees under depth and branching constraints, together with bounds that use the falling constraint to provably reduce the search space. Our formulation can interpolate between rule lists and full decision trees, enabling user-desired model expressivity. Across clinical and public-risk datasets, falling trees match or outperform FRLs and other interpretable baselines, often producing lower-sparsity decisions for high-risk instances. Our results show that falling trees strike a practical balance between interpretability, expressiveness, and risk prioritization for high-stakes settings.

Reinforcement Learning · Multi-agent

Fathima Faizal, Asuman Ozdaglar, Martin Wainwright

We study a cooperative form of multi-agent reinforcement learning with state space dynamics and agent interaction controlled by an underlying graph. Each agent has a local state and action, the evolution of the local state depends only on the states and actions in the $1$-hop neighborhood defined by graph. Structured dynamics of this type arise in various applications, including network resource allocation, co-operative games, epidemic control, and wireless scheduling. The global state-action space scales exponentially in the number of agents, so that computing global optimal policies is intractable in the worst-case. We study conditions under which it is possible to approximate the optimal policies by an $m$-hop local policy for each agent, depending only on its $m$-hop neighborhood. By controlling the propagation of influences via a Dobrushin-type stability matrix, we establish that global optimal policies are sharply approximated by $m$-hop local policies whose sub-optimality gap decays exponentially in $m$.

Deep Learning · Robustness

Miaoyun Zhao, Qiang Zhang

Spurious correlations that lead models to correct predictions for the wrong reasons pose a critical challenge for robust real-world generalization. Existing research attributes this issue to group imbalance and addresses it by maximizing group-balanced or worst-group accuracy, which heavily relies on expensive bias annotations. A compromise approach involves predicting bias information using extensively pretrained foundation models, which requires large-scale data and is limited to physically interpretable biases. To address these challenges, we offer a novel perspective by reframing the spurious correlations as imbalances/mismatches in class-conditional distributions caused by general biases, whether interpretable or not, and propose a simple yet effective robust learning method that eliminates the need for bias annotations or predictions. With the goal of maximizing the conditional entropy (uncertainty) of the label given spurious factors, our method leverages a sample reweighting strategy to achieve class-conditional distribution balancing, which automatically highlights minority groups and classes, effectively dismantling spurious correlations and producing a debiased data distribution for classification. Extensive experiments and analysis demonstrate that our approach consistently delivers state-of-the-art performance, rivaling methods that rely on bias supervision.

Social Aspects · Security

Ji Guo, xiaolong qin, Wenbo Jiang, Cencen Liu, Jielei Wang, Jierun Chen

Vision-Language Models (VLMs) have achieved remarkable success in tasks such as image captioning and visual question answering (VQA). However, as their applications become increasingly widespread, recent studies have revealed that VLMs are vulnerable to backdoor attacks. Existing backdoor attacks on VLMs primarily rely on data poisoning by adding visual triggers and modifying text labels, where the induced image–text mismatch makes poisoned samples easy to detect. To address this limitation, we propose the Clean-Label Backdoor Attack on VLMs via Diffusion Models (CBV), which leverages diffusion models to generate natural poisoned examples via score matching. Specifically, CBV modifies the score during the reverse generation process of the diffusion model to guide the generation of poisoned samples that contain triggered image features. To further enhance the effectiveness of the attack, we incorporate the textual information of the triggered images as multimodal guidance during generation. Moreover, to enhance stealthiness, we introduce a GradCAM-guided Mask (GM) that restricts modifications to only the most semantically important regions, rather than the entire image. We evaluate our method on MSCOCO and VQA v2 with four representative VLMs, achieving over 80\% ASR while preserving normal functionality.

Bohdan Turbal, Blossom Metevier, Max Springer, Aleksandra Korolova

Although there is a rich literature on adversarial attacks on large language models, their current practical impact is limited. Gradient-based attacks such as Greedy Coordinate Gradient (Zou et al., 2023) typically produce high-perplexity, incoherent suffixes that are easily detectable and thus easy to guard against, especially in combination with other defense-in-depth techniques (Bengio et al., 2024). On the other hand, attacks that aim to produce coherent prompts often alter the semantic intent of the original query. When the model complies with such altered query, it often produces a response that is not actually useful for the original query, thus incurring the so-called "jailbreak tax". In this work, we introduce a novel framework that can efficiently generate adversarial attacks against safety-aligned models while maintaining low perplexity and high semantic adherence to the adversary's original intent. The framework, Greedy Coordinate Diffusion (GCD), leverages the generative priors of discrete diffusion language models to guide the search for adversarial suffixes that achieve semantic coherence and adherence. Furthermore, unlike GCG, GCD does not require direct gradient access, allowing it to operate in a gray-box setting. We empirically demonstrate the power of GCD by showing it achieves state-of-the-art attack success rates against aligned models, and its adversarial prompts are not detected by semantic filters such as llama-guard-3.

Social Aspects · Safety

Chengcan Wu, Zhixin Zhang, Mingqian Xu, Zeming Wei, Meng Sun

Large Language Model (LLM)-based Multi-Agent Systems (MAS) have become a popular paradigm of AI applications. However, trustworthiness issues in MAS remain a critical concern. Unlike challenges in single-agent systems, MAS involve more complex communication processes, making them susceptible to corruption attacks. To mitigate this issue, several defense mechanisms have been developed based on the graph representation of MAS, where agents represent nodes and communications form edges. Nevertheless, these methods predominantly focus on static graph defense, attempting to either detect attacks in a fixed graph structure or optimize a static topology with certain defensive capabilities. To address this limitation, we propose a dynamic defense paradigm for MAS graph structures, which continuously monitors communication within the MAS graph, then dynamically adjusts the graph topology, accurately disrupts malicious communications, and effectively defends against evolving and diverse dynamic attacks. Experimental results in increasingly complex and dynamic MAS environments demonstrate that our method significantly outperforms existing MAS defense mechanisms, contributing an effective guardrail for their trustworthy applications.

Deep Learning · Large Language Models

David Baek, Andrew Estornell, Yichi Zhang, Muhammad Faaiz Taufiq, Jean-Francois Ton, Jie Mei, Tao Wang

Test-time scaling methods such as majority vote aggregation and iterative refinement (e.g., self-reflection or multi-agent inference) improve reasoning performance by leveraging multiple solution samples. However, their efficacy depends not only on raw performance, but critically on the distribution of errors across samples. When errors concentrate, (a) aggregation accuracy degrades, as the majority vote may select a shared mistake, and (b) confidence in common mistakes may suppress exploration in iterative refinement. We argue that improving correctness alone is not sufficient to mitigate these issues; to this end, we propose to explicitly shape error distributions to improve aggregation. First, we introduce a theoretically grounded \textbf{diverse failure reward} that incentivizes calibrated disagreement within model errors. We prove that this reward directly optimizes majority-vote accuracy: policies achieving higher reward attain higher expected majority-vote performance, and vice versa. We further show that this theoretical property generalizes to iterative refinement. Second, we introduce \textbf{anti-votes}, in which the model predicts the most common mistake alongside its solution, allowing probability mass on dominant errors to be explicitly reweighted. We identify conditions under which anti-votes are guaranteed to improve majority-vote accuracy. Empirically, across three model families of varying sizes and four benchmarks, we show that both approaches substantially improve majority vote and iterative refinement performance without degrading single-sample accuracy.

General Machine Learning · Clustering

Yipin Hu, Yanxi Liu, Fangxi Liu, Yanwei Yu, Lei Meng, Jie Wen, Guoqing Chao

Handling large-scale incomplete multi-view data poses a significant challenge in unsupervised representation learning. While anchor-based strategies have alleviated computational burdens, they typically rely on shallow bipartite graphs restricted to pairwise relations, failing to capture complex high-order correlations among samples. Furthermore, existing methods often treat observed and missing instances indiscriminately, ignoring the distributional shifts that lead to systematic bias in consensus anchor learning. To address these limitations, we propose a novel framework tailored for scalability and robustness, termed Hypergraph-Augmented Incomplete Multi-View Clustering (HA-IMVC). Unlike traditional approaches, HA-IMVC constructs a consensus anchor-guided hypergraph that explicitly models group-wise interactions, thereby preserving structural integrity even under high missing rates. Crucially, we incorporate a dual-adaptive reweighting mechanism that calibrates importance at both the view and sample levels. This strategy adaptively penalizes severely incomplete samples to mitigate bias while harmonizing inconsistent views. Extensive experiments on diverse benchmarks demonstrate that HA-IMVC achieves superior clustering accuracy and maintains high efficiency, even in scenarios characterized by severe data incompleteness.

Social Aspects · Safety

Chengquan Guo, Yuzhou Nie, Chulin Xie, Zinan Lin, Wenbo Guo, Bo Li

Existing research on CodeGen AI security mainly focuses on red teaming, which aims to uncover vulnerabilities and risks in AI-generated code. However, progress on the blue teaming side remains limited, as effective defenses require a deep security analysis of given tasks and edge cases. To fill in this gap, we propose BlueCodeAgent, an end-to-end blue teaming agent powered by automated red teaming. Our red teaming component generates diverse risky instances, providing effective edge cases and guidance for the subsequent blue teaming process. Our blue teaming agent then conducts multi-level defense, leveraging these red teaming examples to detect previously seen and unseen risk scenarios through constitution summarization and dynamic code analysis. Our evaluation across four representative code-related tasks–bias instruction detection, malicious instruction detection, vulnerable code detection, and prompt injection detection–shows that BlueCodeAgent achieves significant gains over diverse baselines. In particular, for vulnerability detection tasks, BlueCodeAgent integrates dynamic analysis to effectively reduce false positives, a challenging problem as base models tend to be over-conservative. Overall, with GPT-4o as the base model, BlueCodeAgent achieves an average F1 score improvement of 14.7% across four tasks compared to directly prompting the model, attributed to its ability to summarize actionable constitutions and perform dynamic analysis.

General Machine Learning · Everything Else

Zijie Peng, Enneng Yang, Yifei Cheng, Hongliang Yuan, Fei Ma, Xiaochun Cao, Li Shen

Custom diffusion models (CDMs) have demonstrated impressive success in visual personalization tasks by enabling the generation of user-specific concepts. However, existing CDMs typically assume that personalized concepts are static and rely on costly model merging or sequential updates that are prone to catastrophic forgetting as new concepts are introduced. To address these limitations, we propose a Submodular Concept Neuron Selection method (SCNS), to solve CDMs with continual personalized concepts, which formulates continual personalization as a constrained submodular optimization problem to select a minimal yet sufficient set of concept-specific neurons under diminishing returns. SCNS combines a Facility Location-based coverage objective to suppress semantic redundancy, a Fisher-weighted risk proxy to protect previously learned concepts, and a cost-aware greedy rule to balance stability and plasticity with extreme sparsity. Extensive experiments demonstrate that SCNS achieves state-of-the-art performance in image alignment and anti-forgetting, while enabling fusion-free continual personalization by modifying only 0.41% of the total parameters for each concept on average. Our implementation is available at SCNS.

General Machine Learning · Unsupervised and Semi-supervised Learning

Rasmus Tirsgaard, Laurits Fredsgaard, Marisa Wodrich, Mikkel Jordahn, Mikkel Schmidt

Machine learning is transforming molecular sciences by accelerating property prediction, simulation, and the discovery of new molecules and materials. Acquiring labeled data in these domains is often costly and time-consuming, whereas large collections of unlabeled molecular data are readily available. Standard semi-supervised learning methods often rely on label-preserving augmentations, which are challenging to design in the molecular domain, where minor changes can drastically alter properties. In this work, we show that semi-supervised methods that rely on an ensemble consensus can boost predictive accuracy across a diverse range of molecular datasets, task types, and graph neural network architectures. We find that training with an ensemble consensus objective increases robustness in models and exhibit an effect similar to knowledge distillation; an individual member of an ensemble trained this way outperforms a full ensemble trained in a traditional supervised fashion in almost all cases. In addition, this type of semi-supervised training reduces calibration error.

Deep Learning · Other Representation Learning

Wuyang Zhou, Yuxuan Gu, Giorgos Iacovides, Danilo Mandic

The success of Hyper-Connections (HC) in neural networks (NN) has also highlighted issues related to its training instability and restricted scalability. The Manifold-Constrained Hyper-Connections (mHC) mitigate these challenges by projecting the residual connection space onto a Birkhoff polytope, however, it faces two issues: 1) its iterative Sinkhorn-Knopp (SK) algorithm does not always yield exact doubly stochastic residual matrices; 2) mHC incurs a prohibitive $\mathcal{O}(n^3C)$ parameter complexity with $n$ as the width of the residual stream and $C$ as the feature dimension. The recently proposed mHC-lite reparametrizes the residual matrix via the Birkhoff-von-Neumann theorem to guarantee double stochasticity, but also faces a factorial explosion in its parameter complexity, $\mathcal{O} \left( nC \cdot n! \right)$. To address both challenges, we propose **KromHC**, which uses the $\underline{\text{Kro}}$necker products of smaller doubly stochastic matrices to parametrize the residual matrix in $\underline{\text{mHC}}$. By enforcing manifold constraints across the factor residual matrices along each mode of the tensorized residual stream, KromHC guarantees exact double stochasticity of the residual matrices while reducing parameter complexity to $\mathcal{O}(n^2C)$. Comprehensive experiments demonstrate that KromHC matches or even outperforms state-of-the-art (SOTA) mHC variants, while requiring significantly fewer trainable parameters.

Applications · Health / Medicine

Ziyue Wang, Linghan Cai, Chang Low, Haofeng Liu, Junde Wu, Jinyu Wang, Rui Wang, Lei Song, Jiang Bian, Jingjing Fu 等

3D CT analysis spans a continuum from low-level perception to high-level clinical understanding. Existing 3D-oriented analysis methods adopt either isolated task-specific modeling or task-agnostic end-to-end paradigms to produce one-hop outputs, impeding the systematic accumulation of perceptual evidence for downstream reasoning. In parallel, recent multimodal large language models (MLLMs) exhibit improved visual perception and can integrate visual and textual information effectively, yet their predominantly 2D-oriented designs fundamentally limit their ability to perceive and analysis volumetric medical data. To bridge this gap, we propose 3DMedAgent, an unified agent that enables 2D MLLMs to perform general 3D CT analysis without 3D-specific fine-tuning. 3DMedAgent coordinates heterogeneous visual and textual tools through a flexible MLLM agent, progressively decomposing complex 3D analysis into tractable subtasks that transition from global to regional views, from 3D volumes to informative 2D slices, and from visual evidence to structured textual representations. Central to this design, 3DMedAgent maintains a long-term structured memory that aggregates intermediate tool outputs and supports query-adaptive, evidence-driven multi-step reasoning. We further introduce the DeepChestVQA benchmark for evaluating unified perception-to-understanding capabilities in 3D thoracic imaging. Experiments across over 40 tasks demonstrate that 3DMedAgent consistently outperforms general, medical, and 3D-specific MLLMs, highlighting a scalable path toward general-purpose 3D clinical assistants. Anonymized code and data are available \href{https://anonymous.4open.science/r/3DMedAgent-D7D7}{here}.

Deep Learning · Large Language Models

Charlie Zhang, Graham Neubig, Xiang Yue

Recent reinforcement learning (RL) techniques have yielded impressive reasoning improvements in language models, yet it remains unclear whether post-training truly extends a model’s reasoning ability beyond what it acquires during pre-training. A central challenge is the lack of control in modern training pipelines: large-scale pre-training corpora are opaque, mid-training is often underexamined, and RL objectives interact with unknown prior knowledge in complex ways. To resolve this ambiguity, we develop a fully controlled experimental framework that isolates the causal contributions of pre-training, mid-training, and RL-based post-training. Our approach employs synthetic reasoning tasks with explicit atomic operations, parseable step-by-step reasoning traces, and systematic manipulation of training distributions. We evaluate models along two axes: extrapolative generalization to more complex compositions and contextual generalization across surface contexts. Using this framework, we reconcile competing views on RL’s effectiveness. We show that: 1) RL produces true capability gains (pass@128) only when pre-training leaves sufficient headroom and when RL data target the model’s edge of competence, tasks at the boundary that are difficult but not yet out of reach. 2) Contextual generalization requires minimal yet sufficient pre-training exposure, after which RL can reliably transfer. 3) Mid-training significantly enhances performance under fixed compute compared with RL only, demonstrating its central but underexplored role in training pipelines. 4) Process-level rewards reduce reward hacking and improve reasoning fidelity. Together, these results clarify the interplay between pre-training, mid-training, and RL, offering a foundation for understanding and improving reasoning LM training strategies.

Bhaskar Ray Chaudhury, Jugal Garg, Eklavya Sharma, Jiaxin Song

We study equilibrium pricing in oligopolistic data markets with budget-constrained buyers (e.g., ML companies purchasing data to improve model accuracy) and strategic data sellers. Sellers compete by setting prices for their datasets, giving rise to a pricing game whose pure Nash equilibria correspond to equilibrium prices. While equilibrium prices are guaranteed for rivalrous goods via competitive equilibrium, we show that the non-rivalry of data fundamentally alters this picture: an exact Nash equilibrium need not exist, and in fact no 1.364-approximate equilibrium exists under uniform pricing. We therefore investigate relaxed equilibrium notions. Allowing sellers to use beyond-uniform pricing—specifically, piecewise-linear convex pricing functions—guarantees approximate stability within a constant factor: there exists a pricing profile in which no seller can improve revenue by a factor of two by deviating to any uniform price (a 2-approximate Nash equilibrium). Finally, our simulations demonstrate fast convergence and empirical approximation guarantees that outperform the worst-case bound of 2.

Theory · Learning Theory

Ruoxin Yuan, Lijun Ding

Exact sparse methods based on $\ell_0$ constraints are increasingly used for interpretable and scalable time series modeling, where one aims to recover a small set of informative lags/factors while maintaining strong predictive performance and low computational cost. Despite their empirical success, finite-sample and computational guarantees for such methods under temporal dependence remain limited. In this paper, we study $\ell_0$-constrained least squares for time series generated by $\alpha$-mixing stationary Gaussian processes with sparse coefficients. We establish high-probability restricted strong convexity/smoothness (RSC/RSS) for the empirical quadratic loss. Leveraging these conditions, we derive nonasymptotic statistical guarantees and computational complexities for a series of exact sparse methods, including iterative hard thresholding (IHT). We apply our theoretical results to Gaussian vector autoregressive (VAR) models and obtain new guarantees. Experiments on synthetic sparse VAR models and real-world mobility time series demonstrate that exact sparse methods recover lag structure more accurately and interpretably than some classical methods, while achieving comparable prediction error with substantially lower computational cost.

Deep Learning · Large Language Models

Sophie Wang, Phillip Isola, Brian Cheung

How should the sequence of hidden states produced during autoregressive generation be compressed into a representation that reflects the model’s internal state? We study representations derived from generated tokens and compare them to grounded embeddings across several domains. We find that pooling embeddings across tokens produces more informative representations than any individual token. This observation is consistent with semantic information being distributed across generated tokens rather than localized to a single position. In this setting, alignment provides a way to study how a model’s internal representations evolve and pooling offers a more reliable summary of the model's state across generation.

Wei Yeo, Rui Mao, Moloud Abdar, Ranjan Satapathy, Erik Cambria

Multimodal models like CLIP have gained significant attention due to their remarkable zero-shot performance across various tasks. However, studies have revealed that CLIP can inadvertently learn spurious associations between target variables and confounding factors. To address this, we introduce \textsc{Locate-Then-Correct} (LTC), a contrastive framework that identifies spurious attention heads in Vision Transformers via mechanistic insights and mitigates them through targeted ablation. Furthermore, LTC identifies salient, task-relevant attention heads, enabling the integration of discriminative features through orthogonal projection to improve classification performance. We evaluate LTC on benchmarks with inherent background and gender biases, achieving over a > 50% gain in worst-group accuracy compared to non-training post-hoc baselines. Additionally, we visualize the representation of selected heads and find that the presented interpretation corroborates our contrastive mechanism for identifying both spurious and salient attention heads.

Reinforcement Learning · Batch/Offline

Huy Hoang, Tien Mai, Pradeep Varakantham, Tanvi Verma

Offline imitation learning typically learns from expert and unlabeled demonstrations, yet often overlooks the valuable signal in explicitly undesirable behaviors. In this work, we study offline imitation learning from contrasting behaviors, where the dataset contains both expert and undesirable demonstrations along with an unlabeled set of demonstrations. We propose a novel formulation that optimizes a difference of KL divergences over the state-action visitation distributions of expert and undesirable (or bad) data. Although the resulting objective is a DC (Difference-of-Convex) program, we prove that it becomes *convex* when expert demonstrations outweigh undesirable demonstrations, enabling a practical and stable non-adversarial training objective. Our method avoids adversarial training and handles both positive and negative demonstrations in a unified framework. Extensive experiments on standard offline imitation learning benchmarks demonstrate that our approach consistently outperforms state-of-the-art baselines.