论文检索

Social Aspects · Security

Jiazhen Yan, Ziqiang Li, Fan Wang, Boyu Wang, Ziwen He, Zhangjie Fu

The rapid progress of generative models such as GANs and diffusion models has led to the widespread proliferation of AI-generated images, raising concerns about misinformation, privacy violations, and trust erosion in digital media. Although large-scale multimodal models like CLIP offer strong transferable representations for detecting synthetic content, fine-tuning them often induces catastrophic forgetting, which degrades pre-trained priors and limits cross-domain generalization. To address this issue, we propose the Distillation-guided Gradient Surgery Network (DGS-Net), a novel framework that preserves transferable pre-trained priors while suppressing task-irrelevant components. Specifically, we introduce a gradient-space decomposition that separates harmful and beneficial descent directions during optimization. By projecting task gradients onto the orthogonal complement of harmful directions and aligning with beneficial ones distilled from a frozen CLIP encoder, DGS-Net achieves unified optimization of prior preservation and irrelevant suppression. Extensive experiments on 50 generative models demonstrate that our method outperforms state-of-the-art approaches by an average margin of 6.6%, achieving superior detection performance and generalization across diverse generation techniques.

Social Aspects · Alignment

Cameron Tice, Puria Radmard, Samuel Ratnam, Andy Kim, David Africa, Kyle O'Brien

Pretraining corpora contain extensive discourse about AI systems, yet the causal influence of this discourse on downstream alignment remains poorly understood. If prevailing descriptions of AI behaviour are predominantly negative, LLMs may internalise corresponding behavioural priors, giving rise to self-fulfilling misalignment. This paper provides the first controlled study of this hypothesis by pretraining 6.9B-parameter LLMs with varying amounts of (mis)alignment discourse. We find that discussion of AI contributes to misalignment. Upsampling synthetic training documents about AI misalignment leads to a notable increase in misaligned behaviour. Conversely, upsampling documents about aligned behaviour reduces misalignment scores from 45% to 9%. We consider this evidence of self-fulfilling alignment. These effects are dampened, but persist through post-training. Our findings establish the study of how pretraining data shapes alignment priors, or alignment pretraining, as a complement to post-training. We recommend practitioners pretrain for alignment as well as capabilities.

General Machine Learning · Hardware and Software

Anselm Paulus, Andreas René Geist, Vit Musil, Sebastian Hoffmann, Georg Martius

Automatic differentiation (AD) frameworks such as JAX and PyTorch have enabled gradient-based optimization for a wide range of scientific fields. Yet, many ''hard'' primitives in these libraries such as thresholding, Boolean logic, discrete indexing, and sorting operations yield zero or undefined gradients that are not useful for optimization. While numerous ''soft'' relaxations have been proposed that provide informative gradients, the respective implementations are fragmented across projects, making them difficult to combine and compare. This work introduces **SoftJAX** and **SoftTorch**, open-source, feature-complete libraries for *soft differentiable programming*. These libraries provide a variety of soft functions as drop-in replacements for their hard JAX and PyTorch counterparts. This includes (i) elementwise operators such as *clip* or *abs*, (ii) utility methods for manipulating Booleans and indices via fuzzy logic, (iii) axiswise operators such as *sort* or *rank* -- based on optimal transport or permutahedron projections, and (iv) offer full support for straight-through gradient estimation. Overall, SoftJAX and SoftTorch make the toolbox of soft relaxations easily accessible to differentiable programming, as demonstrated through benchmarking and a practical case study.

Deep Learning · Theory

Dong Sun, Rahul Nittala, Rebekka Burkholz

Despite their practical success, it remains unclear why Mixture of Experts (MoE) models can outperform dense networks beyond sheer parameter scaling. We study an iso-parameter regime where inputs exhibit latent modular structure but are corrupted by feature noise, a proxy for noisy internal activations. We show that sparse expert activation acts as a noise filter: compared to a dense estimator, MoEs achieve lower generalization error under feature noise, improved robustness to perturbations, and faster convergence speed. Empirical results on synthetic data and real-world language tasks corroborate the theoretical insights, demonstrating consistent robustness and efficiency gains from sparse modular computation.

Deep Learning · Graph Neural Networks

Celia Rubio-Madrigal, Rebekka Burkholz

Graph neural networks (GNNs) are widely believed to excel at node representation learning through trainable neighborhood aggregations. We challenge this view by introducing Fixed Aggregation Features (FAFs), a training-free approach that transforms graph learning tasks into tabular problems. This simple shift enables the use of well-established tabular methods, offering strong interpretability and the flexibility to deploy diverse classifiers. Across 14 benchmarks, well-tuned multilayer perceptrons trained on FAFs rival or outperform state-of-the-art GNNs and graph transformers on 12 tasks—often using only mean aggregation. The only exceptions are the Roman Empire and Minesweeper datasets, which typically require unusually deep GNNs. To explain the theoretical possibility of non-trainable aggregations, we connect our findings to Kolmogorov–Arnold representations and discuss when mean aggregation can be sufficient. In conclusion, our results call for (i) richer benchmarks benefiting from learning diverse neighborhood aggregations, (ii) strong tabular baselines as standard, and (iii) employing and advancing tabular models for graph data to gain new insights into related tasks.

General Machine Learning · Evaluation

Justin Wang, Andreas Bigger, Xiaohai Xu, Justin W. Lin, Andy Applebaum, Tejal Patwardhan, Alpin Yukseloglu, Olivia Watkins

Smart contracts on public blockchains now manage large amounts of value, and vulnerabilities in these systems can lead to substantial losses. As AI agents become more capable at reading, writing, and running code, it is natural to ask how well they can already navigate this landscape, both in ways that improve security and in ways that might increase risk. We introduce EVMbench, an evaluation that measures the ability of agents to detect, patch, and exploit smart contract vulnerabilities. EVMbench draws on 120 curated vulnerabilities from 37 repositories and, in the most realistic setting, uses programmatic grading based on tests and blockchain state under a local Ethereum execution environment. We evaluate a range of frontier agents and find that they are capable of discovering and exploiting vulnerabilities end-to-end against live blockchain instances. We also compare various agent scaffolds and find that in some cases performance gains due to scaffolding improvements alone rival gains due to increased model quality. We release code, tasks, and tooling to support continued measurement of these capabilities and future work on security.

Deep Learning · Robustness

Jiate Li, Defu Cao, Li Li, Wei Yang, Yuehan Qin, Chenxiao Yu, Tiannuo Yang, Ryan A Rossi, Yan Liu, Xiyang Hu 等

Large language models (LLMs) have been serving as effective backbones for retrieval systems, including Retrieval-Augmentation-Generation (RAG), Dense Information Retriever (IR), and Agent Memory Retrieval. Recent studies have demonstrated that such LLM-based Retrieval (LLMR) is vulnerable to adversarial attacks, which manipulates documents by token-level injections and enables adversaries to either boost or diminish these documents in retrieval tasks. However, existing attack studies mainly (1) presume a known query is given to the attacker, and (2) highly rely on access to the victim model's parameters or interactions, which are hardly accessible in real-world scenarios, leading to limited validity. To further explore the secure risks of LLMR, we propose a practical black-box attack method that generates transferable injection tokens based on zero-shot surrogate LLMs without need of victim queries or victim models knowledge. The effectiveness of our attack raises such a robustness issue that similar effects may arise from benign or unintended document edits in the real world. To achieve our attack, we first establish a theoretical framework of LLMR and empirically verify it. Under the framework, we simulate the transferable attack as a min-max problem, and propose an adversarial learning mechanism that finds optimal adversarial tokens with learnable query samples. Our attack is validated to be effective on benchmark datasets across popular LLM retrievers.

Deep Learning · Large Language Models

Paulius Rauba, Dominykas Seputis, Patrikas Vanagas, Mihaela van der Schaar

Least privilege is a core security principle: grant each request only the minimum access needed to achieve its goal. Deployed language models almost never follow it, instead being exposed through a single API endpoint that serves all users and requests. This gap exists not because least privilege would be unhelpful—deployments would benefit greatly from reducing unnecessary capability exposure. The real obstacle is definitional and mechanistic: what does "access" mean inside a language model, and how can we enforce it without retraining or deploying multiple models? We take inspiration from least privilege in computer systems and define a class of models called *least-privilege language models*, where privilege is *reachable internal computation* during the forward pass. In this view, lowering privilege literally shrinks the model's accessible function class (as opposed to denying access via learned policies). We formalize deployment-time control as a monitor--allocator--enforcer stack, separating (i) request-time signals, (ii) a decision rule that allocates privilege, and (iii) an inference-time mechanism that selects privilege. We then propose *Nested Least-Privilege Networks*, a shape-preserving, rank-indexed intervention that provides a smooth, reversible control knob. We show that this knob yields policy-usable privilege--utility frontiers and enables selective suppression of targeted capabilities with limited collateral degradation across various policies. Most importantly, we see this as a defense of a completely new deployment paradigm which challenges the premise that we can only have output-level control of language models.

General Machine Learning · Hardware and Software

Anselm Paulus, Andreas René Geist, Vit Musil, Sebastian Hoffmann, Georg Martius

Automatic differentiation (AD) frameworks such as JAX and PyTorch have enabled gradient-based optimization for a wide range of scientific fields. Yet, many ''hard'' primitives in these libraries such as thresholding, Boolean logic, discrete indexing, and sorting operations yield zero or undefined gradients that are not useful for optimization. While numerous ''soft'' relaxations have been proposed that provide informative gradients, the respective implementations are fragmented across projects, making them difficult to combine and compare. This work introduces **SoftJAX** and **SoftTorch**, open-source, feature-complete libraries for *soft differentiable programming*. These libraries provide a variety of soft functions as drop-in replacements for their hard JAX and PyTorch counterparts. This includes (i) elementwise operators such as *clip* or *abs*, (ii) utility methods for manipulating Booleans and indices via fuzzy logic, (iii) axiswise operators such as *sort* or *rank* -- based on optimal transport or permutahedron projections, and (iv) offer full support for straight-through gradient estimation. Overall, SoftJAX and SoftTorch make the toolbox of soft relaxations easily accessible to differentiable programming, as demonstrated through benchmarking and a practical case study.

Social Aspects · Privacy

Zhiyuan Cao, Zeyu Ma, Chenhao Yang, HAN ZHENG, Mingang Chen

We propose Obfuscated Semantic Null space Injection for Privacy (OSNIP), a lightweight client-side encryption framework for privacy-preserving LLM inference. Generalizing the geometric intuition of linear kernels to the high-dimensional latent space of LLMs, we formally define the ``Obfuscated Semantic Null Space'', a high-dimensional regime that preserves semantic fidelity while enforcing near-orthogonality to the original embedding. By injecting perturbations that project the original embedding into this space, OSNIP ensures privacy without any post-processing. Furthermore, OSNIP employs a key-dependent stochastic mapping that synthesizes individualized perturbation trajectories unique to each user. Evaluations on 12 generative and classification benchmarks show that OSNIP achieves state-of-the-art performance, sharply reducing attack success rates while maintaining strong model utility under strict security constraints.

Social Aspects · Security

Jonathan Nöther, Adish Singla, Goran Radanovic

LLM-based multi-agent systems have demonstrated impressive capabilities, but they also introduce significant safety risks when individual agents fail or behave adversarially. In this work, we study the automated design of agentic systems that remain safe even when a subset of agents is compromised. We formalize this challenge as a Stackelberg security game between a system designer (the Meta-Agent) and a best-responding Meta-Adversary that selects and compromises a subset of agents to minimize safety. We propose Meta-Adversary–Meta-Agent (MaMa), a novel algorithm for approximately solving this game and automatically designing safe agentic systems. Our approach uses LLM-based adversarial search, where the Meta-Agent iteratively proposes system designs and receives feedback based on the strongest attacks discovered by the Meta-Adversary. Empirical evaluations across diverse environments show that systems designed with MaMa consistently defend against worst-case attacks while maintaining performance comparable to systems optimized solely for task success. Moreover, the resulting systems generalize to stronger adversaries, as well as ones with different attack objectives or underlying LLMs, demonstrating robust safety beyond the training setting.

Social Aspects · Safety

Eyon Jang, Damon Falck, Joschka Cedric Braun, Nathalie Kirch, Achyutha Menon, Perusha Moodley, Scott Emmons, Roland S. Zimmermann, David Lindner

Reinforcement learning (RL) has become essential to the reasoning and alignment post-training of large language models (LLMs). However, successful RL relies on sufficient exploration of diverse actions by the model during training. We study whether RL is robust to *exploration hacking*, where a model strategically alters its exploration during training to influence the subsequent training outcome. First, we create model organisms of exploration hacking by using fine-tuning-based "locking" techniques; we show that these models can successfully resist RL-based capability elicitation in AI R&D and agentic biosecurity environments, while maintaining performance on closely related tasks. Next, we use our model organisms to evaluate the effectiveness of monitoring techniques as detection methods for exploration hacking. Finally, we show that current frontier models can reason effectively about suppressing their exploration when presented with simulated RL environments and encouraged to act strategically. Together, our results empirically establish exploration hacking as a failure mode of RL on sufficiently capable LLMs.

Social Aspects · Security

Jane Downer, Yingdan Shi, Ziyan Liu, Ren Wang, Binghui Wang

Graph Neural Networks (GNNs) are widely deployed in industry, making their intellectual property valuable. However, protecting GNNs from unauthorized use remains a challenge. Watermarking offers a solution by embedding ownership information into models. Existing watermarking methods have two limitations: First, they rarely focus on graph data or GNNs. Second, the \emph{de facto} backdoor-based method relies on manipulating training data, which can introduce ownership ambiguity through misclassification and vulnerability to data poisoning attacks that can interrupt the backdoor mechanism. Our explanation-based watermarking inherits the strengths of backdoor-based methods (e.g., black-box verification) without data manipulation, eliminating ownership ambiguity and data dependencies. In particular, we watermark GNN explanations such that these explanations are statistically distinct from others, so ownership claims must be verified through statistical significance. We theoretically prove that, even with full knowledge of our method, locating the watermark is NP-hard. Empirically, our method demonstrates robustness to fine-tuning and pruning attacks. By addressing these challenges, our approach significantly advances GNN intellectual property protection.

General Machine Learning · Everything Else

Xiuying Wang, Yichen Li, Hang Su, Gaozhuo Liu, Shiwei Li, Chuang Zhao, Jiangming Shi, Imran Razzak

Federated Incremental Learning (FIL) aims to learn streaming tasks across distributed clients without catastrophic forgetting while preserving privacy. Most existing methods focus on sample-based replay techniques, which mitigate forgetting by replaying historical data samples. However, such methods often face challenges related to data privacy risks and significant resource overheads, making them impractical and difficult to deploy on edge devices with limited resources. To address this challenge, we propose a novel and \underline{Li}ghtweight \underline{F}ederated \underline{I}ncremental \underline{L}earning framework called \textbf{Li-FIL} that leverages dense features synthesized by a secure generator on the server to enable efficient feature-based replay on decoupled local models. More specifically, each client extracts high-confidence features from the new task, applies mixup to obtain a dense feature representation, and then privatizes these features before uploading them to the server, which reduces both storage and communication overhead. A generator is deployed on the server to learn the distributions of different clients and generate global features for replay. Moreover, to enable clients to better learn from these dense features, we decouple the local model into two components: a feature extractor and a classifier. This design allows feature replay and the alignment between new and previous features to be conducted separately and more effectively. Extensive experiments demonstrate that Li-FIL outperforms other state-of-the-art methods by up to 10.14 in terms of accuracy on both old and new tasks with superior resource efficiency.

General Machine Learning · Transfer, Multitask and Meta-learning

Zhiyuan Yu, Enneng Yang, Hao Jiang, Guojie Zhu, Feihong He, Peng Wang, Li Shen

Upcycling, a strategy that initializes Mixture-of-Experts (MoE) by replicating pre-trained feed-forward or MoE networks to expand model capacity, has become a popular method in continual learning due to its effectiveness in mitigating catastrophic forgetting. However, existing paradigms rely on indiscriminate expansion prioritize performance at the cost of extreme inefficiency, introducing parameter redundancy without exploiting the structural heterogeneity essential for counteracting forgetting with architectural economy. To address this, we investigate the determinants of forgetting in training dynamics using Centered Kernel Alignment (CKA) and loss landscape flatness to analyze the behavior of pre- and post-expansion MoE layers, uncovering instability in deep-layer representations and heterogeneous expert sensitivity to new tasks, thereby demonstrating the potential of selective upcycling to eliminate redundancy. Consequently, we propose a dynamic bilevel optimization framework to guide adaptive upcycling, featuring an outer loop employing a Gumbel-Softmax differentiable mask to perform Neural Architecture Search (NAS) for adaptive growth, while an inner loop optimizes weight updates via task objectives and CKA-regularized replay. Experiments on TRACE benchmark demonstrate that our proposed method achieves better average accuracy with 80\% forgetting reduction, while effectively eliminating 60\% of redundant parameter expansion that standard upcycling would introduce.

Social Aspects · Safety

Qianshan Wei, Tengchao Yang, Yaochen Wang, Xinfeng Li, Lijun Li, Zhenfei Yin, Yi Zhan, Thorsten Holz, Zhiqiang Lin, XiaoFeng Wang

Large Language Model (LLM) agents use memory to learn from past interactions. However, this reliance on memory introduces a critical security risk: an adversary can inject seemingly harmless records into an agent's memory to manipulate its future behavior. This vulnerability is characterized by two core aspects: First, the malicious effect of injected records is only activated within a specific context, making them hard to detect when individual memory entries are audited in isolation. Second, once triggered, the manipulation can initiate a self-reinforcing error cycle: the corrupted outcome is stored as precedent, which not only amplifies the initial error but also progressively lowers the threshold for similar attacks in the future. To address these challenges, we introduce \emph{A-MemGuard} (\underline{A}gent-\underline{Mem}ory \underline{Guard}), the first defense framework for LLM agent memory. The core idea of our work is the insight that memory itself must become both \emph{self-checking} and \emph{self-correcting}. Without modifying the agent's core architecture, A-MemGuard combines two mechanisms: (1) \textbf{consensus-based validation}, which detects anomalies by comparing reasoning paths derived from multiple related memories and (2) a \textbf{dual-memory structure}, where detected failures are distilled into ``lessons'' stored separately and consulted before future actions, breaking error cycles and enabling adaptation. Comprehensive evaluations on multiple benchmarks show that A-MemGuard effectively cuts attack success rates by over 95\% while incurring a minimal utility cost. This work shifts LLM memory security from static filtering to a proactive, experience-driven model where defenses strengthen over time.

Social Aspects · Security

Zhihan Qin, Jiahao Chen, Chunyi Zhou, Yuwen Pu, Chunqiang Hu, Xiaolei Liu, Shouling Ji

Vision Language Models (VLMs) achieve strong performance across multi-modal tasks but remain vulnerable to universal adversarial perturbations (UAPs). Existing UAP methods mainly operate on the visual modality, overlooking structured textual semantics and cross-modal interactions, which limits their ability to disrupt alignment and generalize across tasks and model architectures. To address these limits, we propose **Multi-modal Attention Distraction Adversarial Attack (MADA-Attack)** framework. We begin by conducting several insight experiments and discover that modality attention distributes differently over layers and early phase of optimization is decisive. Building on these observations, we introduce Semantic Token Manipulation (STM) to steer text-guided attention, and Fused Embedding Training (FET) to jointly optimize textual and visual embedding losses for coordinated misalignment. We further incorporate an Adaptive Data Augmentation (ADA) strategy that dynamically balances attack strength, transferability, and training efficiency. Extensive experiments demonstrate that **MADA-Attack** consistently achieves state-of-the-art performance and strong transferability while remaining computationally lightweight, with an average ASR of 82.60\% and 73.42\% in zero-shot classification and image captioning tasks. For the visual question answering (VQA) and I-T Retrieval task, our method exceeds the SOTA baseline by 10\%. Our code will be available soon.

Deep Learning · Large Language Models

Samuele Marro, Phil Torr

While the Internet's core infrastructure was designed to be open and universal, today’s application layer is dominated by closed, proprietary platforms. Open and interoperable APIs require significant investment, and market leaders have little incentive to enable data exchange that could erode their user lock-in. We argue that LLM-based agents fundamentally disrupt this status quo. Agents can automatically translate between data formats and interact with interfaces designed for humans: this makes interoperability dramatically cheaper and effectively unavoidable. We name this shift *universal interoperability*: the ability for any two digital services to exchange data seamlessly using AI-mediated adapters. Universal interoperability undermines monopolistic behaviours and promotes data portability. However, it can also lead to new security risks and technical debt. Our position is that the ML community should embrace this development while building the appropriate frameworks to mitigate the downsides. By acting now, we can harness AI to restore user freedom and competitive markets without sacrificing security.

Social Aspects · Security

Quang Duc Nguyen, Siyuan Liang, Yiming Li, Fushuo Huo, Dacheng Tao

Time Series Forecasting (TSF) plays a critical role across many domains, yet it is vulnerable to backdoor attacks. However, backdoor defenses tailored to TSF remain underexplored, due to data entanglement and task-formulation shift challenges. To fill this gap, we conduct a systematic evaluation of thirteen representative backdoor defenses across the TSF life cycle and analyze their failure modes. Our results reveal two fundamental issues: (1) data entanglement induces *channel-level signal dilution*, rendering sample-filtering and trigger-synthesis defenses ineffective at localizing backdoors; and (2) task-formulation shift leads to *training-loss degeneration*, causing poisoned and clean windows to become indistinguishable at training stages. Based on these findings, we propose a training-time backdoor defense for TSF, termed TimeGuard. Our method adopts channel-wise pool training as the core paradigm and initializes a high-confidence pool using time-aware criteria to mitigate signal dilution. Moreover, we introduce distance-regularized loss selection to progressively expand the reliable pool during training and ease loss degeneration. Extensive experiments across multiple datasets, forecasting architectures, and TSF backdoor attacks demonstrate that TimeGuard substantially improves robustness, boosting $\mathrm{MAE}\_\mathrm{P}$ by 1.96× over the leading baseline, while preserving clean performance within 5% $\mathrm{MAE}\_\mathrm{C}$.

Applications · Computer Vision

YUANTONG CHEN, Zhengyan Ding, YanFeng Shang

While emerging training-free video anomaly detection (VAD) methods offer advantages such as interpretability and ease of deployment, they often suffer from computational inefficiency due to complex memory retrieval mechanisms or high-latency visual language models (VLMs). To address this, we propose PRISM (Parameter-free Recognition Based on Intrinsic Statistical Modeling), a novel framework for efficient open-set anomaly detection with minimal computational cost. PRISM based on a pre-trained multimodal embedding model, introduces differential amplification and whitening mechanisms to statistically suppress common-mode background noise in the embedding space, thereby significantly improving the signal-to-noise ratio of anomalous events. Extensive experiments on three mainstream datasets demonstrate that PRISM achieves state-of-the-art performance Real-time reasoning ability and interpretability. Furthermore, our statistical model provides a theoretical explanation for the performance gap (particularly mean accuracy (AP)) observed in existing training-free methods on complex datasets such as XD-Violence.