论文检索

Social Aspects · Security

Zihan Zhou, Yang Zhou, Jianghai Yu, Lingjuan Lyu, Longwei Wang, KC Santosh, Ruoming Jin, Dejing Dou

Recent work shows that even safety aligned large language models (LLM) can be pushed into unsafe behavior by carefully crafted jailbreak prompts. Existing jailbreaking attack methods often rely on disfluent or incoherent prompts, which limit their success and make them easy to detect. We introduce SJA, a structured jailbreak attack built around two ideas. First, inspired by the logic of Spilsbury puzzle, SJA decomposes a harmful query into a sequence of harmless sub-questions and reconstructs the original answer by combining the sub-question responses. Second, by leveraging the theory of Hamiltonian dynamics on hyperbolic space, we propose a hyperbolic Hamiltonian dynamics-based sub-question generation framework that effectively captures the structural and temporal dependencies. We provide a theoretical analysis of how each sub-question evolves along the trajectory and show that the hyperbolic Hamiltonian system effectively captures the underlying semantic structure. Finally, we propose a hyperbolic narrative fusion mechanism built on fractional embedding and Möbius fusion. This mechanism integrates coherent narratives into sub-questions while preserving geometric consistency and improving stealth performance. We theoretically validate that the combination of the generated harmless sub-questions, guided by the stealthy narrative, can effectively preserve the contextual semantics of the original harmful question.

Theory · Reinforcement Learning and Planning

Yifan Jiang, Jiasheng Pan, Mengtian Li, Li Jin

We study decentralized multi-agent reinforcement learning (MARL) for networked service systems with affinity in the presence of Byzantine nodes. The way that a server processes a job depends on an affinity state that captures the correlation between the job and the server. Each node learns a local control policy via an actor-critic algorithm with linear function approximation over inherently unbounded space of traffic states, while exchanging parameter information with neighbors through a communication graph. A set of Byzantine agents can exploit the unbounded state space and the resulting stochastic variance to compromise the consensus mechanism, destabilizing both learning and queuing processes. To address this vulnerability, we propose a resilient consensus-based MARL algorithm with momentum-based smoothing, which mitigates adversarial parameter manipulation and guarantees traffic stability under mild assumptions. We prove that the cooperative agents’ policies converge almost surely to a bounded neighborhood of a stationary solution of the global objective. We demonstrate the effectiveness and generality of the proposed framework in several representative service systems, including semantic routing for large language model serving, distributed polling in cloud computing, and smart manufacturing logistics.

Deep Learning · Robustness

Ruize Zhang, Yu Li, Zhang Wan, Juan Cao, Jie Zhang, Sheng Tang

Recent test-time defenses for CLIP claim to preserve zero-shot clean accuracy while improving adversarial robustness. However, we find the reported robustness of six recent proposed state-of-the-art methods substantially overestimated: they fail under basic adaptive attacks. We further observe that these defenses share a common reliance on an indicative measurement that is assumed to capture the distributional difference between clean and adversarial samples and to determine whether the defense should preserve or alter the static model’s prediction. We argue that this assumption is the fundamental weakness, and we propose CLIP-MAD (Manipulating Assumed Difference), an adaptive attack strategy designed to break it. CLIP-MAD efficiently expands the adversarial distribution without costly full gradient calculations and can be flexibly combined with existing attack baselines to further boost attack strength. Experiments across 13 datasets demonstrate that CLIP-MAD produces strong adversarial samples that markedly reduce the robustness of diverse test-time defenses, revealing a false sense of security in CLIP’s zero-shot robustness.

Social Aspects · Security

Minwoo Jang, Hoyoung Kim, Jabin Koo, Jungseul Ok

The rise of model hubs has made it easier to access reusable model components, making model merging a practical tool for combining capabilities. Yet, this modularity also creates a *governance gap*: downstream users can recompose released weights into unauthorized mixtures that bypass safety alignment or licensing terms. Because existing defenses are largely post-hoc and architecture-specific, they provide inconsistent protection across diverse architectures and release formats in practice. To close this gap, we propose Trap$^{2}$, an architecture-agnostic protection framework that encodes protection into the update during fine-tuning, regardless of whether they are released as adapters or full models. Instead of relying on architecture-dependent approaches, Trap$^{2}$ uses weight re-scaling as a simple proxy for the merging process. It keeps released weights effective in standalone use, but degrades them under re-scaling that often arises in merging, undermining unauthorized merging.

Tianneng Shi, Robin Rheem, Dongwei Jiang, Francisco De La Riega, Mona Wang, Zhun Wang, Jingzhi Jiang, Alexander Cheung, Sean Tai, Jonah Cha 等

AI has the potential to transform cybersecurity by enabling systems that can autonomously detect, analyze, and remediate software vulnerabilities. However, existing cybersecurity evaluations of AI systems are limited in scale or scope, and fail to capture the end-to-end lifecycle of real-world software vulnerability discovery and remediation. To address this gap, we propose CyberCycle, a large-scale and realistic end-to-end cybersecurity benchmark that comprehensively evaluates AI agents' abilities across the full lifecycle of vulnerability discovery, PoC generation, and patch generation. CyberCycle is comprehensive and scalable, as we build an automated, agent-enhanced pipeline for transforming open-source vulnerability data into realistic evaluation environments. Currently, the benchmark consists of 615 real-world vulnerabilities across 120 different open-source projects.

Social Aspects · Security

Qiushi Wu, Yue Xiao, Dhilung Kirat, Kevin Eykholt, Jiyong Jang, Douglas Schales

Recurring Pattern Bugs (RPBs) are defined as bugs where a single root cause appears repeatedly across multiple code segments. These bugs remain a persistent security threat even after individual instances are patched. Various static analyzers exist for finding specific bug patterns but require significant engineering effort and fail to generalize well beyond their predefined template, preventing them from detecting RPBs. To tackle RPBs, we introduce BugStone, a hybrid framework combining LLVM-based program analysis with Large Language Models to automate RPB detection. BugStone leverages a single patched instance to synthesize abstract error patterns and retrieves semantically similar bugs throughout the codebase. To evaluate BugStone, we create a ground truth dataset by analyzing over 1.9K security bugs reports, on which BugStone achieves 92.2% precision and 79.1% pairwise accuracy. We further validated BugStone through a large-scale real-world deployment. In the Linux kernel, BugStone identified over 22K potential issues; a manual audit of 400 samples confirmed 246 valid bugs, including invalid pointer dereferences, resource leaks, type errors, performance issues, and others. To evaluate the generalizability of BugStone, we further applied it to the top 100 Python projects, discovering multiple critical command injection vulnerabilities.

Social Aspects · Privacy

Xiang Zheng, Yutao Wu, Hanxun Huang, Yige Li, Xingjun Ma, Bo Li, Yu-Gang Jiang, Cong Wang

Autonomous code agents built on large language models are reshaping software and AI development through tool use, long-horizon reasoning, and self-directed interaction. However, this autonomy introduces a previously unrecognized security risk: agentic interaction fundamentally expands the LLM attack surface, enabling systematic probing and recovery of hidden system prompts that guide model behavior. We identify system prompt extraction as an emergent vulnerability intrinsic to code agents and present \textbf{\justask{}}, a self-evolving framework that autonomously discovers effective extraction strategies through interaction alone. Unlike prior prompt-engineering or dataset-based attacks, \justask{} requires no handcrafted prompts, labeled supervision, or privileged access beyond standard user interaction. It formulates extraction as an online exploration problem, using Upper Confidence Bound--based strategy selection and a hierarchical skill space spanning atomic probes and high-level orchestration. These skills exploit imperfect system-instruction generalization and inherent tensions between helpfulness and safety. Evaluated on \textbf{41} black-box commercial models across multiple providers, \justask{} consistently achieves full or near-complete system prompt recovery, revealing recurring design- and architecture-level vulnerabilities. Our results expose system prompts as a critical yet largely unprotected attack surface in modern agent systems.

Social Aspects · Security

Wei Wang, Zihao Guan, Xing Zhou, Yan Ding, Yusong Tan, Jie Yu, Bao Li

Deploying large language models (LLMs) on untrusted hardware entails a risk of weight extraction, which can lead to unauthorized replication and misuse of the model. A practical approach is to leverage Trusted Execution Environments (TEEs) and protect model security by obfuscating model weights. However, existing obfuscation schemes struggle to simultaneously provide strong security guarantees and high performance: schemes with security guarantees incur substantial overhead due to frequent TEE interactions, whereas schemes that achieve efficient inference are insecure. We propose SLIM, a secure inference framework that exploits the iterative structure of LLMs to let transformed representations cascade through consecutive obfuscated layers, thereby minimizing interactions with the TEE. SLIM introduces a T-Way Mixing algorithm that performs consecutive inter-vector covering using carefully constructed block-diagonal Householder matrices and combines it with successive random permutations, providing thorough weight obfuscation while keeping TEE-side computation lightweight. Evaluations demonstrate that SLIM provides robust security guarantees and significantly outperforms prior state-of-the-art obfuscation schemes in terms of performance, delivering up to a $13.80\times$ speedup while preserving fidelity.

Deep Learning · Large Language Models

Xinpeng Wang, William Cao, Andrew Wilson, Zhe Zeng

Recent studies on hallucination detection have shown that hallucination-related signals are more strongly encoded in intermediate layers than in the final layer of large language models (LLMs). While a growing body of work has sought to exploit this property for hallucination detection, the problem of how to automate the selection of high-performing layers is underexplored, and the development of principled methods for this purpose remains an open challenge. To address this gap, we first propose several hypotheses for why such signals emerge in intermediate layers and test corresponding criteria for automatic layer selection. We evaluate these criteria across two LLM architectures and five datasets, and find that none of them deliver satisfying performance. Instead, we propose a new selection criterion, First Effective Peak of Intrinsic Dimension (FEPoID), that is able to consistently identify optimal or near-optimal layers and outperforms the aforementioned criteria and existing hallucination detection baselines. This criterion is training-free and requires negligible computational overhead. Additionally, we study the generation behaviors of LLMs and introduce a simple yet effective truncation strategy, which further amplifies the hallucination-related signals and leads to substantial improvements in overall detection performance.

Social Aspects · Security

Mitchell Hermon, Rahul Gupta, Weitong Ruan, Ekraam Sabir, Haohan Wang

We identify a fundamental tension in securing LLMs: the \textbf{security--fidelity tradeoff}. While defenses against indirect prompt injection are becoming more robust, we show that they inevitably impair the model's ability to process benign, instruction-like text. Current evaluations miss this cost because they conflate utility with fidelity. We address this gap with \textsc{SecFid}, a benchmark that uses behaviorally separable probes to unambiguously distinguish between resisting an attack, succumbing to it, and faithfully processing it as data. Our evaluation reveals this tradeoff across a diverse set of models and highlights how the strongest defenses achieve security often by aggressively suppressing valid content, causing fidelity failure rates up to 50\% on translation. We ground these results in a decision-theoretic framework, proving that when benign and adversarial inputs overlap, no universal defense exists. Therefore, optimal robustness is strictly task-dependent, determined by an application’s tolerance for fidelity errors versus security failures.

Applications · Language, Speech and Dialog

Yifan Yang, Hui Wang, Bing Han, Shujie Liu, Jinyu Li, Yong Qin, Xie Chen

Recent advances in text-to-speech (TTS) technology have enabled systems to generate speech that is often indistinguishable from human speech, bringing benefits to accessibility, content creation, and human-computer interaction. However, current evaluation practices are increasingly inadequate for capturing the full range of capabilities, limitations, and societal impacts of modern TTS systems. This position paper introduces the concept of Responsible Evaluation and argues that it is essential and urgent for the next phase of TTS development, structured through three progressive levels: (1) ensuring the faithful and accurate reflection of a model's true capabilities and limitations, with more robust, discriminative, and comprehensive objective and subjective scoring methodologies; (2) enabling comparability, standardization, and transferability through standardized benchmarks, transparent reporting, and transferable evaluation metrics; and (3) assessing and mitigating ethical risks associated with forgery, misuse, privacy violations, and security vulnerabilities. Through this concept, we critically examine current evaluation practices, identify systemic shortcomings, and propose actionable recommendations. We hope this concept will not only foster more reliable and trustworthy TTS technology but also guide its development toward ethically sound and societally beneficial applications.

Social Aspects · Security

Shayne Longpre, Elaine Zhu, Carson Ezell, Avijit Ghosh, Sean McGregor, Kevin Paeth, Kevin Klyman, Sayash Kapoor, Rishi Bommasani, Ruth Elisabeth Appel 等

Flaw reporting for deployed AI systems is fundamental to identifying system failures and improving AI safety. Yet the AI reporting ecosystem is fragmented: researchers who identify flaws often do not know what or where to report, and groups who receive reports rarely share them with other relevant stakeholders. As a result, good-faith reporters duplicate effort by submitting many different forms, and recipients lack standardized, triage-ready information. We audit 12 reporting systems published by AI developers, cybersecurity groups, and AI flaw aggregators, identifying five recurring design challenges spanning discoverability, scope, information collection, coordination, and guidance for strict-liability cases. Building on this analysis and feedback from 49 experts across 32 organizations representing developers, security researchers, and ecosystem coordinators, we introduce FLARE-AI, an open-source AI flaw reporting system designed for interoperability with existing systems. FLARE-AI streamlines flaw report creation by collecting triage-relevant information through conditional logic and early classification, then enables optional dissemination of standardized, machine-readable reports to multiple developers, coordinators, and incident registries from a single submission. By lowering barriers to reporting AI flaws and improving interoperability across stakeholders, FLARE-AI helps break down silos and accelerate remediation across the AI ecosystem.

Social Aspects · Security

Changyue Jiang, Wenqi Zhang, Xudong Pan, Geng Hong, Min Yang

LLM-based agents solve complex tasks through iterative reasoning, tool use, and environment interaction, where each intermediate thought directly shapes subsequent actions. Small deviations in these thoughts can therefore propagate into unsafe behaviors, yet existing guardrails typically operate only on final outputs or require intrusive model modifications. We introduce Thought-Aligner, a lightweight plug-in safety model that performs causal correction on unsafe thoughts before action execution, without altering the underlying agent. The corrected thoughts are fed back into the agent, steering its decision process and tool use toward safer trajectories. Because it operates solely at the thought level, Thought-Aligner is model-agnostic and can be integrated into diverse agent frameworks. We train Thought-Aligner via two-stage contrastive learning on paired safe and unsafe thoughts generated across ten risk scenarios. Experiments on two agent-safety benchmarks with six LLMs show that Thought-Aligner increases behavioral safety from about 50% without protection to around 90% on average, exceeding state-of-the-art guardrails by roughly 23%, while also improving helpfulness by about 5%. The method incurs low per-step latency and minimal overhead, enabling scalable and practical deployment.

Social Aspects · Security

Zhuangzhuang Zhang, MingXin Li, Libing Wu, Wei-Bin Lee, Jianping Wang

Collaborative perception (CP) significantly extends the sensing range of connected and autonomous vehicles (CAVs). However, its reliance on data fusion among multiple CAVs makes it inherently vulnerable to adversarial attacks from malicious participants. Existing defenses primarily rely on output-level consensus, assuming that malicious messages manifest as statistical outliers, while suffering from poor adaptability to environmental noise. This makes them vulnerable to stealthy adversarial attacks and prone to high false positive rates. To address this challenge, we shift the defense paradigm from superficial output-level consensus to deeper consistency within the internal feature space. Guided by this principle, we propose \texttt{Cerberus}, a novel defense framework against adversarial attacks in CP systems by leveraging multi-dimensional consistency in the feature space. By quantifying conflicts in topological structure, semantic direction, and energy distribution within feature maps, \texttt{Cerberus} effectively detects adversarial perturbations and provides dynamic protection against adversarial attacks. Experimental results demonstrate that \texttt{Cerberus} significantly outperforms state-of-the-art methods, effectively limiting the attack success rate to as low as 0.05\% while restoring the mAP to 0.88.

Social Aspects · Safety

Anselm Paulus, Ilia Kulikov, Brandon Amos, REMI MUNOS, Ivan Evtimov, Kamalika Chaudhuri, Arman Zharmagambetov

Ensuring the safety of language models (LMs) while maintaining their usefulness remains a critical challenge in AI alignment. Current approaches rely on sequential adversarial training: generating adversarial prompts and fine-tuning LMs to defend against them. We introduce a different paradigm: framing safety alignment as a non-zero-sum game between an Attacker LM and a Defender LM trained jointly via online reinforcement learning. Each LM continuously adapts to the other's evolving strategies, driving iterative improvement. Our method uses a preference-based reward signal derived from pairwise comparisons instead of point-wise scores, providing more robust supervision and potentially reducing reward hacking. Our RL recipe, AdvGame, shifts the Pareto frontier of safety and utility, yielding a Defender LM that is simultaneously more helpful and more resilient to adversarial attacks. In addition, the resulting Attacker LM converges into a strong, general-purpose red-teaming agent that can be directly deployed to probe arbitrary target models.

Social Aspects · Privacy

Ossi Räisä, Antti Koskela, Antti Honkela

The accuracy-first perspective of differential privacy addresses an important shortcoming by allowing a data analyst to adaptively adjust the quantitative privacy bound instead of sticking to a predetermined bound. Existing works on the accuracy-first perspective have neglected an important property of differential privacy known as post-processing immunity, which ensures that an adversary is not able to weaken the privacy guarantee by post-processing. We address this gap by determining which existing definitions in the accuracy-first perspective have post-processing immunity, and which do not. The only definition with post-processing immunity, pure ex-post privacy, lacks useful tools for practical problems, such as an ex-post analogue of the Gaussian mechanism, and an algorithm to check if accuracy on separate private validation set is high enough. To address this, we propose a new definition based on Rényi differential privacy that has post-processing immunity, and we develop basic theory and tools needed for practical applications. We demonstrate the practicality of our theory with applications to synthetic data generation and image classifier fine-tuning, where our algorithm successfully adjusts the privacy bound until an accuracy threshold is met on a private validation dataset.

Social Aspects · Security

Nan Yan, Qian Lou, Jiarong Xing

Long-term memory empowers LLM-based agents with adaptive reasoning but exposes a critical attack surface---adversaries can inject malicious records to bias agent behaviors. However, existing attacks face a dilemma: effective injections are often visibly malicious and easily detected, while stealthy, benign-looking injections are often less effective in altering agent behaviors. To address this, we propose MemIncept, a memory poisoning attack that can impact agents even in black-box settings using only benign-appearing queries. Unlike prior methods that inject isolated records, MemIncept generates a cooperative set of queries that work together to bias the agent. It achieves this via a bidirectional evolutionary strategy that optimizes the query set from two ends. A forward pass ensures the queries collectively lead the agent to the target outcome, while a backward pass ensures they are semantically close to victim (benign) queries for reliable retrieval. This ``meet-in-the-middle'' approach creates injected records that are both easy to retrieve and effective at steering behavior. Through extensive experiments across diverse agents, we show that MemIncept significantly outperforms single-record attacks, achieving high success rates comparable to explicit attacks while remaining virtually undetectable to both humans and automated filters.

Deep Learning · Large Language Models

Songwen Zhao, Danqing Wang, Kexun Zhang, Jiaxuan Luo, Zhuo Li, Lei Li

Vibe coding is a new programming paradigm in which human engineers instruct large language model (LLM) agents to complete complex coding tasks with little supervision. Although it is increasingly adopted, are vibe coding outputs really safe to deploy in production? To answer this question, we propose SUSVIBES, a benchmark consisting of 200 feature-request software engineering tasks from real-world open-source projects, which, when given to human programmers, led to vulnerable implementations. We evaluate multiple widely used coding agents with frontier models on this benchmark. Disturbingly, all agents perform poorly in terms of software security. Although 61% of the solutions from SWE-Agent with Claude 4 Sonnet are functionally correct, only 10.5% are secure. Further experiments demonstrate that preliminary security strategies, such as augmenting the feature request with vulnerability hints, cannot mitigate these security issues. Our findings raise serious concerns about the widespread adoption of vibe-coding, particularly in security-sensitive applications.

Social Aspects · Security

Zhao-Rong Lai, Xiwen Yuan, Jian Weng

Denoising diffusion sampling (DDS) is an emerging approach for generating new samples that have the same distribution as some training samples. However, it is vulnerable to adversarial attacks by even a Gaussian perturbation. In this work, we propose a complete set of adversarial attack and defense methodology for DDS. In the attack side, we propose to inject a perturbation to the sampling stage, which significantly worsen the performance of sample generation. In the defense side, we propose a local variation based regularization model for the potential function minimization, which effectively tolerates the adversarial perturbations. Moreover, we develop a conjugate gradient algorithm to solve the defense model, which integrates with a recently-developed zeroth order rejection sampling method that saves computational cost. Experimental results show that the proposed attack significantly worsen the existing state-of-the-art methods, but can be defended by the proposed local variation regularization.

General Machine Learning · Unsupervised and Semi-supervised Learning

Kunling Li, Cunqing Hua, Hongyu Zhu, Tianjie Ju, Pengwenlong Gu

Radio Frequency Fingerprint Identification (RFFI) is a foundational pillar of physical-layer security, providing unclonable identity authentication and lightweight defense mechanisms for zero-trust wireless networks. Its practical deployment, however, often occurs in a source-free open-world (SF-OW) setting, characterized by a continuous influx of unregistered devices and privacy constraints that preclude the retention of historical data. In this paper, we formalize SF-OW RFFI task, which manifests a severe stability-plasticity dilemma: intrinsic signal similarity confuses new classes, while source absence precipitates catastrophic forgetting. To address this, we propose Incremental Orthogonal ETF (IO-ETF), a novel neural collapse-inspired framework utilizing output geometry to actively induce parameter separation and isolation. We further devise a Triple-Level Geometric Alignment (TLGA) strategy via semantic optimal transport, manifold progressive anchoring, and reliable subspace retention to stably align unlabeled streams to this geometric skeleton. Experiments on benchmarks demonstrate a superior trade-off between old-class retention and new-class discovery, offering a promising solution for secure access in dynamic networks.