Graph Neural Networks (GNNs) have demonstrated remarkable performance across web-centric applications, yet they are vulnerable to sophisticated adversarial attacks, particularly node injection attacks. The success of such attacks heavily relies on their stealthiness, the ability to blend in with the original graph and evade detection. However, existing methods often achieve stealthiness by relying on indirect proxy metrics, lacking consideration for the fundamental characteristics of the injected content, or focusing only on imitating local structures, which leads to the problem of local myopia. To overcome these limitations, we propose a dual-constraint stealthy node injection framework, called Joint Alignment of Nodal and Universal Structures (JANUS). At the local level, we introduce a local feature manifold alignment strategy to achieve geometric consistency in the feature space. At the global level, we incorporate structured latent variables and maximize the mutual information with the generated structures, ensuring the injected structures are consistent with the semantic patterns of the original graph. We model the injection attack as a sequential decision process, which is optimized by a reinforcement learning agent. Experiments on multiple standard datasets demonstrate that the JANUS framework significantly outperforms existing methods in terms of both attack effectiveness and stealthiness. Our code and scripts are available at https://github.com/jiahaozhang121/JANUS-WWW2026.
论文检索
输入标题、作者或关键词，从 8,230 篇学术成果中精准定位
Privacy leakage on Web-based platform has become a critical challenge as users continually share personal information through online social networks, health tracking platforms, and other Web services. Information recipients and third-party can progressively aggregate shared content across the Web, enabling increasingly accurate profiling of individuals. However, existing studies typically treat each disclosure as independent, overlooking the cumulative privacy risks that arise in continual information sharing. In addition, the subjective cognition of both users and adversaries in Web environments, where users and adversaries can dynamically adapt based on observable actions, remains underexplored. To address these challenges, we propose a dynamic Stackelberg game model for continual information sharing scenarios, where user's sequential privacy decisions are optimized to balance privacy protection and data utility. The model explicitly captures the cognitive behaviors of both the user and the adversary, allowing their subjective perceptions to shape the Stackelberg equilibrium. Building on this formulation, we develop a reinforcement learning-based algorithm to derive approximately optimal strategies for mitigating privacy leakage in the context of continual information sharing. Experiments on real-world datasets demonstrate that our method significantly reduces cumulative privacy risks while preserving the utility of shared content. The proposed model further provides actionable insights for the design of privacy-enhancing technologies and web platform policies.
Reasoning models have gained significant attention due to their strong performance, particularly when enhanced with retrieval augmentation. However, these models often incur high computational costs, as both retrieval and reasoning tokens contribute substantially to the overall resource usage. In this work, we make the following contributions: (1) we propose a retrieval-augmented reasoning model that dynamically adjusts the length of the retrieved document list based on the query and retrieval results; (2) we develop a cost-aware advantage function for training of efficient retrieval-augmented reasoning models through reinforcement learning; and (3) we explore both memory- and latency-bound implementations of the proposed cost-aware framework for both proximal and group relative policy optimization algorithms. We evaluate our approach on seven public question answering datasets and demonstrate significant efficiency gains, without compromising effectiveness. In fact, we observed that the model latency decreases by ~16-20% across datasets, while its effectiveness increases by ~5% on average, in terms of exact match.
Reasoning over structured graphs remains a fundamental challenge for Large Language Models (LLMs), particularly when scaling to large graphs. Existing approaches typically follow the retrieval-augmented generation (RAG) paradigm: first retrieving subgraphs relevant to the query and then generating answers conditioned on the retrieved subgraphs. However, such two-phase pipelines often struggle to faithfully incorporate graph structure, since the generation process is ultimately constrained by the quality and completeness of the retrieved subgraph. Although many advanced retrievers have been proposed recently to mitigate this issue, they are usually tailored to the training graphs and generalize poorly to unseen graphs, which limits their practical applicability. In this work, we propose Reasoning by Exploration (RoE), a novel approach that unifies retrieval and generation by framing reasoning over graphs as a process of graph exploration. At each step, the LLM selects candidate nodes and edges to explore, gradually constructing reasoning paths and generating answers along the way. To enable effective exploration, RoE is trained in two stages: supervised fine-tuning (SFT) on gold reasoning paths, followed by reinforcement learning (RL) to enhance exploration effectiveness and generalization. Experiments on benchmark datasets demonstrate that RoE achieves substantial overall improvements over baselines, while also generalizing effectively to unseen graphs. The code can be found at https://github.com/haoyuhan1/RoE.
Recent advances in enabling Large Language Models (LLMs) to use external tools have significantly extended their functional capabilities beyond internal knowledge. However, most existing approaches rely on retrieval-based mechanisms to select suitable tools from massive repositories, which often struggle to align user queries with tool documentation and exhibit limited generalization. To address these issues, we propose ToolBox-RL. The code is available at https://github.com/S-cavy/ToolBox-RL., a novel reinforcement learning framework that unifies query rewriting, intent understanding, and large-scale tool retrieval into an end-to-end optimization process. ToolBox-RL introduces a query rewriting stage to better capture user intent and ensure semantic alignment with tool descriptions, while reinforcement learning encourages autonomous discovery of generalized tool-use strategies through combined cold-start and policy optimization training. Our experiments demonstrate that ToolBox-RL not only achieves the best tool call accuracy on both white-box and black-box tools but also exhibits strong generalization capabilities on out-of-domain dataset. Ablation studies show that ToolBox-RL can adapt to different retrieval methods and maintains good performance even when incorporating a large number of distractor tools.
Large reasoning models have demonstrated strong problem-solving abilities, yet real-world tasks often require external tools and long-horizon interactions. Existing agent frameworks typically follow predefined workflows, which limit autonomous and global task completion. In this paper, we introduce DeepAgent, an end-to-end deep reasoning agent that performs autonomous thinking, tool discovery, and action execution within a single, coherent reasoning process. To manage long-horizon interactions, we introduce an autonomous memory folding mechanism that compresses past interactions into structured episodic, working, and tool memories, reducing error accumulation while preserving critical information. To teach general-purpose tool use efficiently and stably, we develop an end-to-end reinforcement learning strategy, namely ToolPO, that leverages LLM-simulated APIs and applies tool-call advantage attribution to assign fine-grained credit to the tool invocation tokens. Extensive experiments on eight benchmarks, including general tool-use tasks (ToolBench, API-Bank, TMDB, Spotify, ToolHop) and downstream applications (ALFWorld, WebShop, GAIA, HLE), demonstrate that DeepAgent consistently outperforms baselines across both labeled-tool and open-set tool retrieval scenarios. The code and demo are available at https://github.com/RUC-NLPIR/DeepAgent.
Recently, Agentic Reinforcement Learning (Agentic RL) has made significant progress in incentivizing the multi-turn, long-horizon tool-use capabilities of web agents. While mainstream agentic RL algorithms autonomously explore high-uncertainty tool-call steps under the guidance of entropy, excessive reliance on such signals leads to the training collapse. In this paper, we delve into the challenges caused by entropy and propose the Agentic Entropy-Balanced Policy Optimization (AEPO), an agentic RL algorithm designed to balance entropy in both the rollout and policy update phases. AEPO comprises two core components: (1) a dynamic entropy-balanced rollout mechanism that adaptively allocates global and branch sampling budget through entropy pre-monitoring, while imposing a branch penalty on consecutive high-entropy tool-call steps to prevent over-branching issues; and (2) Entropy-Balanced Policy Optimization that inserts a stop-gradient operation into the high-entropy clipping term to preserve and properly rescale gradients on high-entropy tokens, while incorporating entropy-aware advantage estimation to prioritize learning on high-uncertainty tokens. Results across 14 challenging datasets show that AEPO consistently outperforms 7 mainstream RL algorithms. Further analysis reveals that AEPO improves rollout sampling diversity while maintaining stable policy entropy, facilitating scalable training of general web agents. The code is available at https://github.com/RUC-NLPIR/ARPO.
Query rewriting is a fundamental technique in information retrieval (IR). It typically employs the retrieval result as relevance feedback to refine the query and thereby addresses the vocabulary mismatch between user queries and relevant documents. Traditional pseudo-relevance feedback (PRF) and its vector-based extension (VPRF) improve retrieval performance by leveraging top-retrieved documents as relevance feedback. However, they are constructed based on two major hypotheses: the relevance assumption (top documents are relevant) and the model assumption (rewriting methods need to be designed specifically for particular model architectures). While recent large language models (LLMs)-based generative relevance feedback (GRF) enables model-free query reformulation, it either suffers from severe LLM hallucination or, again, relies on the relevance assumption to guarantee the effectiveness of rewriting quality. To overcome these limitations, we introduce an assumption-relaxed framework: Generalized Pseudo Relevance Feedback (GPRF), which performs model-free, natural language rewriting based on retrieved documents, not only eliminating the model assumption but also reducing dependence on the relevance assumption. Specifically, we design a utility-oriented training pipeline with reinforcement learning to ensure robustness against noisy feedback. Extensive experiments across multiple benchmarks and retrievers demonstrate that GPRF consistently outperforms strong baselines, establishing it as an effective and generalizable framework for query rewriting.
Recent studies show that claims incorporating both text and images spread more effectively than those with text alone, presenting significant challenges for multimodal fact-checking. The rapid development of Multi-modal Large Language Models (MLLMs) has greatly advanced research in this field, enabling stronger performance. However, existing MLLM-based fact-checking methods fail to fully exploit visual evidence, and their reliance on rigid fine-tuning templates limits context-aware explanations and leads to weak deep reasoning. To address these limitations, we propose FACTCOMPASS, a novel framework that combines reasoning-aware fine-tuning with large-scale rule-based reinforcement learning and incorporates a semantic- and knowledge-enhanced retrieval module to strengthen deep reasoning and improve evidence utilization. This framework enhances evidence retrieval by obtaining semantically relevant evidence images, enriching the contextual understanding of claim-related images, and refining textual evidence at the knowledge level. To further enhance reasoning, we introduce a self-refining reinforcement fine-tuning strategy: (1) distilling GPT-4o's reasoning from partially fact-checking data for cold-start Chain-of-Thought learning; (2) activating reasoning across broader datasets using prior knowledge and rejection sampling; (3) applying Group Relative Policy Optimization to explore diverse reasoning paths and optimize factual consistency. Extensive experiments have demonstrated the effectiveness of the proposed framework.
Visual data from the Web power image classifiers, which underpin web services, including recommendation and moderation. However, Web data contain spurious correlations and social biases, and neural networks tend to learn biases in data. This can reinforce unfairness in web services and the web data, leading to a vicious cycle. In image classification, networks learn bias attributes for a specific class when most images contain the same attribute for a given class. Hence, training a fair and debiased classifier from a biased dataset demands handling imbalance between a majority of images with bias attributes (bias-aligned samples) and a minority without (bias-conflict samples). In this work, we introduce BiasEdit, a modular framework that automatically detects bias attributes from the original dataset and edits them to construct a debiased dataset. Specifically, BiasEdit first detects unknown bias attributes via statistical dependence and mutual information analysis of visual–linguistic representations, and then explicitly edits those attributes using text-guided image editing to generate realistic bias-conflict samples. Unlike prior works that assume known bias attributes or rely on synthetic mixing, our method operates without manual annotations and leverages off-the-shelf vision–language and editing models. BiasEdit addresses a fundamental challenge in Web-sourced visual AI, mitigating dataset-induced bias and achieving state-of-the-art debiasing performance even when training data are fully biased.
Single-expert fake news detectors, such as Graph Neural Networks (GNNs) and Large Language Models (LLMs), increasingly struggle to counter the diversifying camouflage tactics of modern adversaries, which range from semantic (e.g., mimicking writing styles) to structural (e.g., manipulating propagation paths). To address this, existing methods attempt to build a hybrid model by sequentially incorporating GNNs and LLMs; however, such hybridization blurs the distinction between experts and prevents critical cross-validation. In addition, existing methods rely heavily on vast labeled data, which is costly to acquire, particularly for fake news samples. In this paper, we propose a Semi-supervised Mixture of Experts framework for Fake news detection, namely S2MOE-F. The core idea of S2MOE-F is to establish a robust defense against multifaceted camouflage by cross-validating the complementary judgments of two independent experts, GNN and LLM. On the one hand, S2MOE-F drives experts' judgments by using a One-Class Classification (OCC) objective, which constrains true news within a compact hypersphere and identifies samples outside this boundary as fake, reducing reliance on scarce fake news labels. On the other hand, S2MOE-F generates high-confidence pseudo-labels based on consensus or divergence between experts to exploit abundant unlabeled data. In addition, we propose a novel reinforcement learning (RL)-based routing policy that dynamically determines the dominant expert for input samples without explicit supervision. Finally, we design a disentangled masked Transformer to ensure experts' specialization by reducing inter-expert redundancy. Extensive experiments on real-world datasets sourced from Web platforms and social media demonstrate the superior performance of S2MOE-F.
Graph Retrieval-Augmented Generation (GraphRAG) has shown great effectiveness in enhancing the reasoning abilities of Large Language Models (LLMs) by leveraging graph structures for knowledge representation and modeling complex real-world relationships. However, existing GraphRAG methods still face significant bottlenecks when handling complex problems that require multi-hop reasoning, as their query and retrieval phases are largely based on pre-defined heuristics and do not fully utilize the reasoning potentials of LLMs. To address this problem, we propose GraphRAG-R1, an adaptive GraphRAG framework by training LLMs with process-constrained outcome-based reinforcement learning (RL) to enhance the multi-hop reasoning ability. Our method can decompose complex problems, autonomously invoke retrieval tools to acquire necessary information, and perform effective reasoning. Specifically, we utilize a modified version of Group Relative Policy Optimization (GRPO) that supports rollout-with-thinking capability to train the model. Next, we design two process-constrained reward functions. To handle the shallow retrieval problem, we design a Progressive Retrieval Attenuation (PRA) reward to encourage essential retrievals. Then, to handle the over-thinking problem, we design a Cost-Aware F1 (CAF) reward to balance the model performance with computational costs. We further design a phase-dependent training strategy, containing three training stages corresponding to cold start and these two rewards. These stages empower GraphRAG with format following, behavior shaping, and smartness optimization abilities, respectively. Lastly, our method adopts a hybrid graph-textual retrieval to improve the reasoning capacity. Extensive experimental results demonstrate that GraphRAG-R1 significantly boosts LLM capabilities in solving complex reasoning problems compared to state-of-the-art GraphRAG methods on both in-domain and out-of-domain datasets. Furthermore, our framework can be flexibly integrated with various existing retrieval methods, consistently delivering performance improvements.
Reinforcement learning-based recommendation systems (RLRS) are increasingly favored for their ability to leverage online interactive feedback, enabling adaptive and personalized decision-making. In this setting, user feedback serves as both a behavioral signal and an optimization target, making it essential for policy learning. However, collecting such feedback, e.g., clicks, ratings, and engagement traces, raises serious privacy concerns, posing critical challenges for value estimation, online adaptation, and privacy protection. In this paper, we propose FeedGuard, a critic-guided reinforcement learning framework with privacy-preserving feedback. FeedGuard enhances trajectory modeling via critic guidance, enables joint online fine-tuning with effective exploration–exploitation tradeoffs, and enforces end-to-end privacy protection across the feedback lifecycle via split federated learning and differential privacy. We further provide a formal analysis of its differential privacy guarantees. Extensive experiments on four public recommendation datasets and the VirtualTB platform show that FeedGuard performs well in both offline and online settings, while maintaining rigorous privacy guarantees with minimal degradation.
Online market platforms play an increasingly powerful role in the economy. An empirical phenomenon is that platforms, such as Amazon, Apple, and DoorDash, also enter their own marketplaces, imitating successful products developed by third-party sellers. We formulate a Stackelberg model, where the platform acts as the leader by committing to an entry policy: when will it enter and compete on a product? We study this model through a theoretical and computational framework. We begin with a single seller, and consider different kinds of policies for entry. We characterize the seller's optimal explore-exploit strategy via a Gittins-index policy, and give an algorithm to compute the platform's optimal entry policy. We then consider multiple sellers, to account for competition and information spillover. Here, the Gittins-index characterization fails, and we employ deep reinforcement learning to examine seller equilibrium behavior. Our findings highlight the incentives that drive platform entry and seller innovation, consistent with empirical evidence from markets such as Amazon and Google Play, with implications for regulatory efforts to preserve innovation and market diversity.
In online advertising, the inherent complexity and dynamic nature of advertising environments necessitate the use of auto-bidding services to assist advertisers in bid optimization. The complexity escalates in multi-channel scenarios, where effective allocation of budgets and constraints across channels with distinct behavioral patterns becomes critical for optimizing return on investment. Current approaches predominantly employ either optimization-based strategies or reinforcement learning (RL) techniques. However, optimization-based methods lack the flexibility to adapt to dynamic market conditions, while RL-based approaches struggle to capture essential historical dependencies and observational patterns within the constraints of Markov Decision Process (MDP) frameworks. To address these limitations, we propose AHBid, an Adaptable Hierarchical Bidding framework that integrates generative planning with real-time control. The framework employs a high-level generative planner utilizing diffusion models to dynamically allocate budgets and constraints through effective capture of historical context and temporal patterns. We introduce a constraint enforcement mechanism to ensure compliance with specified constraints, complemented by a trajectory refinement mechanism that enhances adaptability to environmental changes through historical data utilization. The system further incorporates a control-based bidding algorithm that synergistically combines historical knowledge with real-time information, significantly improving both adaptability and operational efficacy. Extensive experiments are conducted using both large-scale offline datasets and online A/B tests, demonstrating the effectiveness of AHBid by yielding a 13.57% increase in overall return compared to existing baselines.
The growing scale of ad auctions on online advertising platforms has intensified competition, making manual bidding impractical and necessitating auto-bidding to help advertisers achieve their economic goals. Current auto-bidding methods have evolved to use offline reinforcement learning or generative methods to optimize bidding strategies, but they can sometimes behave counterintuitively due to the black-box training manner and limited mode coverage of datasets, leading to challenges in understanding task status and generalization in dynamic ad environments. Large language models (LLMs) offer a promising solution by leveraging prior human knowledge and reasoning abilities to improve auto-bidding performance. However, directly applying LLMs to auto-bidding faces difficulties due to the need for precise actions in competitive auctions and the lack of specialized auto-bidding knowledge, which can lead to hallucinations and suboptimal decisions. To address these challenges, we propose a hierarchical Large auto-Bidding Model (LBM) to leverage the reasoning capabilities of LLMs for developing a superior auto-bidding strategy. This includes a high-level LBM-Think model for reasoning and a low-level LBM-Act model for action generation. Specifically, we propose a dual embedding mechanism to efficiently fuse two modalities, including language and numerical inputs, for language-guided training of the LBM-Act; then, we propose an offline reinforcement fine-tuning technique termed GQPO for mitigating the LLM-Think's hallucinations and enhancing decision-making performance without simulation or real-world rollout like previous multi-turn LLM-based methods. Experiments demonstrate the superiority of a generative backbone based on our LBM, especially in an efficient training manner and generalization ability.
Optimizing the advertiser's cumulative value of winning impressions under budget constraints poses a complex challenge in online advertising, under the paradigm of AI-Generated Bidding (AIGB). Advertisers often have personalized objectives but limited historical interaction data, resulting in few-shot scenarios where traditional reinforcement learning (RL) methods struggle to perform effectively. Large Language Models (LLMs) offer a promising alternative for AIGB by leveraging their in-context learning capabilities to generalize from limited data. However, they lack the numerical precision required for fine-grained optimization. To address this limitation, we introduce GRPO-Adaptive, an efficient LLM post-training strategy that enhances both reasoning and numerical precision by dynamically updating the reference policy during training. Built upon this foundation, we further propose DARA, a novel dual-phase framework that decomposes the decision-making process into two stages: a few-shot reasoner that generates initial plans via in-context prompting, and a fine-grained optimizer that refines these plans using feedback-driven reasoning. This separation allows DARA to combine LLMs' in-context learning strengths with precise adaptability required by AIGB tasks. Extensive experiments on both real-world and synthetic data environments demonstrate that our approach consistently outperforms existing baselines in terms of cumulative advertiser value under budget constraints.