Recommender Systems (RS) are fundamental to modern online services. While most existing approaches optimize for short-term engagement, recent work has begun to explore reinforcement learning (RL) to model long-term user value. However, these efforts face significant challenges due to the vast, dynamic action spaces inherent in RS, which hinder stable policy learning. To resolve this bottleneck, we introduce Hierarchical Semantic RL (HSRL), which reframes RL-based recommendation over a fixed Semantic Action Space (SAS). HSRL encodes items as Semantic IDs (SIDs) for policy learning, and maps SIDs back to their original items via a fixed lookup during execution. To align decision-making with SID generation, the Hierarchical Policy Network (HPN) operates in a coarse-to-fine manner, employing hierarchical residual state modeling to refine each level's context from the previous level's residual, thereby reducing representation–decision mismatch. In parallel, a Multi-level Critic (MLC) provides token-level value estimates, enabling fine-grained credit assignment. Across public benchmarks and a large-scale production dataset from a leading short-video advertising platform, HSRL consistently surpasses state-of-the-art baselines. In online deployment over a 7-day A/B testing, it delivers an 18.421% ADVV lift and a 1.251% increase in Revenue, supporting HSRL as a scalable paradigm for RL-based recommendation.
论文检索
输入标题、作者或关键词，从 8,230 篇学术成果中精准定位
Retrieval-Augmented Generation (RAG)-enhanced LLM systems, while powerful, introduce substantial inference costs due to the inclusion of an extra multi-stage pipeline that dynamically retrieves and synthesizes information from external knowledge sources. This high operational cost exposes a critical vulnerability to Inference Cost Attacks (ICAs). However, existing ICAs often rely on the impractical assumption of direct prompt manipulation. We argue that a more feasible and potent threat to RAG-enhanced LLM systems arises from poisoning external knowledge bases (e.g., web knowledge from the Internet). In this work, we introduce the Retrieval-Augmented Inference Cost Attack (RA-ICA), a novel attacking paradigm that targets the computational cost of RAG-enhanced LLM systems by injecting malicious documents into external knowledge corpus. To operationalize this attack, we propose Computational Resource Exhaustion via External Poisoning (CREEP), a novel framework that leverages LLM agents to automatically craft malicious documents that are both semantically relevant for retrieval and potent for inducing an abnormal increase in token consumption during the inference phase. To enhance the attack's effectiveness, we introduce Memory-Augmented Group Relative Policy Optimization (MA-GRPO), a novel reinforcement learning algorithm that fine-tunes the agents by learning from a dynamic memory of historical best adversarial documents. Extensive experiments across three real-world datasets demonstrate that RA-ICA increases token consumption by up to 13.12 times with an over 90% success rate, without degrading the integrity of the generated answer.
While Large Language Models (LLMs) excel at many reasoning tasks, their native inability to produce calibrated, multi-class probability distributions limits their use in high-stakes Web applications like content moderation and fraud detection. Existing methods to elicit probabilities from LLMs either sacrifice their crucial Chain-of-Thought (CoT) reasoning capabilities or suffer from poor calibration. To address this, we introduce a new paradigm, Verbalized Probability Distribution, and a novel training framework, RLVP (Reinforcement Learning with Verbalized Probabilities). RLVP fine-tunes an LLM to generate both an interpretable CoT and a complete, verbalized probability distribution. We overcome the ''insufficient reward granularity'' problem in standard Reinforcement Learning (RL) for classification by using soft probabilities from expert tabular models as a dense reward curriculum. Through large-scale joint training on 169 tabular tasks, we demonstrate that a single RLVP-trained model can surpass a strong, task-specific XGBoost baseline on up to 55% of tasks. More importantly, the trained model achieves state-of-the-art few-shot performance on unseen, heterogeneous Web benchmarks that mix structured data with free text, achieving performance comparable to or superior than expert models trained on the same limited data. This showcases a strong capability for generalization and knowledge transfer to complex Web data. Our work presents a viable path toward building general-purpose, probabilistically-sound, and interpretable foundation models for the Web.
Table reasoning with large language models (LLMs) plays a critical role in building intelligent systems capable of understanding and analyzing tabular data. Despite recent progress, existing methods still face key limitations: their reasoning processes lacks depth and explicit multi-step reasoning, often relying solely on implicit language model understanding. In addition, their reasoning processes suffer from instability, primarily caused by model uncertainty. In this work, we propose STaR, a novel slow-thinking model that can achieve effective and stable table reasoning. To enable effective multi-step reasoning, we design a two-stage training framework consisting of supervised fine-tuning (SFT) warm-up followed by reinforced fine-tuning (RFT). Specifically, in the SFT stage, we construct a high-quality dataset through automatic self-verification. In the RFT stage, we introduce a difficulty-aware reinforcement learning mechanism to further enhance reasoning capabilities. Furthermore, to improve reasoning stability, we introduce trajectory-level uncertainty quantification, which fuses token-level confidence with answer-level consistency, enabling the selection of better reasoning trajectories. Extensive experiments demonstrate that STaR-8B achieves state-of-the-art performance on in-domain benchmarks and exhibits strong generalization to out-of-domain datasets, highlighting its potential for enhancing both effectiveness and stability in table reasoning.
The rapid spread of online misinformation poses a serious threat to public trust and social stability, making automatic fake news detection increasingly critical. In this work, we propose a novel retrieval-augmented fake news detection framework, Triple-R (Rewriting, Retrieval, and iterative Refinement), which emphasizes optimizing the retrieval query. Unlike prior retrieval-augmented methods that adapt either the retriever or the verification module, often overlooking the gap between news text and the evidence needed for verification, our approach focuses on adapting the search query to retrieve the most relevant evidence. Specifically, we employ a small language model as a trainable query rewriter, optimized via reinforcement learning with feedback from a frozen LLM-based fake news detector, to transform the original news text into effective retrieval queries. To further enhance evidence relevance, we introduce an iterative query refinement mechanism, which progressively updates rewritten queries based on previously retrieved results. Finally, the original news text and the evidence retrieved through refined queries are integrated for verification. Experiments on two real-world datasets demonstrate consistent improvements, validating the effectiveness of our approach.
Search-based recommendation is one of the most critical application scenarios in e-commerce platforms. Users' complex search contexts—such as spatiotemporal factors, historical interactions, and current query's information—constitute an essential part of their decision-making, reflecting implicit preferences that complement explicit query terms. Modeling such rich contextual signals and their intricate associations with candidate items remains a key challenge. Although numerous efforts have been devoted to building more effective search methods, existing approaches still show limitations in integrating contextual information, which hinders their ability to fully capture user intent. To address these challenges, we propose a context-aware reasoning-enhanced generative search framework for better understanding the complicated context. Specifically, the framework first unifies heterogeneous user and item contexts into textual representations or text-based semantic identifiers and aligns them. To overcome the lack of explicit reasoning trajectories, we introduce a self-evolving post-training paradigm that iteratively combines supervised fine-tuning and reinforcement learning to progressively enhance the model's reasoning capability. In addition, we identify potential biases in existing RL algorithms when applied to search scenarios and present a debiased variant of GRPO to improve ranking performance. Extensive experiments on search log data collected from a real-world e-commerce platform demonstrate that our approach achieves superior performance compared with strong baselines, validating its effectiveness for search-based recommendation.
Multi-criteria recommender systems (MCRSs) are becoming increasingly important in the Web ecosystem, where platforms such as e-commerce sites and review portals allow users to evaluate items from multiple perspectives. By leveraging criterion-level ratings rather than relying solely on overall scores, MCRSs can enhance personalization and more accurately capture user preferences. However, existing multi-criteria recommendation methods often fail to explicitly model user-specific preferences across criteria or to incorporate item–criterion signals into representation learning. To solve these problems, we propose TaTriGR, a Targeting-Aware Tripartite Graph Recommender, which jointly models user–item–criterion interactions in a unified tripartite graph and integrates user-specific criterion weights through targeting mechanisms. Specifically, TaTriGR encodes user–criterion targeting through personalized weights, while incorporating item–criterion performance as an additional channel to enrich item semantics. A lightweight propagation mechanism then diffuses information across the tripartite structure, and a formal score decomposition shows that predictions satisfy fundamental multi-criteria decision making (MCDM) properties. To further reinforce targeting, TaTriGR introduces two auxiliary objectives: Ideal Point Distillation and Lexicographic Consistency, which encourage criterion-consistent user representations and rankings. Extensive experiments on three real-world datasets demonstrate the effectiveness of TaTriGR, showing relative gains up to 16.1% over the best baseline models. Our implementations are available at https://github.com/nunu1995/TaTriGR.
From Prediction to Understanding: Leveraging Reasoning in Large Language Model-based Recommendations
Recently, large language models (LLMs) have shown great promise in sequential recommendation. Existing methods typically transform users' historical interactions into textual sequences and then feed these sequences into LLMs to generate recommended items. However, since the output of LLMs is limited to recommendations, the models must analyze user interactions and infer user preferences implicitly. This implicit approach constrains the expressive reasoning abilities of LLMs. Moreover, as the output of LLMs consists solely of recommended items, the resulting recommendations lack explainability. To address these issues, we propose RE2, which enables LLMs to explicitly generate reasoning content before providing recommendations. Making the reasoning process explicit helps elicit the reasoning abilities of LLMs and simultaneously enhances the explainability of recommendation results. RE2 consists of three steps: (1) Reasoning Collection, which collects reasoning data by guiding LLMs to generate both reasoning content and recommended items for each interaction sequence. (2) Pattern Imitation, which leverages the collected data to train LLMs via supervised fine-tuning to imitate the pattern of first generating reasoning content and then providing recommendations. (3) Pattern Internalization, which further internalizes this reasoning-and-recommendation pattern and enhances both recommendation performance and the rationality of the reasoning content through reinforcement learning. RE2 can be implemented under both the self-distillation and teacher-distillation frameworks, without requiring external user metadata such as user reviews. Experimental results demonstrate the effectiveness of RE2 in improving recommendation performance and in generating high-quality reasoning content. Furthermore, we show that RE2 can mitigate popularity bias while maintaining recommendation accuracy to some extent. Our data and code are available at https://github.com/zhiyuanc2001/RE2.
Personalized Learning Path Planning (PLPP) aims to design adaptive learning paths that align with individual goals. While large language models (LLMs) show potential in personalizing learning experiences, existing approaches often lack mechanisms for goal-aligned planning. We introduce Pxplore, a novel framework for PLPP that integrates a reinforcement-based training paradigm and an LLM-driven educational architecture. We design a structured learner state model and an automated reward function that transforms abstract objectives into computable signals. We train the policy combining supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO), and deploy it within a real-world learning platform. Extensive experiments validate Pxplore's effectiveness in producing coherent, personalized, and goal-driven learning paths. We release our code and dataset at https://github.com/Pxplore/pxplore-algo.
Large language models have given rise to interactive recommendation agents (IRAs). Through proactive clarification, tool invocation, and dynamic dialogue, IRAs shift recommender systems from passive prediction to interactive, proactive intelligence. For training IRAs, agentic reinforcement learning offers a natural pathway, as it enables models to learn interactive capabilities directly from environmental feedback without requiring costly annotated data. However, this process faces three key challenges: credit assignment in multi-turn interactions, efficient exploration in large action spaces, and coordinated learning of multiple interactive skills. To tackle this challenge, we present a new preference optimization paradigm, GIMO, which treats the interaction and learning process of IRAs in sparse environments as the continuous fulfillment and stimulation of three intrinsic drives: Autonomy, Competence, and Relatedness. Instead of relying solely on external rewards, GIMO estimates the potential between adjacent states in a generative manner to construct intrinsic rewards. Such a generative motivation structure not only enables fine-grained credit assignment, but also naturally inspires a hint-guided action proposal mechanism that facilitates efficient exploration. Furthermore, during the multi-skill coordination training stage, we introduce an explicit KL regularization into policy orchestration to prevent global policy collapse. Theoretically, we prove that GIMO guarantees global policy consistency, and empirically, experiments in interactive recommendation environments built from three datasets demonstrate that GIMO consistently outperforms existing methods. Our code is anonymously available at https://github.com/XueyangFeng/GIMO.
Large Language Models (LLMs) with powerful reasoning capabilities, offer new opportunities for recommendation systems (RS) , especially in understanding user preference. However, the direct application of LLMs to this task faces two major challenges: (1) limited context windows struggle to process the extensive user behavior sequences in real-world scenarios; (2) inherent hallucination effect can lead LLMs to infer spurious preferences that contradict true user intent, thereby degrading recommendation quality. To address this, we propose Rec 2, a Reinforcement learning-constrained segmented user modeling framework for recommendation. The framework segments lengthy behavior sequences into multiple segments and generates user preferences in a cascaded fashion. To suppress hallucinations and align with user intent, we introduce reinforcement learning, which uses the subsequent behavior segment as a supervisory signal to constrain and reward the preference inference process on the current segment, thereby generating more predictive and high-fidelity user preferences. Finally, the aligned LLMs infer preferences for each behavior segment; these are then processed by a specially-designed dynamic preference learning module to model preference evolution and are ultimately aggregated into a unified, dynamic long-term user preference embedding. This representation can be integrated into any recommendation model to boost its performance. Extensive experiments demonstrate that Rec2 significantly enhances recommendation performance by effectively capturing dynamic and authentic user preferences.
As critical infrastructure for Web 3.0, DAG-based blockchains promise high throughput for DeFi, IoT, and DApps. However, realizing this potential is challenging, as system performance is dictated by a multitude of interdependent parameters across network, node, and consensus layers. Manual configuration fails to adapt to dynamic workloads, leading to suboptimal performance. We introduce Alzo, a novel auto-tuner that employs hierarchical reinforcement learning (HRL) to navigate this complex configuration space. By decomposing the DAG blockchain's workflow into distinct stages, Alzo's HRL policy learns from stage-level performance metrics to control critical parameters governing consensus, execution, and graph topology in real-time. Furthermore, we employ a shadow-control loop to ensure the safety of all parameter adjustments. Our experiments show that Alzo significantly outperforms other configurations, achieving higher throughput and lower latency under variable workloads with minimal overhead.
The rise of Internet of Vehicles (IoV) technologies is transforming traffic management from isolated control to a collective, multi-vehicle process. At the heart of this shift is multi-vehicle dynamic navigation, which requires simultaneously routing large fleets under evolving traffic conditions. Existing path search algorithms and reinforcement learning methods struggle to scale to city-wide networks, often failing to capture the nonlinear, stochastic, and coupled dynamics of urban traffic. To address these challenges, we propose CityNav, a hierarchical, LLM-powered framework for large-scale multi-vehicle navigation. CityNav integrates a global traffic allocation agent, which coordinates strategic traffic flow distribution across regions, with local navigation agents that generate locally adaptive routes aligned with global directives. To enable effective cooperation, we introduce a cooperative reasoning optimization mechanism, in which agents are jointly trained with a dual-reward structure: individual rewards promote per-vehicle efficiency, while shared rewards encourage network-wide coordination and congestion reduction. Extensive experiments on four real-world road networks of varying scales (up to 1.6 million roads and 430,000 intersections) and traffic datasets demonstrate that CityNav consistently outperforms nine classical path search and RL-based baselines in city-scale travel efficiency and congestion mitigation. Our results highlight the potential of LLMs to enable scalable, adaptive, and cooperative city-wide traffic navigation, providing a foundation for intelligent, large-scale vehicle routing in complex urban environments. Our project is available at https://github.com/usail-hkust/CityNav.
Adaptive traffic signal control (ATSC) is essential for mitigating urban congestion in modern smart cities, where traffic infrastructure is evolving into interconnected Web-of-Things (WoT) environments with thousands of sensing-and-control nodes. However, existing methods face a critical scalability-coordination tradeoff: centralized approaches optimize global objectives but become computationally intractable at city scale, while decentralized multi-agent methods scale efficiently yet lack network-level coherence, resulting in suboptimal performance. In this paper, we present HALO, a hierarchical reinforcement learning framework that addresses this tradeoff for large-scale ATSC. HALO decouples decision-making into two levels: a high-level global guidance policy employs Transformer-LSTM encoders to model spatio-temporal dependencies across the entire network and broadcast compact guidance signals, while low-level local intersection policies execute decentralized control conditioned on both local observations and global context. To ensure better alignment of global-local objectives, we introduce an adversarial goal-setting mechanism where the global policy proposes challenging-yet-feasible network-level targets that local policies are trained to surpass, fostering robust coordination. We evaluate HALO extensively on multiple standard benchmarks, and a newly constructed large-scale Manhattan-like network with 2,668 intersections under real-world traffic patterns, including peak transitions, adverse weather and holiday surges. Results demonstrate HALO shows competitive performance and becomes increasingly dominant as network complexity grows across small-scale benchmarks, while delivering the strongest performance in all large-scale regimes, offering up to 6.8% lower average travel time and 5.0% lower average delay than the best state-of-the-art.
Influence maximization (IM) aims to select a small set of seed nodes whose activation triggers a maximal cascade. Existing methods typically assume access to the network topology or a reliable surrogate, which is often unavailable in practice due to noisy, privacy-protected, or partially observed links. These settings pose two challenges: (1) the lack of explicit topology removes a key inductive constraint for diffusion modeling, undermining influence estimation, and (2) nonlinear, temporally dependent node interactions yield complex multivariate time series that hinder topology inference. We propose DynaFLUX, an end-to-end generative framework for IM under hidden topology. DynaFLUX learns a compact surrogate of latent dynamics directly from observed time series, and jointly optimizes a seed-selection policy via reinforcement learning. A self-attention pointer network captures long-range dependencies for seed generation, while an influence-prediction module infers a surrogate topology and uses Monte Carlo diffusion to provide policy-gradient rewards. Experiments show that DynaFLUX accurately identifies influential spreaders and consistently outperforms state-of-the-art baselines in unseen topology scenarios.
Social media's openness fosters opinion exchange but complicates guiding users toward global consensus. Existing approaches are often invasive, modifying opinions or forcing cross-group interactions, undermining autonomy and triggering resistance. They also lack long-term planning, risking macro-level polarization despite local agreement. To address this, we propose H-NeiFi, a hierarchical, non-invasive opinion guidance framework. It models experts and non-experts separately and introduces a neighbor filtering mechanism that adaptively shapes communication without altering user opinions. Using multi-agent reinforcement learning with a long-term reward, H-NeiFi optimizes information flow while preserving interaction autonomy. Experiments show that it accelerates consensus by 22.0% to 30.7% and achieves global convergence even without experts, providing a natural and efficient paradigm for social network governance. Our code is available at: https://github.com/shijunguo44/H-NeiFi.
Despite rapid development, large language models (LLMs) still encounter challenges in multi-turn decision-making tasks (i.e., agent tasks) like web shopping and browser navigation, which require making a sequence of intelligent decisions based on environmental feedback. Previous work for LLM agents typically relies on elaborate prompt engineering or fine-tuning with expert trajectories to improve performance. In this work, we take a different perspective: we explore constructing process reward models (PRMs) to evaluate each decision and guide the agent's decision-making process. Unlike LLM reasoning, where each step is scored based on correctness, actions in agent tasks do not have a clear-cut correctness. Instead, they should be evaluated based on their proximity to the goal and the progress they have made. Building on this insight, we propose a re-defined PRM for agent tasks, named AgentPRM, to capture both the interdependence between sequential decisions and their contribution to the final goal. This enables better progress tracking and exploration-exploitation balance. To scalably obtain labeled data for training AgentPRM, we employ a Temporal Difference-based (TD-based) estimation method combined with Generalized Advantage Estimation (GAE), which proves more sample-efficient than prior methods. Extensive experiments across different agentic tasks show that AgentPRM is over 8× more compute-efficient than baselines, and it demonstrates robust improvement when scaling up test-time compute. Moreover, we perform detailed analyses to show how our method works and offer more insights, e.g., applying AgentPRM to the reinforcement learning of LLM agents.
Knowledge Graph Question Answering (KGQA) aims to answer natural language questions by reasoning over structured knowledge graphs (KGs). While large language models (LLMs) have advanced KGQA through their strong reasoning capabilities, existing methods continue to struggle to fully exploit both the rich knowledge encoded in KGs and the reasoning capabilities of LLMs, particularly in complex scenarios. They often assume complete KG coverage and lack mechanisms to judge when external information is needed, and their reasoning remains locally myopic, failing to maintain coherent multi-step planning, leading to reasoning failures even when relevant knowledge exists. We propose Graph-RFT, a novel two-stage reinforcement fine-tuning KGQA framework with a ''plan–KGsearch–and–Websearch–during–think'' paradigm, that enables LLMs to perform autonomous planning and adaptive retrieval scheduling across KG and web sources under incomplete knowledge conditions. Graph-RFT introduces a chain-of-thought (CoT) fine-tuning method with a customized plan–retrieval dataset activates structured reasoning and resolves the GRPO cold-start problem. It then introduces a novel plan–retrieval guided reinforcement learning process integrates explicit planning and retrieval actions with a multi-reward design, enabling coverage-aware retrieval scheduling. It employs a Cartesian-inspired planning module to decompose complex questions into ordered sub-questions, and logical expression to guide tool invocation for globally consistent multi-step reasoning. This reasoning–retrieval process is optimized with a multi-reward combining outcome and retrieval-specific signals, enabling the model to learn when and how to combine KG and web retrieval effectively. Experiments on multiple KGQA benchmarks demonstrate that Graph-RFT achieves superior performance over strong baselines, even with smaller LLM backbones, and substantially improves complex question decomposition, factual coverage, and tool coordination.
Deductive and abductive reasoning are two critical paradigms for analyzing knowledge graphs, enabling applications from financial query answering to scientific discovery. Deductive reasoning on knowledge graphs usually involves retrieving entities that satisfy a complex logical query, while abductive reasoning generates plausible logical hypotheses from observations. Despite their clear synergistic potential, where deduction can validate hypotheses and abduction can uncover deeper logical patterns, existing methods address them in isolation. To bridge this gap, we propose DARK, a unified framework for Deductive and Abductive Reasoning in Knowledge graphs. As a masked diffusion model capable of capturing the bidirectional relationship between queries and conclusions, DARK has two key innovations. First, to better leverage deduction for hypothesis refinement during abductive reasoning, we introduce a self-reflective denoising process that iteratively generates and validates candidate hypotheses against the observed conclusion. Second, to discover richer logical associations, we propose a logic-exploration reinforcement learning approach that simultaneously masks queries and conclusions, enabling the model to explore novel reasoning compositions. Extensive experiments on multiple benchmark knowledge graphs show that DARK achieves competitive performance on both deductive and abductive reasoning tasks, demonstrating the significant benefits of our unified approach.