We present ReSuMe, a general framework for mutual enhancement of dense retrieval systems and document summarizers through reinforcement learning. The framework jointly optimizes a language model for generating retrieval-oriented summaries and adapts the retrieval model to these summaries through alternating fine-tuning phases. We employ Group Relative Policy Optimization (GRPO) to fine-tune the language model based on retrieval relevance rather than linguistic quality alone, while the retrieval model is iteratively updated using contrastive learning on the generated summaries. This co-optimization process addresses the fundamental distribution shift problem that arises when retrieval models trained on full documents must operate on synthetic summaries during inference. By progressively reducing this distribution gap, our framework yields two key benefits: improved retrieval performance and a high-quality document summarizer optimized for retrieval tasks. We demonstrate our framework using Contriever on the MS-MARCO dataset, achieving consistent improvements of 13.2% in MRR@10 and 6.7% in Recall@100 over the baseline. The framework is model-agnostic and can be applied to enhance any dense retrieval system while simultaneously producing an effective document summarization model.
论文检索
输入标题、作者或关键词，从 15,773 篇学术成果中精准定位
This study investigates the problem of keeping information up-to-date when crawling data sources that change over time (e.g., websites or location data). Traditional crawling methods often treat data sources independently, making it difficult to capture relationships and propagate updates efficiently. We propose using graph structures to model these relationships and show that, unfortunately, finding the theoretically optimal solution can be intractable. To address this, we introduce a specific graphical model (the latent Bernoulli process model) and demonstrate the complexity of even simple tasks within this framework. We tackle the crawling problem using a reinforcement learning-based algorithm and demonstrate its superiority over traditional baselines on both real and synthetic data. This work highlights the power of graph-structured crawling in helping users stay informed within a dynamic information landscape.
Accurate estimation of delivery time (EDT) is a critical factor in web e-commerce user experience. The pursuit of higher EDT accuracy has predominantly centered on designing increasingly complex model architectures. While valuable, this architecture-centric paradigm creates a tension between its high iteration costs and the industrial demand for agile deployment. This work, therefore, explores a complementary dimension: enhancing model performance by optimizing the learning process itself. We propose EDTF, a novel, plug-and-play composite learning framework that empowers existing models by augmenting their learning objective. EDTF first transforms the traditional regression problem into a structured ordinal classification task to address the training difficulties inherent in direct regression and preserve temporal order. It then introduces a cross-view consistency paradigm, decomposing the prediction task into two related views: the macroscopic end-to-end delivery time and the microscopic next-hop duration. By enforcing a self-supervised signal that aligns the sum of future next-hop durations with the overall EDT, our framework enables models to learn more robust temporal representations without extra features. Extensive experiments on a large-scale industrial dataset show that EDTF, as a plugin, consistently enhances performance and accelerates convergence across five diverse architectures. Critically, an EDTF-optimized model has been successfully deployed in a live production environment, demonstrating significant improvements over its predecessor. This work thus presents a validated and valuable new paradigm for the economical and efficient application of web services reliant on trajectory-based forecasting, from e-commerce to ride-hailing and food delivery.
Inspired by advances in LLMs, reasoning-enhanced sequential recommendation performs multi-step deliberation before making final predictions, unlocking greater potential for capturing user preferences. However, current methods are constrained by static reasoning trajectories that are ill-suited for the diverse complexity of user behaviors. They suffer from two key limitations: (1) a static reasoning direction, which uses flat supervision signals misaligned with human-like hierarchical reasoning, and (2) a fixed reasoning depth, which inefficiently applies the same computational effort to all users, regardless of pattern complexity. These rigidity lead to suboptimal performance and significant computational waste. To overcome these challenges, we propose DTRec, a novel and effective framework that explores the Dynamic reasoning Trajectory for Sequential Recommendation along both direction and depth. To guide the direction, we develop Hierarchical Process Supervision (HPS), which provides coarse-to-fine supervisory signals to emulate the natural, progressive refinement of human cognitive processes. To optimize the depth, we introduce the Adaptive Reasoning Halting (ARH) mechanism that dynamically adjusts the number of reasoning steps by jointly monitoring three indicators. Extensive experiments on three real-world datasets demonstrate the superiority of our approach, achieving up to a 24.5% performance improvement over strong baselines while simultaneously reducing computational cost by up to 41.6%.
Optimizing numerical systems and mechanism design is crucial for enhancing player experience in Massively Multiplayer Online (MMO) games. Traditional optimization approaches rely on iterative online experiments or parameter tuning over abstracted statistical models, which can be inaccurate, time-consuming and potentially impair players' experience. Although simplified offline simulation systems are frequently employed as alternatives, their low fidelity constrains agents' ability to faithfully replicate real players' reasoning processes and behavioral responses to interventions. To address these limitations, we propose a generative agent-based MMO simulation system with hundreds of agents empowered by Large Language Models (LLMs). By applying Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) on large-scale real player behavioral data, we adapt LLMs from general priors to game-specific domains, enabling realistic and interpretable player decision-making. In parallel, a data-driven environment model trained on real gameplay logs reconstructs dynamic in-game systems. Experiments demonstrate strong consistency with real-world player behaviors and plausible causal responses under interventions, providing a reliable, interpretable, and cost-efficient framework for data-driven numerical design optimization.
Large-scale digital platforms generate billions of timestamped user-item interactions (events) that are crucial for predicting user attributes in, e.g., fraud prevention and recommendations. While self-supervised learning (SSL) effectively models the temporal order of events, it typically overlooks the global structure of the user-item interaction graph. To bridge this gap, we propose three model-agnostic strategies for integrating this structural information into contrastive SSL: enriching event embeddings, aligning client representations with graph embeddings, and adding a structural pretext task. Experiments on four financial and e-commerce datasets demonstrate that our approach consistently improves the accuracy (up to a 2.3% AUC) and reveals that graph density is a key factor in selecting the optimal integration strategy.
Large Language Models (LLMs) have shown significant potential for improving recommendation systems through their inherent reasoning capabilities and extensive knowledge base. Yet, existing studies predominantly address warm-start scenarios with abundant user-item interaction data, leaving the more challenging cold-start scenarios, where sparse interactions hinder traditional collaborative filtering methods, underexplored. To address this limitation, we propose novel reasoning strategies designed for cold-start item recommendations within the Netflix domain. Our method utilizes the advanced reasoning capabilities of LLMs to effectively infer user preferences, particularly for newly introduced or rarely interacted items. We systematically evaluate supervised fine-tuning, reinforcement learning-based fine-tuning, and hybrid approaches that combine both methods to optimize recommendation performance. Extensive experiments on real-world data demonstrate significant improvements in both methodological efficacy and practical performance in cold-start recommendation contexts. Remarkably, our reasoning-based fine-tuned models outperform Netflix's production ranking model by up to 8% in certain cases.
Balancing multiple objectives is critical for user satisfaction in modern recommender and search systems, yet current Multi-Task Fusion (MTF) methods rely on static, manually-tuned weights that fail to capture individual user intent. While Reinforcement Learning (RL) offers a path to personalization, traditional approaches often falter due to training instability and the sparse rewards inherent in these large-scale systems. To address these limitations, we propose Group-Relative Reinforcement Learning with Adaptive Dirichlet Exploration (GRADE), a novel and robust framework for personalized multi-task fusion. GRADE leverages a critic-free, Group Relative Policy Optimization (GRPO) paradigm, enabling stable and efficient policy learning by evaluating the relative performance of candidate weight groups. Its core innovations include employing the Dirichlet distribution for principled and structured exploration of the weight space, and a composite reward function that combines sparse user feedback with dense model priors and rule-based constraints to guide the search effectively. Deployed in the in-app marketplace of an application with over hundreds of millions daily active users, GRADE significantly outperforms established baselines, achieving substantial gains in rigorous large-scale A/B tests: +0.595% in CTR, +1.193% in CVR, +1.788% in OPM, and +1.568% in total order volume. Following its strong performance, GRADE has been fully deployed in the marketplace search scenario of Kuaishou, serving hundreds of millions of users.
Kuaishou serving hundreds of millions of searches daily, the quality of short-video search is paramount. However, it suffers from a severe Matthew effect on long-tail queries: sparse user behavior data causes models to amplify low-quality content such as clickbait and shallow content. The recent advancements in Large Language Models (LLMs) offer a new paradigm, as their inherent world knowledge provides a powerful mechanism to assess content quality, agnostic to sparse user interactions. To this end, we propose a LLM-driven multimodal reranking framework, which estimates user experience without real user behavior. The approach involves a two-stage training process: the first stage uses multimodal evidence to construct high-quality annotations for supervised fine-tuning, while the second stage incorporates pairwise preference optimization to help the model learn partial orderings among candidates. At inference time, the resulting experience scores are used to promote high-quality but underexposed videos in reranking, and further guide page-level optimization through reinforcement learning. Experiments show that the proposed method achieves consistent improvements over strong baselines in offline metrics including AUC, NDCG@K, and human preference judgement. An online A/B test covering 15% of traffic further demonstrates gains in both user experience and consumption metrics, confirming the practical value of the approach in long-tail video search scenarios.
The proliferation of ride-hailing services has fundamentally transformed urban mobility patterns, making accurate ride-hailing forecasting crucial for optimizing passenger experience and urban transportation efficiency. However, ride-hailing forecasting faces significant challenges due to geospatial heterogeneity and high susceptibility to external events. This paper proposes MVGR-Net (Multi-View Geospatial Representation Learning), a novel framework that addresses these challenges through a two-stage approach. In the pre-training stage, we learn comprehensive geospatial representations by integrating Points-of-Interest and temporal mobility patterns to capture regional characteristics from both semantic attribute and temporal mobility pattern views. The forecasting stage leverages these representations through a prompt-empowered framework that fine-tunes Large Language Models while incorporating external events. Extensive experiments on DiDi's real-world datasets demonstrate the state-of-the-art performance.
In modern e-commerce search systems, dense retrieval has become an indispensable component. By computing similarities between query and item (product) embeddings, it efficiently selects candidate products from large-scale repositories. With the breakthroughs in large language models (LLMs), mainstream embedding models have gradually shifted from BERT to LLMs for more accurate text modeling. However, these models still adopt direct-embedding methods, and the semantic accuracy of embeddings remains inadequate. Therefore, contrastive learning is heavily employed to achieve tight semantic alignment between positive pairs. Consequently, such models tend to capture statistical co-occurrence patterns in the training data, biasing them toward shallow lexical and semantic matches. For difficult queries exhibiting notable lexical disparity from target items, the performance degrades significantly. In this work, we propose the Large Reasoning Embedding Model (LREM), which novelly integrates reasoning processes into representation learning. For difficult queries, LREM first conducts reasoning to achieve a deep understanding of the original query, and then produces a reasoning-augmented query embedding for retrieval. This reasoning process effectively bridges the semantic gap between original queries and target items, significantly improving retrieval accuracy. Specifically, we adopt a two-stage training process: the first stage optimizes the LLM on carefully curated Query-CoT-Item triplets with SFT and InfoNCE losses to establish preliminary reasoning and embedding capabilities, and the second stage further refines the reasoning trajectories via reinforcement learning (RL). Extensive offline and online experiments validate the effectiveness of LREM, leading to its deployment on China's largest e-commerce platform since August 2025.
Large Language Models (LLMs) pose a new paradigm of modeling and computation for information tasks. Recommendation systems are a critical application domain poised to benefit significantly from the sequence modeling capabilities and world knowledge inherent in these large models. In this paper, we introduce PLUM, a framework designed to adapt pre-trained LLMs for industry-scale recommendation tasks. PLUM consists of item tokenization using Semantic IDs, continued pre-training (CPT) on domain-specific data, and task-specific fine-tuning for recommendation objectives. For fine-tuning, we focus particularly on generative retrieval, where the model is directly trained to generate Semantic IDs of recommended items based on user context. We conduct comprehensive experiments on large-scale internal video recommendation datasets. Our results demonstrate that PLUM achieves substantial improvements for retrieval compared to a heavily-optimized production model built with large embedding tables. We also present a scaling study for the model's retrieval performance, our learnings about CPT, a few enhancements to Semantic IDs, along with an overview of the training and inference methods that enable launching this framework to billions of users in YouTube.
Query-product relevance prediction is fundamental to e-commerce search and has become even more critical in the era of AI-powered shopping, where semantic understanding and complex reasoning directly shape the user experience and exert an indirect, yet substantial impact on business conversion. Large Language Models (LLMs) enable generative, reasoning-based approaches, typically aligned via supervised fine-tuning (SFT) or preference optimization methods like Direct Preference Optimization (DPO). However, the increasing complexity of business rules and user queries exposes the inability of existing methods to endow models with robust reasoning capacity for long-tail and challenging cases. Efforts to address this via reinforcement learning strategies like Group Relative Policy Optimization (GRPO) often suffer from sparse terminal rewards, offering insufficient guidance for multi-step reasoning, which in turn slows convergence. To address these challenges, we propose TaoSR-AGRL, an Adaptive Guided Reinforcement Learning framework for LLM-based relevance prediction in Taobao Search Relevance. TaoSR-AGRL introduces two key innovations: (1) Rule-aware Reward Shaping, which decomposes the final relevance judgment into dense, structured rewards aligned with domain-specific relevance criteria; and (2) Adaptive Guided Replay, which identifies low-accuracy rollouts during training and injects targeted ground-truth guidance to steer the policy away from stagnant, rule-violating reasoning patterns toward compliant trajectories. TaoSR-AGRL was evaluated on large-scale datasets and through online evaluations on Taobao Search. It consistently outperforms DPO and GRPO baselines, improving relevance accuracy and rule adherence, with measurable gains in user engagement and stable training dynamics. The model has been deployed on Taobao, serving hundreds of millions of users.
In e-commerce search, the diverse ways in which users express intentions lead to lexical and semantic gaps between queries and product descriptions, making query rewriting (QR) indispensable for improving matching efficiency. With the development of LLMs, QR has evolved from discriminative approaches to various LLM-based alignment methods. However, these methods typically treat all queries uniformly, without fundamentally distinguishing their rewriting difficulty or underlying linguistic issues, making the rewritten query deviate from human expectations. To address this limitation, we propose AWHCP (Aligning with Human Cognition and Preference), a novel framework that adopts a human-centric perspective and introduces the Problem–Intention–Fix–Rewrite (PIFR) paradigm. Built upon PIFR, AWHCP establishes a multi-granularity alignment training framework that simultaneously aligns with both system retrieval preferences and human rewriting behaviors. First, we construct high-quality PIFR-structured data and perform supervised fine-tuning to enable the model to learn human-like rewriting patterns. Second, we apply beam search to generate multiple candidates and leverage system-side feedback signals to conduct coarse-grained direct preference alignment, endowing the model with initial difficulty-aware reasoning capabilities. Third, we introduce a multi-dimensional rewrite quality judgment model trained via Group Relative Policy Optimization (GRPO), enabling fine-grained alignment with nuanced human rewriting preferences. Deployed on 1688's main search engine since August 2025, AWHCP has demonstrated strong effectiveness through extensive offline evaluations and large-scale online A/B tests, leading to a +3.9% gain in UV-L2O.
We present BanditLP, a scalable multi-stakeholder contextual bandit framework that unifies neural Thompson Sampling (TS) for learning objective-specific outcomes with a large-scale linear program (LP) for constrained action selection at serving time. The methodology is application-agnostic, compatible with arbitrary neural architectures, and deployable at web scale, with an LP solver capable of handling billions of variables. Experiments on public benchmarks and synthetic data show consistent gains over strong baselines. We apply this approach in LinkedIn's email marketing system and demonstrate business win, illustrating the value of integrated exploration and constrained optimization in production.
Traditional search engines return uniform results for identical queries, overlooking users' personalized intents. While personalized search has been extensively studied, research on query rewriting for personalized intents has been constrained by traditional approaches like statistical co-occurrence and synonym expansion. Current work primarily addresses multi-turn dialogue scenarios in Conversational AI rather than exploring applications in large-scale search engines. This typically stems from the absence of real search scenario data and the difficulty of inferring users' intents through personalized reasoning. While Large Language Models (LLMs) combined with Chain-of-Thought (CoT) capabilities provide possibilities for personalized reasoning, CoT introduces additional reasoning overhead that is difficult to accept in online scenarios requiring low latency. To address this, this paper proposes PicQue (Personalized Efficient Query Rewrite), a personalized query rewriting model training pipeline aimed at achieving high accuracy with low latency. PicQue contains a two-stage training that first employs a novel Hybrid Supervised Fine-Tuning strategy to retain the model's reasoning capabilities while allowing the decoding process to skip CoT, thereby obtaining CoT's accuracy gains without increasing latency. Building on this foundation, PicQue conducts second-stage reinforcement learning using Group Relative Policy Optimization (GRPO) to further improve rewriting accuracy and reduce the risk of over-rewriting. We also propose a Guided Search strategy to optimize GRPO training, alleviating the reduction in training sample utilization when all sampling rollouts are wrong. Extensive offline and online experiments demonstrate PicQue's effectiveness. In offline metrics, PicQue achieves over 7% improvement in rewriting accuracy compared to baseline methods and compresses up to 95% of decoding tokens. In online A/B tests, user satisfaction increases by 1.78% and query change rate decreases by 0.71%, achieving significant online gains.
Recommender Systems (RS) are fundamental to modern online services. While most existing approaches optimize for short-term engagement, recent work has begun to explore reinforcement learning (RL) to model long-term user value. However, these efforts face significant challenges due to the vast, dynamic action spaces inherent in RS, which hinder stable policy learning. To resolve this bottleneck, we introduce Hierarchical Semantic RL (HSRL), which reframes RL-based recommendation over a fixed Semantic Action Space (SAS). HSRL encodes items as Semantic IDs (SIDs) for policy learning, and maps SIDs back to their original items via a fixed lookup during execution. To align decision-making with SID generation, the Hierarchical Policy Network (HPN) operates in a coarse-to-fine manner, employing hierarchical residual state modeling to refine each level's context from the previous level's residual, thereby reducing representation–decision mismatch. In parallel, a Multi-level Critic (MLC) provides token-level value estimates, enabling fine-grained credit assignment. Across public benchmarks and a large-scale production dataset from a leading short-video advertising platform, HSRL consistently surpasses state-of-the-art baselines. In online deployment over a 7-day A/B testing, it delivers an 18.421% ADVV lift and a 1.251% increase in Revenue, supporting HSRL as a scalable paradigm for RL-based recommendation.
Web recommendations provide personalized items from massive catalogs for users, which rely heavily on retrieval stages to trade off the effectiveness and efficiency of selecting a small relevant set from billion-scale candidates in online digital platforms. As one of the largest Chinese search engine and news feed providers, Baidu resorts to Deep Neural Network (DNN) and graph-based Approximate Nearest Neighbor Search (ANNS) algorithms for accurate relevance estimation and efficient search for relevant items. However, current retrieval at Baidu fails in comprehensive user-item relational understanding due to dissected interaction modeling, and performs inefficiently in large-scale graph-based ANNS because of suboptimal traversal navigation and the GPU computational bottleneck under high concurrency. To this end, we propose a GPU-accelerated Multi-relational Parallel Graph Retrieval (GMP-GR) framework to achieve effective yet efficient retrieval in web-scale recommendations. First, we propose a multi-relational user-item relevance metric learning method that unifies diverse user behaviors through multi-objective optimization and employs a self-covariant loss to enhance pathfinding performance. Second, we develop a hierarchical parallel graph-based ANNS to boost graph retrieval throughput, which conducts breadth-depth-balanced searches on a large-scale item graph and cost-effectively handles irregular neural computation via adaptive aggregation on GPUs. In addition, we integrate system optimization strategies in the deployment of GMP-GR in Baidu. Extensive experiments demonstrate the superiority of GMP-GR in retrieval accuracy and efficiency. Deployed across more than twenty applications at Baidu, GMP-GR serves hundreds of millions of users with a throughput exceeding one hundred million requests per second.
In real-world industrial scenarios, large language models (LLMs) require Continuous Learning (CL) to adapt to diverse tasks as opera- tional requirements diversify, demanding self-evolution capabilities to autonomously refine their knowledge and adapt to dynamic envi- ronments. However, existing CL approaches, such as replay-based and parameter isolation techniques, struggle with the catastrophic forgetting problem: new task training degrades performance on prior tasks due to the model's adaptation to new data distributions, which weakens its generalization to old tasks. To address this issue, we propose a novel parameter-efficient adversarial MoE framework, MoE-CL, for industrial-scale self-evolving continual instruction tuning of LLMs. Specifically, MoE-CL employs a dual-expert archi- tecture to enable self-evolution: a dedicated LoRA expert for each task to preserve task-specific knowledge, ensuring parameter inde- pendence and mitigating forgetting, and a shared LoRA expert to facilitate cross-task knowledge transfer. Specifically, a task-aware discriminator within a Generative Adversarial Network (GAN) is integrated into the shared expert to suppress task-irrelevant noise, ensuring only task-aligned knowledge is transferred during se- quential task training. Through adversarial training, the shared ex- pert learns generalized representations that mimic the task-aware discriminator, while dedicated experts retain task-specific details, balancing knowledge retention and cross-task generalization—key to the model's self-evolution by autonomously optimizing knowl- edge integration across tasks. Extensive experiments on a public MTL5 benchmark and an industrial Tencent3 benchmark validate MoE-CL's effectiveness in self-evolving continual learning. In real- world A/B testing on content compliance review in the Tencent Video Platform, MoE-CL reduced manual review costs by 15.3%,demonstrating its applicability for large-scale industrial deploy- ment where self-evolution is critical for adapting to evolving op- erational demands. Implementation code is publicly available at https://github.com/BAI-LAB/MoE-CL.
Large Language Models (LLMs) enable Web-scale multilingual content analysis but face critical challenges in scaling to long-tail languages and ensuring robustness. Current research is split between two isolated trajectories: a Macro-Paradigm (system-level engineering) and a Micro-Paradigm (internal model intervention). We argue that a true Web-scale solution requires their systematic fusion, balancing large-scale data processing with fine-grained model control. We introduce the Control-Tower Framework (CTF), a novel methodology designed to systematically enhance powerful, pre-trained base models. Inspired by control-theoretic ideas, CTF transforms a base model into a controllable analysis engine via three synergistic stages: (1) Micro-enhanced pre-training that injects linguistic priors (e.g., syntax) to build a robust semantic foundation; (2) a control-inspired fine-tuning stage where a heuristic dynamic feedback loop, driven by micro-level error signals (e.g., knowledge editing loss), actively adjusts the macro-scale learning curriculum; and (3) Macro-optimized inference using Minimum Bayes Risk (MBR) decoding to enhance robustness on noisy user-generated content (UGC). Extensive experiments show that CTF surpasses the leading open-weights model, Tower+ 9B FT, by a substantial margin of +2.18 XCOMET-XXL on low-resource languages (WMT24++). Crucially, CTF unlocks large-scale cross-lingual Web mining by converting unstructured Web text into machine-analyzable assets. We evidence this with substantial gains across both document-level (on MARC) and aspect-based (on SemEval-2016) sentiment analysis tasks. Our work offers a practical pathway toward building more reliable, scalable, and controllable global information ecosystems.