Trending news detection in low-traffic search environments faces a fundamental cold-start problem, where a lack of query volume prevents systems from identifying emerging or long-tail trends. Existing methods relying on keyword frequency or query spikes are inherently slow and ineffective in these sparse settings, lagging behind real-world shifts in attention. We introduce RTTP, a novel Real-Time Trending Prediction framework that generates search queries directly from news content instead of waiting for users to issue them. RTTP leverages a continual learning LLM (CL-LLM) that converts posts into search-style queries and scores them using engagement strength + creator authority, enabling early trend surfacing before search volume forms. To ensure adaptation without degrading reasoning, we propose Mix-Policy DPO, a new preference-based continual learning approach that combines on-policy stability with off-policy novelty to mitigate catastrophic forgetting during model upgrades. Deployed at production scale on Facebook and Meta AI products, RTTP delivers +91.4% improvement in tail-trend detection precision@500 and +19% query generation accuracy over industry baselines, while sustaining stable performance after multi-week online training. This work demonstrates that LLM-generated synthetic search signals, when aligned and continually updated, unlock timely trend understanding in low-traffic search environments.
论文检索
输入标题、作者或关键词，从 510 篇学术成果中精准定位
Balancing multiple objectives is critical for user satisfaction in modern recommender and search systems, yet current Multi-Task Fusion (MTF) methods rely on static, manually-tuned weights that fail to capture individual user intent. While Reinforcement Learning (RL) offers a path to personalization, traditional approaches often falter due to training instability and the sparse rewards inherent in these large-scale systems. To address these limitations, we propose Group-Relative Reinforcement Learning with Adaptive Dirichlet Exploration (GRADE), a novel and robust framework for personalized multi-task fusion. GRADE leverages a critic-free, Group Relative Policy Optimization (GRPO) paradigm, enabling stable and efficient policy learning by evaluating the relative performance of candidate weight groups. Its core innovations include employing the Dirichlet distribution for principled and structured exploration of the weight space, and a composite reward function that combines sparse user feedback with dense model priors and rule-based constraints to guide the search effectively. Deployed in the in-app marketplace of an application with over hundreds of millions daily active users, GRADE significantly outperforms established baselines, achieving substantial gains in rigorous large-scale A/B tests: +0.595% in CTR, +1.193% in CVR, +1.788% in OPM, and +1.568% in total order volume. Following its strong performance, GRADE has been fully deployed in the marketplace search scenario of Kuaishou, serving hundreds of millions of users.
Query-product relevance prediction is fundamental to e-commerce search and has become even more critical in the era of AI-powered shopping, where semantic understanding and complex reasoning directly shape the user experience and exert an indirect, yet substantial impact on business conversion. Large Language Models (LLMs) enable generative, reasoning-based approaches, typically aligned via supervised fine-tuning (SFT) or preference optimization methods like Direct Preference Optimization (DPO). However, the increasing complexity of business rules and user queries exposes the inability of existing methods to endow models with robust reasoning capacity for long-tail and challenging cases. Efforts to address this via reinforcement learning strategies like Group Relative Policy Optimization (GRPO) often suffer from sparse terminal rewards, offering insufficient guidance for multi-step reasoning, which in turn slows convergence. To address these challenges, we propose TaoSR-AGRL, an Adaptive Guided Reinforcement Learning framework for LLM-based relevance prediction in Taobao Search Relevance. TaoSR-AGRL introduces two key innovations: (1) Rule-aware Reward Shaping, which decomposes the final relevance judgment into dense, structured rewards aligned with domain-specific relevance criteria; and (2) Adaptive Guided Replay, which identifies low-accuracy rollouts during training and injects targeted ground-truth guidance to steer the policy away from stagnant, rule-violating reasoning patterns toward compliant trajectories. TaoSR-AGRL was evaluated on large-scale datasets and through online evaluations on Taobao Search. It consistently outperforms DPO and GRPO baselines, improving relevance accuracy and rule adherence, with measurable gains in user engagement and stable training dynamics. The model has been deployed on Taobao, serving hundreds of millions of users.
In e-commerce search, the diverse ways in which users express intentions lead to lexical and semantic gaps between queries and product descriptions, making query rewriting (QR) indispensable for improving matching efficiency. With the development of LLMs, QR has evolved from discriminative approaches to various LLM-based alignment methods. However, these methods typically treat all queries uniformly, without fundamentally distinguishing their rewriting difficulty or underlying linguistic issues, making the rewritten query deviate from human expectations. To address this limitation, we propose AWHCP (Aligning with Human Cognition and Preference), a novel framework that adopts a human-centric perspective and introduces the Problem–Intention–Fix–Rewrite (PIFR) paradigm. Built upon PIFR, AWHCP establishes a multi-granularity alignment training framework that simultaneously aligns with both system retrieval preferences and human rewriting behaviors. First, we construct high-quality PIFR-structured data and perform supervised fine-tuning to enable the model to learn human-like rewriting patterns. Second, we apply beam search to generate multiple candidates and leverage system-side feedback signals to conduct coarse-grained direct preference alignment, endowing the model with initial difficulty-aware reasoning capabilities. Third, we introduce a multi-dimensional rewrite quality judgment model trained via Group Relative Policy Optimization (GRPO), enabling fine-grained alignment with nuanced human rewriting preferences. Deployed on 1688's main search engine since August 2025, AWHCP has demonstrated strong effectiveness through extensive offline evaluations and large-scale online A/B tests, leading to a +3.9% gain in UV-L2O.
Traditional search engines return uniform results for identical queries, overlooking users' personalized intents. While personalized search has been extensively studied, research on query rewriting for personalized intents has been constrained by traditional approaches like statistical co-occurrence and synonym expansion. Current work primarily addresses multi-turn dialogue scenarios in Conversational AI rather than exploring applications in large-scale search engines. This typically stems from the absence of real search scenario data and the difficulty of inferring users' intents through personalized reasoning. While Large Language Models (LLMs) combined with Chain-of-Thought (CoT) capabilities provide possibilities for personalized reasoning, CoT introduces additional reasoning overhead that is difficult to accept in online scenarios requiring low latency. To address this, this paper proposes PicQue (Personalized Efficient Query Rewrite), a personalized query rewriting model training pipeline aimed at achieving high accuracy with low latency. PicQue contains a two-stage training that first employs a novel Hybrid Supervised Fine-Tuning strategy to retain the model's reasoning capabilities while allowing the decoding process to skip CoT, thereby obtaining CoT's accuracy gains without increasing latency. Building on this foundation, PicQue conducts second-stage reinforcement learning using Group Relative Policy Optimization (GRPO) to further improve rewriting accuracy and reduce the risk of over-rewriting. We also propose a Guided Search strategy to optimize GRPO training, alleviating the reduction in training sample utilization when all sampling rollouts are wrong. Extensive offline and online experiments demonstrate PicQue's effectiveness. In offline metrics, PicQue achieves over 7% improvement in rewriting accuracy compared to baseline methods and compresses up to 95% of decoding tokens. In online A/B tests, user satisfaction increases by 1.78% and query change rate decreases by 0.71%, achieving significant online gains.
Federated learning is a distributed collaborative training framework that enables multiple clients to share model updates and jointly train deep neural networks without exchanging raw data. Although extensive research has explored data augmentation techniques in federated settings, the naturally non-IID data distributions among clients render blind augmentation prone to severe degradation of the learned model. To solve this problem, we suggest Responsive Augmentation and Approximate Update Method for Fed erated Learning with Non-IID Data (RAFed), aimed at alleviating feature shift in client samples. We leverage a Responsive Augmentation Method to accumulate shared data augmentation policy knowledge through local learning, guiding the policy gradient to consider the impact of data augmentation on unseen local data, and employ an Approximate Update Mechanism to reduce communication costs and achieve efficient policy search. To improve the adaptability of data augmentation policies to local data distributions, we introduce a Dynamic Adaptive Method for searching personalized augmentation policies tailored to heterogeneous clients. Experiments on four popular datasets show that RAFed achieves superior test accuracy and lower communication costs compared to related baselines while providing privacy advantages. The code is available via https://github.com/anonymously123-stcak/RAFed.
Semantic graphs are fundamental to the Web, enabling applications such as semantic search, recommendation, and knowledge-intensive reasoning. In decentralized Web environments, however, these graphs are distributed across organizations and constrained by strict privacy policies, making centralized training infeasible. Federated learning provides a promising solution, yet its effectiveness is severely limited by the dual incompleteness of real-world semantic graphs: missing node attributes and incomplete relational structures. Such dual missingness, often heterogeneous and unobserved across clients, causes substantial degradation in model performance. We present FedCL, a missingness-aware federated contrastive learning framework for dual-incomplete semantic graphs. FedCL introduces two key components: a topology estimation module, grounded in rate–distortion theory, that privately quantifies structural incompleteness across clients, and a federated reconstruction module that leverages these estimations to generate plausible relations without inferring sensitive attributes. To further improve robustness, FedCL integrates graph contrastive learning across reconstructed subgraphs, ensuring semantic consistency across heterogeneous and incomplete client graphs. Experiments on benchmark citation and Web datasets demonstrate that FedCL consistently outperforms state-of-the-art baselines in accuracy and robustness under heterogeneous missingness, while preserving strong privacy guarantees. These results highlight FedCL as a scalable and trustworthy approach for federated learning on incomplete semantic graphs, advancing privacy-preserving knowledge sharing on the Web.
Conversational search allows multi-turn user-system interactions to support complex information seeking. It is challenging due to frequent topic shifts and ambiguous intentions in conversation. Although existing methods attempt to optimize retrieval performance through query rewriting or session encoding, they still face crucial challenges, including (1) Noisy context: the diverse types of noises in the conversation context prevent models from achieving reliable and consistent contextual understanding; (2) Poor interpretability: the lack of transparency in how results are generated undermines user trust, hindering the practical application of conversational search systems. In this paper, we propose LogiCGR, a novel framework that utilizes curriculum learning and group relative policy optimization (GRPO) to perform logic-enhanced retrieval, improving the robustness and interpretability of conversational search. Specifically, LogiCGR equips large language models (LLMs) with logical denoising and generative retrieval, integrating them seamlessly through an adaptive framework. Additionally, we introduce a lightweight module that works with generative retrieval for self-dual-path retrieval, thus delivering complementary performance gains. Extensive experiments and intuitive case studies demonstrate that our proposed LogiCGR outperforms state-of-the-art baselines in both retrieval performance and interpretability. The code and data are available at https://github.com/GenIRAG/LogiCGR.
Deep search agents, which autonomously iterate through multi-turn web-based reasoning, represent a promising paradigm for complex information-seeking tasks. However, current agents suffer from critical inefficiency: they conduct excessive searches as they cannot accurately judge when to stop searching and start answering. This stems from outcome-centric training that prioritize final results over the search process itself. We identify the root cause as misaligned decision boundaries, the threshold determining when accumulated information suffices to answer. This causes over-search (redundant searching despite sufficient knowledge) and under-search (premature termination yielding incorrect answers). To address these errors, we propose a comprehensive framework comprising two key components. First, we introduce causal intervention-based diagnosis that identifies boundary errors by comparing factual and counterfactual trajectories at each decision point. Second, we develop Decision Boundary Alignment for Deep Search agents (DAS), which constructs preference datasets from causal feedback and aligns policies via preference optimization. Experiments on public datasets demonstrate that decision boundary errors are pervasive across state-of-the-art agents. Our DAS method effectively calibrates these boundaries, mitigating both over-search and under-search to achieve substantial gains in accuracy and efficiency. Our code and data are publicly available at: https://github.com/Applied-Machine-Learning-Lab/WWW2026_DAS.
Extracting multi-step explanations from knowledge graphs poses a combinatorial challenge requiring both heuristic guidance (as candidates proliferate with depth) and credit assignment (as path quality emerges over extended sequences). Frontier LLMs, strong on knowledge/reasoning benchmarks, offer a compelling source of such heuristics, yet their knowledge comes sans guarantees and compositional performance degrades as chains lengthen. We thus present TESSERA, a 3-part neuro-symbolic framework that uses LLMs in a circumscribed role: for local discriminative judgement rather than autonomous multi-step generation; the knowledge graph then defines the hypothesis space enforcing hard structural constraints, and MCTS coordinates the long-horizon search with principled credit assignment via backpropagation. LLMs perform dual roles as a prior policy biasing exploration and a comparative state evaluator supplying reward signals. Evaluation on drug mechanism elucidation across two complementary knowledge graphs demonstrates fidelity to curated biology while surfacing coherent alternative mechanisms, with ablations confirming discriminative contribution from both LLM components. Beyond its current application, our framework offers a general paradigm for compositional reasoning over structured knowledge.
The deployment of edge servers plays a crucial role in supporting large-scale edge computing systems, where multiple conflicting objectives—such as latency, energy consumption, load balancing, and service reliability—must be jointly optimized in complex, dynamic environments. Existing solutions often struggle to scale effectively or to balance these objectives in a unified learning framework. In this paper, we propose GMM-TDQN, a two-stage multi-objective reinforcement learning framework for large-scale edge server deployment. The first stage employs a Gaussian Mixture Model (GMM) to capture spatial and workload heterogeneity, enabling an efficient reduction of the deployment search space. Building upon this structured initialization, the second stage formulates the deployment problem as a sequential decision-making task and adopts a Transformer-enhanced Deep Q-Network (TDQN) to learn adaptive deployment policies that balance multiple objectives. Extensive experiments on real-world datasets demonstrate that GMM-TDQN consistently outperforms state-of-the-art methods, achieving reductions of 29.18% in average latency and 17.55% in energy consumption, while improving load balancing by 27.50% and system reliability by 32.55%. These results validate the effectiveness and scalability of the proposed framework for multi-objective edge server deployment.
This paper studies a risk-sensitive formulation of the multi-robot search problem, termed multi-robot mean-standard deviation search (MuRMSS), in which a team of robots cooperatively search for a moving target by minimizing a linear combination of the mean and standard deviation of search time. However, the standard deviation term is inherently non-additive, making it difficult to estimate, incompatible with canonical multi-robot search algorithms, and preventing consistent decomposition into individual robot utilities, which is essential for scalable multi-robot cooperation. In view of these challenges, we propose MV-FAC, which comprises a mean-variance temporal-difference module that jointly learns the mean and variance of search time, a factorization module that decomposes them into individual utilities, and a decentralized policy optimization module that minimizes each robot’s individual mean-std objective. We further establish and prove the mean-std individual global minimization (MS-IGM) theorem, thereby ensuring consistency between individual- and team-level objectives. Extensive simulation studies on standard multi-robot search benchmarks demonstrate that MV-FAC achieves the best overall mean-std search-time performance. We also validate MV-FAC's practicality by deploying it on a physical multi-robot system for moving target search in a real-world building environment.
Reward specification plays a central role in reinforcement learning (RL), guiding the agent’s behavior. To express non-Markovian rewards, formalisms such as reward machines have been introduced to capture dependencies on histories. However, traditional reward machines lack the ability to model precise timing constraints, limiting their use in time-sensitive applications. In this paper, we propose timed reward machines (TRMs), which are an extension of reward machines that incorporate timing constraints into the reward structure. TRMs enable more expressive specifications with tunable reward logic, for example, imposing costs for delays and granting rewards for timely actions. We study model-free RL frameworks (i.e., tabular Q-learning) for learning optimal policies with TRMs under digital and real-time semantics. Our algorithms integrate the TRM into learning via abstractions of timed automata and employ counterfactual-imagining heuristics that exploit the TRM's structure to improve search. Experimentally, we demonstrate that our algorithm learns policies that achieve high rewards while satisfying the timing constraints specified by the TRM on popular RL benchmarks.
Under the shift toward Industry 4.0, mass-customized manufacturing systems have introduced complex scheduling problems, such as the Flexible Job Shop Scheduling Problem (FJSSP). Recent Deep Reinforcement Learning (DRL)-based heuristics have shown promise, yet existing methods often suffer from two key limitations: they typically rely on single-policy optimization, which limits exploration, and on imprecise reward functions, which fail to accurately reflect decision quality. To address these challenges simultaneously, we propose PGMPO (Preference-Guided Multi-Policy Optimization), a novel learning framework consisting of (1) a simple but effective multi-policy modeling approach that allows a single network to represent multiple decision-makers, and (2) a preference-driven model optimization method that effectively guides policies to learn diverse and specialized problem-solving strategies without the need for explicit reward functions. Experimental results demonstrate that PGMPO substantially boosts the performance of existing neural solvers across several benchmarks.
Joint optimization of pricing, dispatching, and routing is critical for hub-based mobility services but challenging due to complex decision couplings and strict service guarantees, such as Order Response Rate (ORR). Conventional constrained reinforcement learning often struggles in this mixed continuous--combinatorial action space, suffering from oscillatory behavior in Lagrangian dual variables and unstable constraint satisfaction. To address this, we propose PID-SACA, a unified framework that integrates an entropy-regularized actor--critic policy for continuous pricing and dispatching assisted by an embedded routing solver for execution-aware feedback. Crucially, we adapt the PID control mechanism to the Lagrangian dual update process. This approach leverages proportional, integral, and derivative feedback to dampen oscillations caused by stochastic gradient variance, ensuring robust long-term constraint enforcement. We provide theoretical analysis on the boundedness of dual variables, and experiments on publicly available large-scale mobility datasets demonstrate that PID-SACA significantly outperforms baselines, achieving high revenue with stable service compliance. Code: https://github.com/jerry0375/PID-SACA
The three-dimensional bin packing problem (3D-BPP) is widely applied in logistics and warehousing. However, existing learning-based approaches often neglect stability constraints and struggle to generalize across diverse bin dimensions. To address this, we propose a novel deep reinforcement learning framework, One4Many-StablePacker (O4M-SP). The primary advantage of O4M-SP lies in its ability to handle variable bin dimensions in a single training process while explicitly enforcing two types of practical stability constraints: support constraints, which ensure an item's bottom center lies within the convex hull of the underlying contact area, and weight constraints, which restrict the total vertical load on an item to its bearing capacity. Our training method introduces two innovative mechanisms. First, a weighted reward function integrates the loading rate with a novel height difference metric for packing layouts, promoting improved bin utilization via flatter packing configurations. Second, clipped policy gradient optimization with tailored policy drifting mitigates entropy collapse, encouraging exploration at critical decision nodes during packing to prevent premature convergence. Extensive experiments demonstrate that O4M-SP generalizes effectively across diverse bin dimensions and significantly outperforms baseline methods. Furthermore, the framework exhibits strong practical applicability by solving complex scenarios under strict stability constraints.
Systematic safety evaluation of large language models must uncover diverse policy violations under tight query budgets. However, most red-teaming methods optimize attack success rate and repeatedly probe a narrow set of vulnerabilities, yielding redundant failures and leaving rarer yet critical violation categories unexplored. Under fixed budgets, such inefficient exploration delays the first discovery and limits category coverage. To address these limitations, we propose the Budget-Efficient Adaptive Cognitive Offense Navigator (BEACON), a budget-aware safety testing framework that uses Cognitive-Guided Monte Carlo Tree Search to navigate the violation search space under fixed budgets. BEACON innovatively approaches safety testing as a budget-constrained failure discovery process, aiming to identify diverse safety violations as early as possible within a fixed query budget. It also provides an efficiency-oriented evaluation perspective that measures early discovery and harm category coverage under budget constraints. Experiments on standard benchmarks and frontier LLMs show that BEACON discovers failures earlier and achieves higher coverage across policy violation categories. These results underscore the value of evaluating safety testing through discovery efficiency rather than attack success rate alone. Warning: This paper contains examples of harmful language and images, and reader discretion is recommended.
For minimally rigid graphs, the same edge-length data can admit multiple realizations (up to translations and rotations). Finding graphs with exceptionally many realizations is an extremal problem in rigidity theory, but exhaustive search quickly becomes infeasible due to the super-exponential growth of the number of candidate graphs and the high cost of realization-count evaluation. We propose a reinforcement-learning approach that constructs minimally rigid graphs via 0- and 1-extensions, also known as Henneberg moves. We optimize realization-count invariants using the Deep Cross-Entropy Method with a policy parameterized by a Graph Isomorphism Network encoder and a permutation-equivariant extension-level action head. Empirically, our method matches the known optima for planar realization counts and improves the best known bounds for spherical realization counts, yielding new record graphs.
Symbolic regression (SR) aims to recover compact and interpretable mathematical expressions from data. Genetic programming (GP) directly searches over symbolic structures, but its population dynamics can make it difficult to reliably preserve and accumulate useful subexpressions. In contrast, reinforcement learning (RL)-based SR has shown strong empirical performance, suggesting that learned sampling dynamics may capture useful regularities in symbolic search. Motivated by this contrast, we analyze expressions sampled during RL training and identify a recurring pattern, termed progressive subexpression reuse, where useful simple subexpressions emerge early, become increasingly frequent, and support the formation of more complex structures. Based on this observation, we propose Reinforcement Genetic Programming (RGP), a purely GP-based and non-RL framework that explicitly realizes a stage-wise retention--reintroduction loop through a dynamically maintained subexpression pool and pool-guided population initialization. Experiments on standard SR benchmarks show that RGP matches or outperforms strong baselines without RL policy training, suggesting that progressive subexpression reuse is an effective mechanism for SR search. We release our code at https://github.com/wuxiangdong586-max/RGP.
Multimodal reasoning requires a path that retains integrity over a wide range of constraints, from visual grounding to logic consistency. However, the current Process Reward Models (PRMs) focus on heuristically defined rewards that equally weigh these factors, which may lead to the concealment of individual dimension failures (such as visual hallucinations) by the dominating factors, without guaranteeing the validity of the reasoning process in general. Therefore, to overcome the limitation, the paper proposes the concept of Multimodal Multi-Dimensional Scalarization Process Reward Modeling (MMS-PRM), a paradigm specifically developed to enforce the worst dimension’s robustness in multimodal reasoning. Specifically, a hierarchical fine-grained reward space is developed to represent the multimodal risks in the reasoning tasks, and a Chebyshev-based Monte Carlo Tree Search (MCTS) algorithm is introduced, in which the primary focus during the path searching is given to the worst-performing dimension. Moreover, a curriculum-based Direct Preference Optimization (DPO) approach is developed to gradually learn the balanced reasoning skills in the policy. The experimental results show that, without the dimension collapse issue, the MMS-PRM approach significantly improves the reliability of the multimodal reasoning performance and reaches competitive results in various challenging tasks. The code is available at https://github.com/leibniz-Man/MMS-PRM