We present a data-driven approach for physics-based, muscle-driven dexterous control that enables musculoskeletal hands to perform precise piano playing for novel pieces of music outside the reference dataset. Our approach combines high-frequency muscle-level control with low-frequency latent-space coordination in a hierarchical architecture. At the low level, general single-hand policies are trained via reinforcement learning to generate dynamic muscle-tendon activations while tracking trajectories from a large reference motion dataset. The resulting tracking policies are then distilled into variational autoencoder (VAE) models, yielding smooth and structured latent spaces that abstract away low-level muscle dynamics. For the high level, we train piece-specific policies to operate in this latent space, coordinating bimanual motions based on specific goals, denoted by note events extracted from given musical scores, to synthesize performances beyond the reference data. High-level control is formulated as a decentralized multiagent reinforcement learning problem combined with adversarial learning for motion imitation. In addition, we present an enhanced musculoskeletal hand model that supports fine control of fingers for accurate low-level motion tracking and diverse high-level motion synthesis. We evaluate the control pipeline of our approach on a diverse piano repertoire spanning multiple musical styles and technical demands. Results demonstrate that our approach can synthesize coordinated bimanual motions with accurate key presses, and achieve the state-of-the-art performance of piano playing in physics-based dexterous control, while generalizing to sheet music that is not presented in the reference dataset. We also show that our musculoskeletal hand model demonstrates superior biomechanical stability and tracking precision compared to the existing model, and validate that our musculoskeletal hand model and muscle-driven controller can generate physiologically plausible activation patterns that align with human electromyography (EMG) recordings when subjects perform multiple tasks.
论文检索
输入标题、作者或关键词，从 8,230 篇学术成果中精准定位
Retargeting human kinematic reference motion onto a robot's morphology remains a formidable challenge. Existing methods often produce physical inconsistencies, such as foot sliding, self-collisions, or dynamically infeasible motions, which hinder downstream imitation learning. We propose a bilevel optimization framework that jointly adapts reference motions to a robot's morphology while training a tracking policy using reinforcement learning. To make the optimization tractable, we derive an approximate gradient for the upper-level loss. Our framework requires only a sparse set of semantic rigid-body correspondences and eliminates the need for manual tuning by identifying optimal values for a parameterization expressive enough to preserve characteristic motion across different embodiments. Moreover, by integrating retargeting directly with physics simulation, we produce physically plausible motions that facilitate robust imitation learning. We validate our method in simulation and on hardware, demonstrating challenging motions for morphologies that differ significantly from a human, including retargeting onto a quadruped.
As instruction-based editing models and multimodal large language models continue to advance, a wide range of image editing tasks have become feasible. However, achieving precise and consistent geometric image editing, such as translating, scaling and rotating in 3D space, remains a major challenge. In this work, we introduce BoxCtrl, a 3D-aware visual prompting framework to address this problem. Unlike text-only or coarse 2D-guided approaches, our method introduces informative RGB 3D bounding boxes projected onto 2D images as a visual prompting. Each box is outlined and its three orthogonal faces are individually painted with different RGB colors, which simultaneously encode its position, size and orientation, providing a compact and intuitive in-context visual example. The key to BoxCtrl’s success lies in its well-designed bounding boxes, which serve as visual prompting that decouple geometric control from appearance control. This design enables the model to learn consistent correspondences between faces with the same color in the latent space, leading to a more precise understanding of geometric intention and more accurate editing result. We introduce a two-stage training paradigm comprising Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL). To address the scarcity of paired training data, we first construct a large-scale synthetic dataset for the SFT stage, which equips the model with fundamental editing capabilities. However, the generalization of the SFT model is constrained by the synthetic-to-real domain gap. To bridge this, we incorporate an online RL stage leveraging unpaired real-world data. Guided by a comprehensive reward function evaluating both geometric accuracy and visual fidelity, our SFT-RL strategy significantly enhances geometric precision while maintaining photorealistic quality. Extensive experiments demonstrate that BoxCtrl achieves state-of-the-art performance across translation, rotation, scaling, and composite editing tasks. Our code is available at https://github.com/beaglew/BoxCtrl.
Developing controllers capable of completing a wide range of tasks in a natural and life-like manner is a key challenge in enabling practical applications of physics-based character animation. In this work, we introduce Generative Pretrained Controllers (GPC), which leverage tokenization and next-token modeling to create general-purpose, reusable generative controllers from large-scale motion datasets. Our framework utilizes end-to-end reinforcement learning to jointly optimize a "motion vocabulary", modeled via Finite Scalar Quantization (FSQ), along with a corresponding control policy that can map the discrete codes to physics-based controls. After the "codebook" has been learned, the underlying structure of this large vocabulary is modeled by training a GPT-style autoregressive transformer, leading to a powerful generative controller that generates controls for a physically simulated character by performing next-token prediction. Once the generative controller has been trained, we propose a suite of adaptation techniques for finetuning the controller for new downstream tasks. Our proposed framework greatly simplifies the training process compared to previous tokenized methods, and achieves a 99.98% success rate in reproducing a vast corpus of motion clips. The generative controller exhibits a variety of natural emergent behaviors, such as responsive behaviors to perturbations and recovery behaviors after falling. This results in highly robust general purpose controllers for a variety of downstream applications.
Despite rapid advances in multimodal large language models, agricultural applications remain constrained by the lack of multilingual speech data, unified multimodal architectures, and comprehensive evaluation benchmarks. To address these challenges, we present AgriGPT-Omni, an agricultural omni-framework that integrates speech, vision, and text in a unified framework.(1) First, we construct a scalable data synthesis and collection pipeline that converts agricultural texts and images into training data, resulting in the largest agricultural speech dataset to date, including 492K synthetic and 1.4K real speech samples across six languages.(2) Second, based on this, we train the first agricultural Omni-model via a three-stage paradigm: textual knowledge injection, progressive multimodal alignment, and GRPO-based reinforcement learning, enabling unified reasoning across languages and modalities.(3) We further propose AgriBench-Omni-2K, the first tri-modal benchmark for agriculture, covering diverse speech–vision–text tasks and multilingual slices, with standardized protocols and reproducible tools. Experiments show that AgriGPT-Omni significantly outperforms general-purpose baselines on multilingual and multimodal reasoning as well as real-world speech understanding.
Large Language Models (LLMs) are now widely deployed across modern web services, but their safe and trustworthy use in real-world settings critically depends on accurate alignment with human preferences. Preference alignment is typically achieved using methods such as reinforcement learning or direct preference optimization (DPO), whose effectiveness in practice hinges on the quality of labeled preference data. However, a fundamental practical challenge remains: preference datasets inevitably contain noise. Through a systematic analysis of mainstream preference datasets, we find that roughly 25% of preference pairs show clear inconsistencies between reward-model evaluations and human annotations. Such inconsistent examples do not convey reliable preference signals; training directly on them therefore not only fails to improve alignment but can even degrade model behavior. To address this problem, we propose Noise-Aware Preference Alignment for LLMs via Confidence and Polarity Reweighting (AlignCP), a fully automated, human-free framework for noise-aware preference alignment. AlignCP derives two interpretable metrics from reward-model outputs: Confidence, which measures the reliability of each preference judgment, and Polarity, which evaluates whether the reward-model ranking agrees with the original human label. These metrics are combined to assign a training weight to each sample—amplifying high-confidence, label-consistent pairs while down-weighting or discarding low-confidence, contradictory, or otherwise noisy pairs. Unlike methods that rely on human re-inspection, repeated relabeling, or heavy computational reruns, AlignCP performs automated data-quality control with minimal overhead and no human intervention. Empirical results show that AlignCP substantially outperforms existing data-centric alignment approaches on standard preference benchmarks and remains more robust under noisy supervision.
Effective urban planning is crucial for enhancing residents' quality of life and ensuring societal stability, playing a pivotal role in the sustainable development of cities. Current planning methods heavily rely on human experts, which are time-consuming and labor-intensive, or utilize deep learning algorithms, often limiting stakeholder involvement. To bridge these gaps, we propose Intelli-Planner, a novel framework integrating Deep Reinforcement Learning (DRL) with large language models (LLMs) to facilitate participatory and customized planning scheme generation. Intelli-Planner utilizes demographic, geographic data, and planning preferences to determine high-level planning requirements and demands for each functional type. During training, a knowledge enhancement module is employed to enhance the decision-making capability of the policy network. Additionally, we establish a multi-dimensional evaluation system and employ LLM-based stakeholders for satisfaction scoring. Experimental validation across diverse urban settings shows that Intelli-Planner surpasses traditional baselines and achieves comparable performance to state-of-the-art DRL-based methods in objective metrics, while enhancing stakeholder satisfaction and convergence speed. These findings underscore the effectiveness and superiority of our framework, highlighting the potential for integrating the latest advancements in LLMs with DRL approaches to revolutionize tasks related to functional areas planning. Code and data are available at https://github.com/chicosirius/Intelli-Planner.
As web platforms evolve towards greater personalization and emotional complexity, conversational agents must transcend superficial empathy to demonstrate identity-aware emotional reasoning. However, existing systems face two limitations: (1) reliance on situation-centric datasets lacking persistent user identity, which hampers the capture of personalized affective nuances; and (2) dependence on opaque, coarse reward signals that hinder development of verifiable empathetic reasoning. To address these gaps, we introduce KardiaBench, a large-scale user-grounded benchmark comprising 178,080 QA pairs across 22,080 multi-turn conversations anchored to 671 real-world profiles. The dataset is constructed via a model-in-the-loop pipeline with iterative rubric-guided refinement to ensure psychological plausibility and persona consistency. This progressive empathy pipeline that integrates user comprehension, contextual reasoning, and emotion perception into conversations, followed by iterative critique and rubric-based refinement to ensure psychological plausibility, emotional fidelity, and persona consistency. Building on this, we propose Kardia-R1, a framework that trains models for interpretable, stepwise empathetic cognition. Kardia-R1 leverages Rubric-as-Judge Empathetic Reinforcement Learning (Rubric-ERL), a GRPO-based method that uses explainable, human-aligned rubric rewards to tightly couple user understanding, emotional inference, and supportive response generation. Extensive experiments across four LLM backbones demonstrate that Kardia-R1 consistently outperforms other methods in emotion accuracy, empathy, relevance, persona consistency, and safety.
The vision of an inclusive World Wide Web is impeded by a severe linguistic divide, particularly for communities in low-resource regions of Southeast Asia. While large language models (LLMs) offer a potential solution for translation, their deployment in data-poor contexts faces a dual challenge: the scarcity of high-quality, culturally relevant data and the prohibitive energy costs of training on massive, noisy web corpora. To resolve the tension between digital inclusion and environmental sustainability, we introduce Sustainable Agent-Guided Expert-tuning (SAGE). This framework pioneers an energy-aware paradigm that prioritizes the ''right data'' over ''big data''. Instead of carbon-intensive training on unfiltered datasets, SAGE employs a reinforcement learning (RL) agent, optimized via Group Relative Policy Optimization (GRPO), to autonomously curate a compact training set. The agent utilizes a semantic reward signal derived from a small, expert-constructed set of community dialogues to filter out noise and cultural misalignment. We then efficiently fine-tune open-source LLMs on this curated data using Low-Rank Adaptation (LoRA). We applied SAGE to translation tasks between English and seven low-resource languages (LRLs) in Southeast Asia. Our approach establishes new state-of-the-art performance on BLEU-4 and COMET-22 metrics, effectively capturing local linguistic nuances. Crucially, SAGE surpasses baselines trained on full datasets while reducing data usage by 97.1% and training energy consumption by 95.2%. By delivering high-performance models with a minimal environmental footprint, SAGE offers a scalable and responsible pathway to bridge the digital divide in the Global South.
In the rapidly evolving web-based financial ecosystem where digital banking services become critical infrastructure for underserved communities, credit card fraud disproportionately affects vulnerable populations relying on financial platforms. However, previous studies overlook extreme data scarcity conditions, particularly at small-to-medium web banks that serve as crucial gateways for vulnerable communities. This paper addresses the fundamental challenge of building inclusive and secure financial systems operable at true web scale. To overcome this deficiency, we propose a novel web-based fairness-aware federated fraud detection model, CLARF, which utilizes the designed privacy-enhanced representation fusion and fraud-aware contrastive learning modules to enhance detection performance under conditions of data scarcity and label imbalance. Furthermore, current federated fraud detection systems critically neglect vulnerability to backdoor attacks, where malicious actors can implant hidden triggers during model aggregation, compromising system integrity. We propose a novel dynamic web Min-Max adversarial game framework where attackers employ hybrid multi-stage reinforcement learning with multi-dimensional reward mechanisms to dynamically evolve triggers that achieve excellent tradeoff between stealthiness and effectiveness. Defender adapts a closed-loop Selection-Evaluation-Suppression framework where high-reliability clients are selected via Fisher information to carry out reverse trigger engineering. Then clients' confidence scores are calculated as weights to minimize Attack Success Rate (ASR) during aggregation. Extensive experiments on six financial fraud datasets demonstrate the superiority of CLARF model and Min-Max adversarial game paradigm compared with multiple SOTA models.
We present ReSuMe, a general framework for mutual enhancement of dense retrieval systems and document summarizers through reinforcement learning. The framework jointly optimizes a language model for generating retrieval-oriented summaries and adapts the retrieval model to these summaries through alternating fine-tuning phases. We employ Group Relative Policy Optimization (GRPO) to fine-tune the language model based on retrieval relevance rather than linguistic quality alone, while the retrieval model is iteratively updated using contrastive learning on the generated summaries. This co-optimization process addresses the fundamental distribution shift problem that arises when retrieval models trained on full documents must operate on synthetic summaries during inference. By progressively reducing this distribution gap, our framework yields two key benefits: improved retrieval performance and a high-quality document summarizer optimized for retrieval tasks. We demonstrate our framework using Contriever on the MS-MARCO dataset, achieving consistent improvements of 13.2% in MRR@10 and 6.7% in Recall@100 over the baseline. The framework is model-agnostic and can be applied to enhance any dense retrieval system while simultaneously producing an effective document summarization model.
This study investigates the problem of keeping information up-to-date when crawling data sources that change over time (e.g., websites or location data). Traditional crawling methods often treat data sources independently, making it difficult to capture relationships and propagate updates efficiently. We propose using graph structures to model these relationships and show that, unfortunately, finding the theoretically optimal solution can be intractable. To address this, we introduce a specific graphical model (the latent Bernoulli process model) and demonstrate the complexity of even simple tasks within this framework. We tackle the crawling problem using a reinforcement learning-based algorithm and demonstrate its superiority over traditional baselines on both real and synthetic data. This work highlights the power of graph-structured crawling in helping users stay informed within a dynamic information landscape.
Optimizing numerical systems and mechanism design is crucial for enhancing player experience in Massively Multiplayer Online (MMO) games. Traditional optimization approaches rely on iterative online experiments or parameter tuning over abstracted statistical models, which can be inaccurate, time-consuming and potentially impair players' experience. Although simplified offline simulation systems are frequently employed as alternatives, their low fidelity constrains agents' ability to faithfully replicate real players' reasoning processes and behavioral responses to interventions. To address these limitations, we propose a generative agent-based MMO simulation system with hundreds of agents empowered by Large Language Models (LLMs). By applying Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) on large-scale real player behavioral data, we adapt LLMs from general priors to game-specific domains, enabling realistic and interpretable player decision-making. In parallel, a data-driven environment model trained on real gameplay logs reconstructs dynamic in-game systems. Experiments demonstrate strong consistency with real-world player behaviors and plausible causal responses under interventions, providing a reliable, interpretable, and cost-efficient framework for data-driven numerical design optimization.
Large Language Models (LLMs) have shown significant potential for improving recommendation systems through their inherent reasoning capabilities and extensive knowledge base. Yet, existing studies predominantly address warm-start scenarios with abundant user-item interaction data, leaving the more challenging cold-start scenarios, where sparse interactions hinder traditional collaborative filtering methods, underexplored. To address this limitation, we propose novel reasoning strategies designed for cold-start item recommendations within the Netflix domain. Our method utilizes the advanced reasoning capabilities of LLMs to effectively infer user preferences, particularly for newly introduced or rarely interacted items. We systematically evaluate supervised fine-tuning, reinforcement learning-based fine-tuning, and hybrid approaches that combine both methods to optimize recommendation performance. Extensive experiments on real-world data demonstrate significant improvements in both methodological efficacy and practical performance in cold-start recommendation contexts. Remarkably, our reasoning-based fine-tuned models outperform Netflix's production ranking model by up to 8% in certain cases.
Balancing multiple objectives is critical for user satisfaction in modern recommender and search systems, yet current Multi-Task Fusion (MTF) methods rely on static, manually-tuned weights that fail to capture individual user intent. While Reinforcement Learning (RL) offers a path to personalization, traditional approaches often falter due to training instability and the sparse rewards inherent in these large-scale systems. To address these limitations, we propose Group-Relative Reinforcement Learning with Adaptive Dirichlet Exploration (GRADE), a novel and robust framework for personalized multi-task fusion. GRADE leverages a critic-free, Group Relative Policy Optimization (GRPO) paradigm, enabling stable and efficient policy learning by evaluating the relative performance of candidate weight groups. Its core innovations include employing the Dirichlet distribution for principled and structured exploration of the weight space, and a composite reward function that combines sparse user feedback with dense model priors and rule-based constraints to guide the search effectively. Deployed in the in-app marketplace of an application with over hundreds of millions daily active users, GRADE significantly outperforms established baselines, achieving substantial gains in rigorous large-scale A/B tests: +0.595% in CTR, +1.193% in CVR, +1.788% in OPM, and +1.568% in total order volume. Following its strong performance, GRADE has been fully deployed in the marketplace search scenario of Kuaishou, serving hundreds of millions of users.
Kuaishou serving hundreds of millions of searches daily, the quality of short-video search is paramount. However, it suffers from a severe Matthew effect on long-tail queries: sparse user behavior data causes models to amplify low-quality content such as clickbait and shallow content. The recent advancements in Large Language Models (LLMs) offer a new paradigm, as their inherent world knowledge provides a powerful mechanism to assess content quality, agnostic to sparse user interactions. To this end, we propose a LLM-driven multimodal reranking framework, which estimates user experience without real user behavior. The approach involves a two-stage training process: the first stage uses multimodal evidence to construct high-quality annotations for supervised fine-tuning, while the second stage incorporates pairwise preference optimization to help the model learn partial orderings among candidates. At inference time, the resulting experience scores are used to promote high-quality but underexposed videos in reranking, and further guide page-level optimization through reinforcement learning. Experiments show that the proposed method achieves consistent improvements over strong baselines in offline metrics including AUC, NDCG@K, and human preference judgement. An online A/B test covering 15% of traffic further demonstrates gains in both user experience and consumption metrics, confirming the practical value of the approach in long-tail video search scenarios.
In modern e-commerce search systems, dense retrieval has become an indispensable component. By computing similarities between query and item (product) embeddings, it efficiently selects candidate products from large-scale repositories. With the breakthroughs in large language models (LLMs), mainstream embedding models have gradually shifted from BERT to LLMs for more accurate text modeling. However, these models still adopt direct-embedding methods, and the semantic accuracy of embeddings remains inadequate. Therefore, contrastive learning is heavily employed to achieve tight semantic alignment between positive pairs. Consequently, such models tend to capture statistical co-occurrence patterns in the training data, biasing them toward shallow lexical and semantic matches. For difficult queries exhibiting notable lexical disparity from target items, the performance degrades significantly. In this work, we propose the Large Reasoning Embedding Model (LREM), which novelly integrates reasoning processes into representation learning. For difficult queries, LREM first conducts reasoning to achieve a deep understanding of the original query, and then produces a reasoning-augmented query embedding for retrieval. This reasoning process effectively bridges the semantic gap between original queries and target items, significantly improving retrieval accuracy. Specifically, we adopt a two-stage training process: the first stage optimizes the LLM on carefully curated Query-CoT-Item triplets with SFT and InfoNCE losses to establish preliminary reasoning and embedding capabilities, and the second stage further refines the reasoning trajectories via reinforcement learning (RL). Extensive offline and online experiments validate the effectiveness of LREM, leading to its deployment on China's largest e-commerce platform since August 2025.
Query-product relevance prediction is fundamental to e-commerce search and has become even more critical in the era of AI-powered shopping, where semantic understanding and complex reasoning directly shape the user experience and exert an indirect, yet substantial impact on business conversion. Large Language Models (LLMs) enable generative, reasoning-based approaches, typically aligned via supervised fine-tuning (SFT) or preference optimization methods like Direct Preference Optimization (DPO). However, the increasing complexity of business rules and user queries exposes the inability of existing methods to endow models with robust reasoning capacity for long-tail and challenging cases. Efforts to address this via reinforcement learning strategies like Group Relative Policy Optimization (GRPO) often suffer from sparse terminal rewards, offering insufficient guidance for multi-step reasoning, which in turn slows convergence. To address these challenges, we propose TaoSR-AGRL, an Adaptive Guided Reinforcement Learning framework for LLM-based relevance prediction in Taobao Search Relevance. TaoSR-AGRL introduces two key innovations: (1) Rule-aware Reward Shaping, which decomposes the final relevance judgment into dense, structured rewards aligned with domain-specific relevance criteria; and (2) Adaptive Guided Replay, which identifies low-accuracy rollouts during training and injects targeted ground-truth guidance to steer the policy away from stagnant, rule-violating reasoning patterns toward compliant trajectories. TaoSR-AGRL was evaluated on large-scale datasets and through online evaluations on Taobao Search. It consistently outperforms DPO and GRPO baselines, improving relevance accuracy and rule adherence, with measurable gains in user engagement and stable training dynamics. The model has been deployed on Taobao, serving hundreds of millions of users.
Traditional search engines return uniform results for identical queries, overlooking users' personalized intents. While personalized search has been extensively studied, research on query rewriting for personalized intents has been constrained by traditional approaches like statistical co-occurrence and synonym expansion. Current work primarily addresses multi-turn dialogue scenarios in Conversational AI rather than exploring applications in large-scale search engines. This typically stems from the absence of real search scenario data and the difficulty of inferring users' intents through personalized reasoning. While Large Language Models (LLMs) combined with Chain-of-Thought (CoT) capabilities provide possibilities for personalized reasoning, CoT introduces additional reasoning overhead that is difficult to accept in online scenarios requiring low latency. To address this, this paper proposes PicQue (Personalized Efficient Query Rewrite), a personalized query rewriting model training pipeline aimed at achieving high accuracy with low latency. PicQue contains a two-stage training that first employs a novel Hybrid Supervised Fine-Tuning strategy to retain the model's reasoning capabilities while allowing the decoding process to skip CoT, thereby obtaining CoT's accuracy gains without increasing latency. Building on this foundation, PicQue conducts second-stage reinforcement learning using Group Relative Policy Optimization (GRPO) to further improve rewriting accuracy and reduce the risk of over-rewriting. We also propose a Guided Search strategy to optimize GRPO training, alleviating the reduction in training sample utilization when all sampling rollouts are wrong. Extensive offline and online experiments demonstrate PicQue's effectiveness. In offline metrics, PicQue achieves over 7% improvement in rewriting accuracy compared to baseline methods and compresses up to 95% of decoding tokens. In online A/B tests, user satisfaction increases by 1.78% and query change rate decreases by 0.71%, achieving significant online gains.