Experiments are informative but should be used judiciously as a costly resource. Well-constructed theory may serve as a substitute. We develop a ''Theory Instead of Experiment'' (TIE) framework and, in collaboration with Tencent, apply the framework to assess how much value (e.g., user clicks) each creator contributes to its WeChat Official Accounts Platform. This TIE application models content demand and supply upon the counterfactual departure of a creator. The demand model predicts user clicks based on estimated user preferences, while the supply model captures the platform's content distribution response. Together, they predict how each creator influences user engagement through the platform's content distribution strategy. We test the predictions of the TIE system with 168 experiments, each examining a different mix of creators and involving more than 9 million unique users. The TIE system and the experiments demonstrate a 97% correlation on the key performance metric (change in user clicks). Based on its low costs, high accuracy, granular output, and minimal latency, Tencent has deployed the TIE system as the default approach to creator valuation, assessing tens of millions of creators each day while avoiding a 2.5% user click loss associated with a typical experiment.
论文检索
输入标题、作者或关键词，从 844 篇学术成果中精准定位
A long-standing challenge for pushing sensor-based human activity recognition (HAR) to industrial usage is the distribution shift between training data and testing data: significant variations in data distribution lead to a notable decline in performance. Recently, Large Language Models (LLMs) have demonstrated exceptional generalization capability, which provides a new opportunity to mitigate the distribution shift problem of HAR. However, since LLMs are inherently designed and trained on textual data, their potential to enhance generalization in HAR applications remains an open question. In this paper, we introduce LLM4HAR, a novel LLM-based model to improve cross-domain HAR. LLM4HAR consists of three main modules: (i) the Sensor Data Adaptation module, which aligns IMU signals with LLMs via sensor embedding(ii) the Sensor Knowledge Learning module, which injects sensor knowledge into LLMs for activity recognition, and (iii) the Efficiency Enhancement module, which employs a partial training strategy and reduces the model size by more than 10 times. Extensive evaluations show that LLM4HAR outperforms the existing methods by 13.82% in average F1 score, demonstrating the feasibility and effectiveness of transferring knowledge from pretrained LLMs to enhance HAR. Further, LLM4HAR has been adopted by JD Logistics to support downstream applications such as Courier Welfare Improvement and Map Data Generation.
Enhancing Learned Knowledge in LoRA Adapters Through Efficient Contrastive Decoding on Ascend NPUs
PDF ↗Huawei Cloud users leverage LoRA (Low-Rank Adaptation) as an efficient and scalable method to fine-tune and customize large language models (LLMs) for application-specific needs. However, tasks that require complex reasoning or deep contextual understanding are often hindered by biases or interference from the base model when using typical decoding methods like greedy or beam search. These biases can lead to generic or task-agnostic responses from the base model instead of leveraging the LoRA-specific adaptations. In this paper, we introduce Contrastive LoRA Decoding (CoLD), a novel decoding framework designed to maximize the use of task-specific knowledge in LoRA-adapted models, resulting in better downstream performance. CoLD utilizes contrastive decoding by scoring candidate tokens based on the divergence between the probability distributions of a LoRA-adapted expert model and the corresponding base model. This approach prioritizes tokens that better align with the LoRA's learned representations, enhancing performance for specialized tasks. Despite its effectiveness, a naive implementation of CoLD can be computationally expensive. Each decoding step requires evaluating multiple token candidates across both the base and expert models, creating significant inefficiencies. To address this challenge, we developed an optimized kernel for Huawei's Ascend NPU. CoLD demonstrates consistent performance improvements, achieving up to a 5.54% increase in task accuracy while reducing end-to-end latency by 28% compared to greedy decoding. This work offers valuable insights into practical and efficient decoding strategies for fine-tuned LLMs, particularly in resource-constrained environments, and has broad implications for applied data science in both cloud and on-premises settings.
Optimizing Case-Based Reasoning System for Functional Test Script Generation with Large Language Models
PDF ↗In this work, we explore the potential of large language models (LLMs) for generating functional test scripts, which necessitates understanding the dynamically evolving code structure of the target software. To achieve this, we propose a case-based reasoning (CBR) system utilizing a 4R cycle (i.e., retrieve, reuse, revise, and retain), which maintains and leverages a case bank of test intent descriptions and corresponding test scripts to facilitate LLMs for test script generation. To improve user experience further, we introduce Re4, an optimization method for the CBR system, comprising reranking-based retrieval finetuning and reinforced reuse finetuning. Specifically, we first identify positive examples with high semantic and script similarity, providing reliable pseudo-labels for finetuning the retriever model without costly labeling. Then, we apply supervised finetuning, followed by a reinforcement learning finetuning stage, to align LLMs with our production scenarios, ensuring the faithful reuse of retrieved cases. Extensive experimental results on two product development units from Huawei Datacom demonstrate the superiority of the proposed CBR+Re4. Notably, we also show that the proposed Re4 method can help alleviate the repetitive generation issues with LLMs.
Enhancing Text-Based Hierarchical Multilabel Classification for Mobile Applications via Contrastive Learning
PDF ↗A hierarchical labeling system for mobile applications (apps) benefits a wide range of downstream businesses that integrate the labeling with their proprietary user data, to improve user modeling. Such a label hierarchy can define more granular labels that capture detailed app features beyond the limitations of traditional broad app categories. In this paper, we address the problem of hierarchical multilabel classification for apps by using their textual information such as names and descriptions. We present: 1) HMCN (Hierarchical Multilabel Classification Network) for handling the classification from two perspectives: the first focuses on a multilabel classification without hierarchical constraints, while the second predicts labels sequentially at each hierarchical level considering such constraints; 2) HMCL (Hierarchical Multilabel Contrastive Learning), a scheme that is capable of learning more distinguishable app representations to enhance the performance of HMCN. Empirical results on our Tencent App Store dataset and two public datasets demonstrate that our approach performs well compared with state-of-the-art methods. The approach has been deployed at Tencent and the multilabel classification outputs for apps have helped a downstream task--credit risk management of users--improve its performance by 10.70% with regard to the Kolmogorov-Smirnov metric, for over one year.
In recent years, the recommendation content on e-commerce platforms has become increasingly rich -- a single user feed may contain multiple entities, such as selling products, short videos, and content posts. To deal with the multi-entity cross-domain recommendation problem, an intuitive solution is to adopt the shared-network-based architecture for joint training. The underlying idea is to transfer the knowledge from one type of entity (source entity) to another (target entity). However, different from the conventional same-entity cross-domain recommendation, multi-entity knowledge transfer encounters several important challenges: (1) data distributions of the source entity and target entity are naturally different, making the shared-network-based joint training susceptible to the negative transfer issue(2) the corresponding feature schema of each entity is not exactly aligned (e.g., price is an essential feature for selling product while missing for content posts), making the existing approaches no longer appropriate. Recent researchers have also experimented with the pre-training and fine-tuning paradigm. Again, they only take into account the scenarios with the same entity type and feature schemas, which is inappropriate in our case. To this end, we design a pre-training & fine-tuning based Multi-entity Knowledge Transfer framework called MKT. MKT utilizes a multi-entity pre-training module to extract transferable knowledge across different entities. In particular, a feature alignment module is first applied to scale and align different feature schemas. Afterward, a couple of knowledge extractors are employed to extract the common and independent knowledge. In the end, the extracted common knowledge is adopted for target entity model training. Through extensive offline and online experiments on public and industrial datasets, we demonstrated the superiority of MKT over multiple State-Of-The-Art methods. MKT has also been deployed for content post recommendations on our production system.
Retrieval-Augmented Generation (RAG) grounds large language models in external evidence, yet it still falters when answers must be pieced together across semantically distant documents. We close this gap with the Hierarchical Lexical Graph (HLG), a three-tier index that (i) traces every atomic proposition to its source(ii) clusters propositions into latent topics, and (iii) links entities and relations to expose cross-document paths. On top of HLG we build two complementary, plug-and-play retrievers: StatementGraphRAG, which performs fine-grained entity-aware beam search over propositions for high-precision factoid questions, and TopicGraphRAG, which selects coarse topics before expanding along entity links to supply broad yet relevant context for exploratory queries. Additionally, existing benchmarks lack the complexity required to rigorously evaluate multi-hop summarization systems, often focusing on single-document queries or limited datasets. To address this, we introduce a synthetic dataset generation pipeline that curates realistic, multi-document question-answer pairs, enabling robust evaluation of multi-hop retrieval systems. Extensive experiments across five datasets demonstrate that our methods outperform naive chunk-based RAG, achieving an average relative improvement of 23.1% in retrieval recall and correctness. Open-source Python library is available at https://github.com/awslabs/graphrag-toolkit.
Defending against today's increasingly sophisticated and large-scale cyberattacks demands accurate, real-time threat intelligence. Traditional approaches struggle to scale, integrate diverse telemetry, and adapt to a constantly evolving security landscape. We introduce Threat Intelligence Tracking via Adaptive Networks (Titan), an industry-scale graph mining framework that generates cyber threat intelligence at unprecedented speed and scale. Titan introduces a suite of innovations specifically designed to address the complexities of the modern security landscape, including: (1) a dynamic threat intelligence graph that maps the intricate relationships between millions of entities, incidents, and organizations; (2) real-time update mechanisms that automatically decay and prune outdated intel; (3) integration of security domain knowledge to bootstrap initial reputation scores; and (4) reputation propagation algorithms that uncover hidden threat actor infrastructure. Integrated into Microsoft Unified Security Operations Platform (USOP), which is deployed across hundreds of thousands of organizations worldwide, Titan's threat intelligence powers key detection and disruption capabilities. With an impressive average macro-F1 score of 0.89 and a precision-recall AUC of 0.94, Titan identifies millions of high-risk entities each week, enabling a 6x increase in non-file threat intelligence. Since its deployment, Titan has increased the product's incident disruption rate by a remarkable 21%, while reducing the time to disrupt by a factor of 1.9x, and maintaining 99% precision, as confirmed by customer feedback and thorough manual evaluation by security experts--ultimately saving customers from costly security breaches.
Complicated Semantic Alignment for Long-Tail Query Rewriting in Taobao Search Based on Large Language Model
PDF ↗In the realm of e-commerce search, semantic matching has consistently been a core issue, as it directly affects user experience and company revenue. However, users' queries often fail to effectively retrieve relevant products due to discrepancies between the user's expression habits and product names written by merchants. Even existing large language model (LLM) based query rewriting methods can bridge the semantic gap for most queries, they are still ineffective for long-tail queries with complicated semantic. In this paper, we propose Complicated Semantic Alignment Query Rewrite(CSA-QR) framework, which mitigates the semantic differences in long-tail queries with complicated semantics. CSA-QR comprises three stages: high-quality supervised fine-tuning (SFT) dataset generation, multi-dimensional alignment dataset generation, and binary feedback Proximal Policy Optimization (PPO) for reinforcement alignment. Initially, we utilize general large language models to generate rewrite candidates, followed by manual annotation to discriminate the candidates, then use the retrieval augmentation generation (RAG) based on existing annotations to produce a higher quality SFT dataset. Subsequently, we decouple the feedback data into user semantic consistency and merchant expression consistency dimensions to collect multi-dimensional alignment data. Finally, we introduce a binary feedback method to train the reward model, enabling it to better guide alignment training within our context. We also identify a set of more appropriate reward model evaluation metrics to guide our iterations. Offline experiments demonstrate the effectiveness of this method in improving retrieval performance. Online A/B tests reveal that our method significantly boosts critical metrics such as product click-through rate (CTR), gross merchandise volume (GMV) and number of transaction (#Trans) for long-tail complicated queries. CSA-QR has been deployed on Taobao, one of China's most popular online shopping platforms, since September 2024.
Customer expansion, i.e., growing a business's existing customer base by acquiring new customers, is critical for scaling operations and sustaining the long-term profitability of logistics companies. Although state-of-the-art works model this task as a single-node classification problem under a heterogeneous graph learning framework and achieve good performance, they struggle with extremely positive label sparsity issues in our scenario. Multi-task learning (MTL) offers a promising solution by introducing a correlated, label-rich task to enhance the label-sparse task prediction through knowledge sharing. However, existing MTL methods result in performance degradation because they fail to discriminate task-shared and task-specific structural patterns across tasks. This issue arises from their limited consideration of the inherently complex structure learning process of heterogeneous graph neural networks, which involves the multi-layer aggregation of multi-type relations. To address the challenge, we propose a Structure-Aware Hierarchical Information Sharing Framework (SrucHIS), which explicitly regulates structural information sharing across tasks in logistics customer expansion. SrucHIS breaks down the structure learning phase into multiple stages and introduces sharing mechanisms at each stage, effectively mitigating the influence of task-specific structural patterns during each stage. We evaluate StrucHIS on both private and public datasets, achieving a 51.41% average precision improvement on the private dataset and a 10.52% macro F1 gain on the public dataset. StrucHIS is further deployed at one of the largest logistics companies in China and demonstrates a 41.67% improvement in the success contract-signing rate over existing strategies, generating over 453K new orders within just two months.
High Click-Through Rate (CTR) imagery has proven commercial value for food delivery platforms, driving a need for strategies to generate visually compelling images. Our investigations reveal a positive correlation between appropriate food backgrounds and subsequent user engagement. Despite advancements in diffusion models, inpainting new backgrounds does not guarantee high CTR, and fine-tuning diffusion models for this purpose is prohibitively expensive for the fast-paced online food delivery advertising sector. Consequently, there is a lack of cost-effective, transferable generation frameworks tailored to high-CTR food images. In this paper, we propose FoRAGe, a novel high-CTR Food image Retrieval-Augmented Generation pipeline leveraging ControlNet based on Stable Diffusion. Specifically, we construct a comprehensive food image database encompassing a diverse range of background environments. During image generation, FoRAGe retrieves high-quality background exemplars featuring analogous food subjects from the database and employs the retrieved backgrounds as conditions to guide image synthesis via the ControlNet model. Subsequently, a multimodal CTR prediction model is utilized to identify and select optimal images for deployment. Extensive online experiments demonstrate a significant increase in CTR for images generated by our proposed pipeline, and ablation studies further elucidate the impact of different strategies and configurations. Code is available at https://github.com/jiaxu-feng/FoRAGe.
The internet offers a massive repository of unstructured information, but it's a significant challenge to convert this into a structured format. At Pinterest, the ability to accurately extract structured product data from e-commerce websites is essential to enhance user experiences and improve content distribution. In this paper, we present Pinterest's system for attribute extraction, which achieves remarkable accuracy and scalability at a manageable cost. Our approach leverages a novel webpage representation that combines structural, visual, and text modalities into a compact form, optimizing it for small model learning. This representation captures each visible HTML node with its text, style and layout information. We show how this allows simple models such as eXtreme Gradient Boosting (XGBoost) to extract attributes more accurately than much more complex Large Language Models (LLMs) such as Generative Pre-trained Transformer (GPT). Our results demonstrate a system that is highly scalable, processing over 1,000 URLs per second, while being 1000 times more cost-effective than the cheapest GPT alternatives.
Synergizing Implicit and Explicit User Interests: A Multi-Embedding Retrieval Framework at Pinterest
PDF ↗Industrial recommendation systems are typically composed of multiple stages, including retrieval, ranking, and blending. The retrieval stage plays a critical role in generating a high-recall set of candidate items that covers a wide range of diverse user interests. Effectively covering the diverse and long-tail user interests within this stage poses a significant challenge: traditional two-tower models struggle in this regard due to limited user-item feature interaction and often bias towards top use cases. To address these issues, we propose a novel multi-embedding retrieval framework designed to enhance user interest representation by generating multiple user embeddings conditioned on both implicit and explicit user interests. Implicit interests are captured from user history through a Differentiable Clustering Module (DCM), whereas explicit interests, such as topics that the user has followed, are modeled via Conditional Retrieval (CR). These methodologies represent a form of conditioned user representation learning that involves condition representation construction and associating the target item with the relevant conditions. Synergizing implicit and explicit user interests serves as a complementary approach to achieve more effective and comprehensive candidate retrieval as they benefit on different user segments and extract conditions from different but supplementary sources. Extensive experiments and A/B testing reveal significant improvements in user engagements and feed diversity metrics. Our proposed framework has been successfully deployed on Pinterest home feed.
COEF-VQ: Cost-Efficient Video Quality Understanding through a Cascaded Multimodal LLM Framework
PDF ↗Recently, with the emergence of recent Multimodal Large Language Model (MLLM) technology, it has become possible to exploit its video understanding capability on different classification tasks. In practice, we face the difficulty of huge requirements for GPU resource if we need to deploy MLLMs online. In this paper, we propose COEF-VQ, a novel cascaded MLLM framework designed to enhance video quality understanding on the short-video platform while optimizing computational efficiency. Our approach integrates an entropy-based pre-filtering stage, where a lightweight model assesses uncertainty and selectively filters cases before passing them to the more computationally intensive MLLM for final evaluation. By prioritizing high-uncertainty samples for deeper analysis, our framework significantly reduces GPU usage while maintaining the strong classification performance of a full MLLM deployment. To demonstrate the effectiveness of COEF-VQ, we deploy this new framework onto the video management platform (VMP) at the short-video platform, and perform a series of detailed experiments on two in-house tasks related to video quality understanding. We show that COEF-VQ leads to substantial performance gains from the offline evaluation in these two tasks and effectively enhances platform safety with limit resource consumption, significantly reducing inappropriate content video view rate by 9.9% in a online A/B test without affecting engagement. Post-launch monitoring confirmed sustained improvements, validating its real-world impact.
Miscreants use domain names for malicious purposes such as phishing websites or fake webshops. Reactive approaches such as blocklists play an important role in fighting such abuse but have limitations, namely that the domains are typically only included in such a list after abuse has been reported (e.g., there may already be some victims). We propose RegCheck, a system designed to proactively flag suspicious domains at registration time. The core of RegCheck is a machine learning classifier that assesses the risk that the domain name will be used for malicious purposes based on characteristics known at the time of registration. Based on this assessment, it flags some registrations and requires them to undergo additional verification prior to the domain name being activated. The system has been developed collaboratively between SIDN (.nl) and DNS Belgium (.be) and has been deployed as a real-time system at the .be registry since March 2024. Since its deployment, the registry has witnessed a decrease in the number of .be domain name registrations that have been revoked for breaching the terms and conditions, indicating a decline in the number of active malicious registrations.
Technology firms conduct randomized controlled experiments (''A/B tests'') to learn which actions to take to improve business outcomes. In firms with mature experimentation platforms, experimentation programs can consist of many thousands of tests. To effectively scale experimentation, firms rely on decision rules: standard operating procedures for mapping the results of an experiment to a choice of treatment arm to launch to the general user population. Despite the critical role of decision rules in translating experimentation into business decisions, rigorous guidance on how to evaluate and choose decision rules is scarce. This paper proposes to evaluate decision rules based on their cumulative returns to business north star metrics. Although intuitive and easy to explain to decision-makers, this quantity can be difficult to estimate, especially when experiments have weak signal-to-noise ratios. We develop a cross-validation estimator that is much less biased than the naive plug-in estimator under conditions realistic to digital experimentation. We demonstrate the efficacy of our approach via a case study of 123 historical A/B tests at Netflix, where we used it to show that a new decision rule would have increased cumulative returns to the north star metric by an estimated 33%, directly leading to the adoption of the new rule.
Satellite images play a crucial role in weather analytics, and recent advancements in satellite technology have significantly enhanced the accuracy and reliability of weather predictions. In this paper, we introduce SkySearch, a large-scale satellite video search system deployed at the Korean Meteorological Administration (KMA). SkySearch is designed to aid weather experts in making timely and accurate forecasts by rapidly and precisely searching for satellite videos in the database that resemble current weather conditions. SkySearch employs self-supervised learning to compress large volumes of high-resolution satellite videos into low-dimensional embeddings, addressing the absence of labeled satellite data. Within this latent space, relationships between videos are modeled as a graph, enabling efficient searches. Given a query video, SkySearch rapidly identifies a small subset of similar videos by traversing the graph. When the query video represents the current conditions, it can optionally be augmented with predicted future frames to search for videos that reflect both the current conditions and expected evolution. Finally, the ranked list of videos is provided to weather forecasters through a user-friendly interface. We demonstrate that deployment and empirical effectiveness of SkySearch through both numerical and qualitative evaluations. In summary, SkySearch is: (a) Scalable: processes queries from a large-scale database of satellite images spanning over a decade and delivers results within seconds(b) Accurate: returns numerically and qualitatively similar videos to the query video, and (c) Label-free: does not require labeled videos.
The proliferation of complex non-factoid questions in modern information seeking (IS) systems exposes critical limitations in conventional Retrieval-Augmented Generation (RAG) approaches, particularly their static search strategies and the lack of systematic multi-source information integration capabilities. Facing these limitations, we present PASS (Proactive Agent-driven Search System), a novel multi-agent framework that operationalizes human-like proactive search strategies through five specialized agents: Revealer for intent analysis, Navigator for search planning, Seeker/Reader for adaptive retrieval, and Writer for response synthesis, systematically expanding the search space through iterative query refinement and multi-perspective knowledge integration. Crucially, our framework demonstrates remarkable adaptability to mid-sized LLMs, demonstrating its scalability in resource-constrained environments. To comprehensively assess the effectiveness of the proposed framework, we carry out extensive experiments on both mid-sized and proprietary large-scale LLMs, evaluating response quality for complex non-factoid questions using a newly introduced nugget-based assessment. Experimental results from offline nugget-based evaluation and online A/B Tests confirm substantial improvements in answer quality, advancing proactive information seeking methodologies and offering practical pathways for democratizing complex reasoning capabilities to resource-constrained environments.
EMD-Period: Detecting Multi-periodicity in Industrial Cloud Clusters via Time-Frequency Decomposition
PDF ↗Accurate periodicity detection of performance metrics in cloud platform is essential for enhancing monitoring accuracy and ensure service quality of cloud computing. However, in real-world industrial settings, such as large-scale cloud computing platform MaxCompute at Alibaba Cloud, performance metrics often exhibit complex multi-periodicity and temporal nonstationarity. Additionally, these metrics are frequently contaminated by noise and anomalies, rendering traditional periodicity detection methods ineffective. To address these challenges, this paper proposes a robust and general period detection method with low deployment cost and high usability. The method employs Empirical Mode Decomposition (EMD) to decompose complex time series into Intrinsic Mode Functions (IMFs), isolating noise, periodic, and trend components. Significant periods are identified using statistical tests to eliminate irrelevant components, while an automatic clustering mechanism mitigates the mode-mixing problem inherent in EMD. Extensive experiments on synthetic and real-world datasets demonstrate that the proposed method outperforms state-of-the-art periodicity detection techniques, achieving over 10% and 17% performance improvements in public single and multiple periodicity detection tasks, respectively. Deployed in Alibaba Cloud's MaxCompute platform, it monitors millions of tasks, resulting in a 7.8% increase in fault detection coverage and enabling faults to be detected 13.15% more rapidly. The method's low deployment cost and user-friendly implementation make it highly accessible and practical for diverse cloud monitoring applications.
The LifeTime Value (LTV) prediction, which endeavors to forecast the cumulative purchase contribution of a user to a particular item, remains a vital challenge that advertisers are keen to resolve. A precise LTV prediction system enhances the alignment of user interests with meticulously designed advertisements, thereby generating substantial profits for advertisers. Nonetheless, this issue is complicated by the paucity of data typically observed in real-world advertising scenarios. The purchase rate among registered users is often as critically low as 0.1%, resulting in a dataset where the majority of users make only several purchases. Consequently, there is insufficient supervisory signal for effectively training the LTV prediction model. An additional challenge emerges from the interdependencies among tasks with high correlation. It is a common practice to estimate a user's contribution to a game over a specified temporal interval. Varying the lengths of these intervals corresponds to distinct predictive tasks, which are highly correlated. For instance, predictions over a 7-day period are heavily reliant on forecasts made over a 3-day period, where exceptional cases can adversely affect the accuracy of both tasks. In order to comprehensively address the aforementioned challenges, we introduce an innovative framework denoted as Graph-Represented Pareto-Optimal LifeTime Value prediction (GRePO-LTV). Graph representation learning is initially employed to address the issue of data scarcity. Subsequently, Pareto-Optimization is utilized to manage the interdependence of prediction tasks. Our method is evaluated using a proprietary offline mini-game recommendation dataset in conjunction with an online A/B test. The implementation of our method results in a significant enhancement within the offline dataset. Moreover, the A/B test demonstrates encouraging outcomes, increasing average Gross Merchandise Value (GMV) by 8.4%.