Fairness-aware graph learning has gained increasing attention in recent years. Nevertheless, there lacks a comprehensive benchmark to evaluate and compare different fairness-aware graph learning methods, which blocks practitioners from choosing appropriate ones for broader real-world applications. In this paper, we present an extensive benchmark on ten representative fairness-aware graph learning methods. Specifically, we design a systematic evaluation protocol and conduct experiments on seven real-world datasets to evaluate these methods from multiple perspectives, including group fairness, individual fairness, the balance between different fairness criteria, and computational efficiency. Our in-depth analysis reveals key insights into the strengths and limitations of existing methods. Additionally, we provide practical guidance for applying fairness-aware graph learning methods in applications. To the best of our knowledge, this work serves as an initial step towards comprehensively understanding representative fairness-aware graph learning methods to facilitate future advancements in this area. Open-source code can be found at: https://github.com/yushundong/Fairness-Aware-Graph-Learning-Benchmark.
论文检索
输入标题、作者或关键词，从 844 篇学术成果中精准定位
WikiRAG: Revisiting Wikidata KGC Datasets with Community Updates and Retrieval-Augmented Generation
PDF ↗Link prediction is an important task for knowledge graph completion and curation, and it has received significant attention from the research community. However, researchers often train and evaluate new models on small or outdated datasets that do not reflect the current state of knowledge, thereby disregarding new information and the rich textual content often linked to knowledge graphs. As a result, many opportunities to leverage these dimensions are missed. We introduce WikiRAG, a framework for knowledge completion and evaluation derived from Wikidata and Wikipedia, which enables research integrating retrieval techniques and large language models. Our framework combines the following contributions: (i) We revisit the Wikidata5M dataset by updating it to reflect the current state of Wikidata and providing automated tools for its periodic maintenance. (ii) We enrich the dataset with long-form textual content sourced from Wikipedia, enabling research that goes beyond traditional graph structures and shallow text methods toward dense retrieval techniques. (iii) We propose a simple yet effective baseline that leverages retrieval-augmented generation, demonstrating the utility of the dataset and integrating language model capabilities for link prediction. The revised dataset, coined Wikidata5M-RE, shows that the original graph grew by roughly 50% in the number of edges, while 10% of the edges have been removed. A comparative analysis of classic methods demonstrates that these changes can impact downstream task evaluation. Finally, our evaluation of WikiRAG's KGC method shows an improvement of up to 9% in link prediction accuracy over state-of-the-art baselines, setting the stage for a new avenue in knowledge completion that uses deep information extraction. The source code, data, and other artifacts have been made available on the project website: https://github.com/colab-nyuad/WikiRAG
The analysis of the privacy properties of Privacy-Preserving Ads APIs is an area of research that has received strong interest from academics, industry, and regulators. Despite this interest, the empirical study of these methods is severely hindered by the lack of publicly available data. Reliable empirical analysis of the privacy properties of an API, in fact, requires access to a dataset consisting of realistic API outputs for a large collection of users; however, privacy concerns prevent the general release of such data to the public. In this work, we address this problem by developing a novel methodology to construct synthetic API outputs that are simultaneously realistic enough to enable accurate study and provide strong privacy protections. We focus on one of the Privacy-Preserving Ads APIs: the Topics API, part of Google Chrome's Privacy Sandbox, which enables interest-based advertising without relying on third-party cookies. We developed a methodology to generate a differentially-private dataset that closely matches the re-identification risk properties of the real Topics API data. The use of differential privacy provides strong theoretical bounds on the leakage of private user information from this release. Our methodology is based on first computing a large number of differentially-private statistics describing how output API traces evolve over time. Then, we design a parameterized distribution over sequences of API traces and optimize its parameters so that they closely match the statistics obtained. Finally, we create the synthetic data by drawing from this distribution. Our work is complemented by an open-source release of the anonymized dataset obtained by this methodology. We hope this will enable external researchers to analyze the API in-depth and replicate prior and future work on a realistic large-scale dataset. We believe that this work will contribute to fostering transparency regarding the privacy properties of Privacy-Preserving Ads APIs.
We introduce MentalChat16K, an English benchmark dataset combining a synthetic mental health counseling dataset and a dataset of anonymized transcripts from interventions between Behavioral Health Coaches and Caregivers of patients in palliative or hospice care. Covering a diverse range of conditions like depression, anxiety, and grief, this curated dataset is designed to facilitate the development and evaluation of large language models for conversational mental health assistance. By providing a high-quality resource tailored to this critical domain, MentalChat16K aims to advance research on empathetic, personalized AI solutions to improve access to mental health support services. The dataset prioritizes patient privacy, ethical considerations, and responsible data usage. MentalChat16K presents a valuable opportunity for the research community to innovate AI technologies that can positively impact mental well-being. The dataset is available at https://huggingface.co/datasets/ShenLab/MentalChat16K and the code and documentation are hosted on GitHub at https://github.com/PennShenLab/MentalChat16K.
In this paper, we introduce a comprehensive benchmark for learning-based monocular depth estimation methods. In recent years, depthmap calculation methods have achieved impressive results. The process of comparing the performance of different models requires a lot of resources and time, which can be costly for many developers. The goal of our benchmark is to provide a comparison of how the top models perform on various datasets. We present a table with results based on our tests performed on several popular datasets and one dataset introduced in this paper. In addition, we provide a toolkit that allows each model selected for the benchmark to be tested on any suitable dataset.
The rapid rise of online social networks underscores the need to understand the heterogeneous strengths of online relationships. Yet, efforts to assess tie strength (TS) are hindered by the lack of ground-truth labels, differing research perspectives, and limited model performance in real-world settings. To address this gap, we introduce BTS, a comprehensive Benchmark for Tie Strength prediction, aiming to establish a standardized foundation for evaluating and advancing TS prediction methodologies. Specifically, our contributions are: TS Pseudo-Label Techniques -- we categorize TS into seven standardized pseudo-labeling techniques based on prior literature; TS Dataset Collection -- we present a representative collection of three social networks and perform data analysis by investigating the class distributions and correlations across the generated pseudo-labels; TS Pseudo-Label Evaluation Framework -- we propose a standardized framework to evaluate the pseudo-label quality from the perspective of tie resilience; Benchmarking -- we evaluate existing tie strength prediction model performance using the BTS dataset collection, exploring the effects of different experiment settings, models, and evaluation criteria on the results. Furthermore, we derive key insights to enhance existing methods and shed light on promising directions for future research in this domain. The BTS dataset collection, along with the curation codes and experimental scripts, is all available at: https://github.com/XueqiC/Awesome-Tie-Strength-Prediction.
Metamaterials, engineered materials with architected structures across multiple length scales, offer unprecedented and tunable mechanical properties that surpass those of conventional materials. However, leveraging advanced machine learning (ML) for metamaterial discovery is hindered by three fundamental challenges: (C1) Data Heterogeneity Challenge arises from heterogeneous data sources, heterogeneous composition scales, and heterogeneous structure categories; (C2) Model Complexity Challenge stems from the intricate geometric constraints of ML models, which complicate their adaptation to metamaterial structures; and (C3) Human-AI Collaboration Challenge comes from the “dual black-box” nature of sophisticated ML models and the need for intuitive user interfaces. To tackle these challenges, we introduce a unified framework, named MetamatBench, that operates on three levels. (1) At the data level, we integrate and standardize 5 heterogeneous, multi-modal metamaterial datasets. (2) The ML level provides a comprehensive toolkit that adapts 17 state-of-the-art ML methods for metamaterial discovery. It also includes a comprehensive evaluation suite with 12 novel performance metrics with finite element-based assessments to ensure accurate and reliable model validation. (3) The user level features a visual-interactive interface that bridges the gap between complex ML techniques and non-ML researchers, advancing property prediction and inverse design of metamaterials for research and applications. MetamatBench offers a unified platform that enables machine learning researchers and practitioners to develop and evaluate new methodologies in metamaterial discovery. For accessibility and reproducibility, we open-source our benchmark and the code-base at https://github.com/cjpcool/Metamaterial-Benchmark.
ClimateIQA: A New Dataset and Benchmark to Advance Vision-Language Models in Meteorology Anomalies Analysis
PDF ↗Meteorological heatmaps play a vital role in deciphering extreme weather phenomena, yet their inherent complexities-marked by irregular contours, unstructured patterns, and complex color variations-present unique analytical hurdles for state-of-the-art Vision-Language Models (VLMs). Current state-of-the-art models like GPT-4o, Qwen-VL, and LLaVA 1.6 struggle with tasks such as precise color identification and spatial localization, resulting in inaccurate or incomplete interpretations. To address these challenges, we introduce Sparse Position and Outline Tracking (SPOT), a novel algorithm specifically designed to process irregularly shaped colored regions in visual data. SPOT identifies and localizes these regions by extracting their spatial coordinates, enabling structured representations of irregular shapes. Building on SPOT, we construct ClimateIQA, a novel meteorological visual question answering (VQA) dataset, comprising 26,280 high-resolution heatmaps and 762,120 instruction samples for wind gust, total precipitation, wind chill index and heat index analysis. ClimateIQA enhances VLM training by incorporating spatial cues, geographic metadata, and reanalysis data, improving model accuracy in interpreting and describing extreme weather features. Furthermore, we develop Climate-Zoo, a suite of fine-tuned VLMs based on SPOT-empowered ClimateIQA, which significantly outperforms existing models in meteorological heatmap tasks.
With the increasing use of Large Language Models (LLMs) in fields such as e-commerce, domain-specific concept evaluation benchmarks are crucial for assessing their domain capabilities. Existing LLMs may generate factually incorrect information within the complex e-commerce applications. Therefore, it is necessary to build an e-commerce concept benchmark. Existing benchmarks encounter two primary challenges: (1) handle the heterogeneous and diverse nature of tasks(2) distinguish between generality and specificity within the e-commerce field. To address these problems, we propose ChineseEcomQA, a scalable question-answering benchmark focused on fundamental e-commerce concepts. ChineseEcomQA is built on three core characteristics: Focus on Fundamental Concept, E-commerce Generality and E-commerce Expertise. Fundamental concepts are designed to be applicable across a diverse array of e-commerce tasks, thus addressing the challenge of heterogeneity and diversity. Additionally, by carefully balancing generality and specificity, ChineseEcomQA effectively differentiates between broad e-commerce concepts, allowing for precise validation of domain capabilities. We achieve this through a scalable benchmark construction process that combines LLM validation, Retrieval-Augmented Generation (RAG) validation, and rigorous manual annotation. Based on ChineseEcomQA, we conduct extensive evaluations on mainstream LLMs and provide some valuable insights. We hope that ChineseEcomQA could guide future domain-specific evaluations, and facilitate broader LLM adoption in e-commerce applications.
Similarity search lies at the heart of many modern applications, ranging from databases to deep learning to data series analysis. As such, a vast effort has been invested in developing algorithms, data structures and implementations to speed up this crucial subroutine. To empirically validate these approaches, several benchmarking efforts have been initiated covering a wide array of datasets. In this paper, we observe that usually little control is exercised on the hardness of the workloads with which methods are tested and compared. To address this issue, we first evaluate several query hardness measures with respect to their ability to capture the empirical hardness of a query, i.e. the effort invested by an index data structure to provide an answer. Then, we propose two methods, deemed Hephaestus-Annealing and Hephaestus-Gradient, for synthesizing query workloads so that they meet a user-specified hardness target. Both methods allow to produce workloads with the desired hardness: we find that Hephaestus-Gradient is faster, while Hephaestus-Annealing makes fewer assumptions on the target hardness measure. The resulting workloads can be used to gain insights into the behavior of similarity search algorithms.
The emergence of foundation models with high generalization performance increases expectations for their application in AI Contact Centers (AICC) services. However, research on AICC services based on foundation models remains insufficient, particularly in the context of non-English languages. To bridge this gap, we construct a Korean phone call-based customer service dataset designed to evaluate the feasibility and fairness of foundation model-based AICC systems. Our utterance-level audio dataset incorporates diverse speaker attributes such as accent, gender, and age, ensuring comprehensive assessments across various demographic factors. Furthermore, we provide dialect-aware scripts to enhance linguistic diversity and realism in AICC agent evaluation. Building upon this dataset, we propose a customized prompt engineering approach to optimize foundation models for AICC services. Our system prompt improves the natural flow of conversation while ensuring accurate information generation by LLMs. In addition, we address the limitations of conventional human-based evaluations, which are costly and susceptible to bias, by introducing an automatic hierarchical evaluation metric. This metric leverages LLMs to systematically assess AI agent responses, considering both overall conversation quality and factual accuracy. Finally, using our fine-grained Korean speech dataset, we conduct an experimental fairness evaluation of AI agents in AICC services. We also demonstrate that our proposed evaluation metric exhibits a higher correlation with human judgments compared to previous predominant metrics. Our work contributes to the development of more reliable, fair, and effective AICC services by providing a high-quality dataset and robust evaluation framework. The code and materials are available at https://anonymous.4open.science/r/fomo4aicc-EC45.
Combinatorial optimization (CO) is fundamental to several real-world applications, from logistics and scheduling to hardware design and resource allocation. Deep reinforcement learning (RL) has recently shown significant benefits in solving CO problems, reducing reliance on domain expertise and improving computational efficiency. However, the absence of a unified benchmarking framework leads to inconsistent evaluations, limits reproducibility, and increases engineering overhead, raising barriers to adoption for new researchers. To address these challenges, we introduce RL4CO, a unified and extensive benchmark with in-depth library coverage of 27 CO problem environments and 23 state-of-the-art baselines. Built on efficient software libraries and best practices in implementation, RL4CO features modularized implementation and flexible configurations of diverse environments, policy architectures, RL algorithms, and utilities with extensive documentation. RL4CO helps researchers build on existing successes while exploring and developing their own designs, facilitating the entire research process by decoupling science from heavy engineering. We finally provide extensive benchmark studies to inspire new insights and future work. RL4CO has already attracted numerous researchers in the community and is open-sourced at https://github.com/ai4co/rl4co.
Given the dominance of dense retrievers which do not generalize well beyond their training dataset distributions, domain-specific test sets are essential in evaluating retrieval performance. Few test datasets are available for retrieval systems intended for use by healthcare providers in a point-of-care setting. To fill this gap we have collaborated with medical professionals to create CURE, a test dataset for passage retrieval composed of 2,000 expert written queries spanning 10 medical domains with a monolingual (English) and two cross-lingual (French/Spanish to English) conditions. In this paper, we describe how CURE was constructed and provide baseline results to showcase its effectiveness as an evaluation tool. CURE is published with a CC BY-NC 4.0 license and can be accessed on https://huggingface.co/datasets/clinia/CUREv Hugging Face and as a retrieval task on fhttps://github.com/embeddings-benchmark/mteb MTEB.
Bridging the Binary Analysis Gap: A Cross-Compiler Dataset and Neural Framework for Industrial Control Systems
PDF ↗Industrial Control Systems (ICS) rely heavily on Programmable Logic Controllers (PLCs) to manage critical infrastructure, yet analyzing PLC executables remains challenging due to diverse proprietary compilers and limited access to source code.To bridge this gap, we introduce PLC-BEAD, a comprehensive dataset containing 2431 compiled binaries from 700+ PLC programs across four major industrial compilers (CoDeSys, GEB, OpenPLC-V2, OpenPLC-V3). This novel dataset uniquely pairs each binary with its original Structured Text source code and standardized functionality labels, enabling both binary-level and source-level analysis. We demonstrate the dataset's utility through PLCEmbed, a transformer-based framework for binary code analysis that achieves 93% accuracy in compiler provenance identification and 42% accuracy in fine-grained functionality classification across 22 industrial control categories. Through comprehensive ablation studies, we analyze how compiler optimization levels, code patterns, and class distributions influence model performance. We provide detailed documentation of the dataset creation process, labeling taxonomy, and benchmark protocols to ensure reproducibility. Both PLC-BEAD and PLCEmbed are released as open-source resources to foster research in PLC security, reverse engineering, and ICS forensics, establishing new baselines for data-driven approaches to industrial cybersecurity.
This paper presents TTGL, a large-scale, multi-scenario universal graph learning framework developed at TikTok, which generates unified embeddings that capture diverse user interests across multiple contexts. TTGraph handles over 100 billion edges and 10 billion nodes daily, linking user behaviors from various scenarios such as video, search, live streaming and e-commerce. TTGL showcases a versatile strategy designed for handling large amounts of data and practical limitations, highlighting the balance between efficiency and precision in a high-volume production setting. TTGL utilizes a heterogeneous graph to unify user behaviors across different TikTok scenarios, dynamically learning shared embeddings that capture complex data patterns. It implements advanced domain-adaptive graph sampling and unsupervised training methods to handle its scale while ensuring fair representation. The graph embeddings are fed into online serving pipelines, delivering real-time recommendations that effectively balance efficiency and revenue generation. These effective graph online serving solutions are highly versatile, demonstrating their impact not only in TikTok's Feed video recommendations but also across various business scenarios such as e-commerce and live streaming. Through extensive offline evaluations and online A/B testing, we demonstrate that TTGL significantly improves key metrics across multiple contexts, achieving a 0.35% increase in 7-day retention, a 3.27% rise in watch live duration, a 1.32% boost in gross merchandise value, a 1.07% increase in paid order count, and a 1.0% improvement in search page views.
ADMIRE: ADaptive method to enhance Multiple Image REsolutions in text-rich multi-image understanding
Recent advancements in multi-image understanding have been significant, yet methods that treat all images equally face challenges when handling dense visual OCR information and ultra-long sequences. This results in difficulty focusing on the relevant pages, leading to redundancy and limited effectiveness. To address these issues, we propose an effcient and effective method that enhances multi-image document understanding by dynamically adjusting image resolutions based on attention from large visual language models. This approach enhances both performance and efficiency without modifying the underlying model architecture. Firstly, we propose Text-Guided Key Image Selector to focus on important images. Secondly, we sort visual information by importance score, adaptively increasing the tokens for important images while compressing or dropping the tokens for less important or irrelevant visual inputs. Experimental results across four open datasets and one closed industry dataset, using two mainstream models, demonstrate the efficacy of our approach. We achieve state-of-the-art performance for OCR-free methods, with scores of 82.78 on MP-DocVQA and 56.05 on DUDE (a 7.42-point improvement), using the Qwen2vl-7B model. Despite a slight increase in inference time, our model still shows notable improvements of 4.78 points on MP-DocVQA and 3.28 points on DUDE. Our code will be publicly available for noncommercial use at https://github.com/Alipay-Med/admire.git
Recent advances in graph machine learning (ML) with the introduction of Graph Neural Networks (GNNs) have led to a widespread interest in applying these approaches to business applications at scale. GNNs enable differentiable end-to-end (E2E) learning of model parameters given graph structure which enables optimization towards popular node, edge (link) and graph-level tasks. While the research innovation in new GNN layers and training strategies has been rapid, industrial adoption and utility of GNNs has lagged considerably due to the unique scale challenges that large-scale graph ML problems create. In this work, we share our approach to training, inference, and utilization of GNNs at Snapchat. To this end, we present GiGL (Gigantic Graph Learning), an open-source library to enable large-scale distributed graph ML to the benefit of researchers, ML engineers, and practitioners. We use GiGL internally at Snapchat to manage the heavy lifting of GNN workflows, including graph data preprocessing from relational DBs, subgraph sampling, distributed training, inference, and orchestration. GiGL is designed to interface cleanly with open-source GNN modeling libraries prominent in academia like PyTorch Geometric (PyG), while handling scaling and productionization challenges that make it easier for internal practitioners to focus on modeling. GiGL is used in multiple production settings, and has powered over 35 launches across multiple business domains in the last 2 years in the contexts of friend recommendation, content recommendation and advertising. This work details high-level design and tools the library provides, scaling properties, case studies in diverse business settings with large-scale graphs up to hundreds of millions of nodes, tens of billions of edges, and hundreds of node and edge features, and several key lessons learned in employing graph ML at scale on large social data. GiGL is open-sourced at https://github.com/Snapchat/GiGL.
Metro passenger flow prediction is crucial for effective urban transportation management. However, its practical adoption is hindered by data silos from distributed automatic fare collection (AFC) systems, compromising prediction accuracy. While federated graph learning facilitates privacy-preserving collaboration, existing methods struggle with the unique challenges of cross-line metro passenger flow prediction, particularly in handling time-evolving spatial correlations and heterogeneous temporal correlations. To address these challenges, we present FedMetro, a novel metro passenger flow prediction system based on federated graph learning. We introduce a federated dynamic graph learning approach with cross-attention mechanisms to capture spatial-temporal correlations in passenger flow. Additionally, we propose a dynamic mask-based communication compression method to mitigate communication bottlenecks in federated inference. Extensive evaluations on three real-world metro AFC datasets demonstrate that FedMetro significantly outperforms baseline methods, achieving up to 17.08% higher accuracy while reducing federated inference communication overhead by 77.99%. Practical deployments further confirm its effectiveness in delivering accurate station-level predictions across metro lines. Our code is available at https://github.com/AlexMufeng/FedMetro.
Evaluation plays a crucial role in the development of ranking algorithms on search and recommender systems. It enables online platforms to create user-friendly features that drive commercial success in a steady and effective manner. The online environment is particularly conducive to applying causal inference techniques, such as randomized controlled experiments (known as A/B test), which are often more challenging to implement in fields like medicine and public policy. However, businesses face unique challenges when it comes to effective A/B test. Specifically, achieving sufficient statistical power for conversion-based metrics can be time-consuming, especially for significant purchases like booking accommodations. While offline evaluations are quicker and more cost-effective, they often lack accuracy and are inadequate for selecting candidates for A/B test. To address these challenges, we developed interleaving and counterfactual evaluation methods to facilitate rapid online assessments for identifying the most promising candidates for A/B tests. Our approach not only increased the sensitivity of experiments by a factor of up to 100 (depending on the approach and metrics) compared to traditional A/B testing but also streamlined the experimental process. The practical insights gained from usage in production can also benefit organizations with similar interests.
Optimizing Recall or Relevance? A Multi-Task Multi-Head Approach for Item-to-Item Retrieval in Recommendation
PDF ↗The task of item-to-item (I2I) retrieval is to identify a set of relevant and highly engaging items based on a given item. I2I retrieval is a crucial component in modern recommendation systems, where users' previously engaged items serve as trigger items to retrieve relevant content for future engagement. However, existing I2I models in industry are primarily built on co-engagement data and optimized using the recall measure, which overly emphasizes co-engagement patterns while failing to capture semantic relevance. This often leads to overfitting short-term co-engagement trends at the expense of long-term benefits such as discovering novel interests and promoting content diversity. To address this challenge, we propose MTMH, a Multi-Task and Multi-Head I2I retrieval model that achieves both high recall and semantic relevance. Our model consists of two key components: 1) a multi-task learning loss for formally optimizing the trade-off between recall and relevance, and 2) a multi-head I2I retrieval architecture for retrieving both highly co-engaged and semantically relevant items. We evaluate MTMH using proprietary data from a commercial platform serving billions of users and demonstrate that it can improve recall by up to 14.4% and semantic relevance by up to 56.6% compared with prior state-of-the-art models. We also conduct live experiments to verify that MTMH can enhance both short-term consumption metrics and long-term user-experience-related metrics. Our work provides a principled approach for jointly optimizing I2I recall and semantic relevance, which has significant implications for improving the overall performance of recommendation systems.