Path planning is a critical challenge for Autonomous Underwater Vehicles (AUVs) due to complex underwater environments, including ocean currents, dynamic obstacles, and limited sensing capabilities. The lack of a standardized benchmarking framework has hindered direct comparisons between algorithms, slowing progress in the field. To address this, we introduce an open-source benchmarking platform for underwater AUV path planning, designed to provide a unified evaluation environment, automated performance assessment, and reproducible experiments. Built on the HoloOcean simulation platform, our benchmark incorporates realistic underwater dynamics, such as ocean currents, static and dynamic obstacles, and sensor models. It supports a range of path planning tasks, from basic obstacle avoidance to complex scenarios with current disturbances. The platform is compatible with classical algorithms (e.g., A*, RRT), evolutionary methods (e.g., GA, ACO), and deep reinforcement learning (e.g., Soft Actor-Critic, SAC). We define key evaluation metrics, including path efficiency (length, smoothness, energy consumption), task success rate, collision rate, and computational cost. Automated tools enable systematic algorithm comparisons across scenarios, generating standardized performance results and visualizations. This open-source, extensible framework aims to advance underwater path planning research by enabling fair comparisons and guiding future algorithm development. It provides a scalable foundation for evaluating AUV path planning methods under simulated real-world conditions, fostering innovation in the field. All source code and experimental configurations will be available on the GitHub Page: https://github.com/IoET-y/UP-bench.
论文检索
输入标题、作者或关键词，从 844 篇学术成果中精准定位
When Graph Meets Multimodal: Benchmarking and Meditating on Multimodal Attributed Graph Learning
PDF ↗Multimodal Attributed Graphs (MAGs) are ubiquitous in real-world applications, encompassing extensive knowledge through multimodal attributes attached to nodes (e.g., texts and images) and topological structure representing node interactions. Despite its potential to advance diverse research fields like social networks and e-commerce, MAG representation learning (MAGRL) remains underexplored due to the lack of standardized datasets and evaluation frameworks. In this paper, we first propose MAGB, a comprehensive MAG benchmark dataset, featuring curated graphs from various domains with both textual and visual attributes. Based on the MAGB dataset, we further systematically evaluate two mainstream MAGRL paradigms: GNN-as-Predictor, which integrates multimodal attributes via Graph Neural Networks (GNNs), and VLM-as-Predictor, which harnesses Vision Language Models (VLMs) for zero-shot reasoning. Extensive experiments on MAGB reveal the following critical insights: (i) Modality significances fluctuate drastically with specific domain characteristics. (ii) Multimodal embeddings can elevate the performance ceiling of GNNs. However, intrinsic biases among modalities may impede effective training, particularly in low-data scenarios. (iii) VLMs are highly effective at generating multimodal embeddings that alleviate the imbalance between textual and visual attributes. These discoveries, which illuminate the synergy between multimodal attributes and graph topologies, contribute to reliable benchmarks, paving the way for future research.
Despite efforts to improve the quality of service (QoS) and throughput in Large Language Model (LLM) serving systems, progress is often limited by the lack of publicly available real-world workloads. Consequently, evaluations usually depend on synthetic or oversimplified load patterns, and systems that appear promising in testing frequently underperform once deployed. This work presents BurstGPT, an LLM serving workload with 10.31 million traces from regional Azure OpenAI GPT services over 213 days. BurstGPT captures LLM serving characteristics from user, model and system perspectives: (1) User request concurrency: burstiness variations of requests in Azure OpenAI GPT services, revealing diversified concurrency patterns in different services and model types. (2) User conversation patterns: counts and intervals within conversations for service optimizations. (3) Model response lengths: auto-regressive serving processes of GPT models, showing statistical relations between requests and their responses. (4) System response failures: failures of conversation and API services, showing intensive resource needs and limited availability of LLM services in Azure. The details of the characteristics can serve multiple purposes in LLM serving optimizations, such as system evaluation and trace provisioning. In our demo evaluation with BurstGPT, frequent variations in BurstGPT reveal declines in efficiency, stability, or reliability in realistic LLM serving. We identify that the generalization of KV cache management, scheduling and disaggregation optimizations can be improved under realistic workload evaluations. BurstGPT is publicly available now at https://github.com/HPMLL/BurstGPT and is widely used to develop prototypes of LLM serving frameworks in the industry.
Living environments play a vital role in the prevalence and progression of diseases, and understanding their impact on patient's health status becomes increasingly crucial for developing AI models. However, due to the lack of long-term and fine-grained spatial and temporal data in public and population health studies, most existing studies fail to incorporate environmental data, limiting the models' performance and real-world application. To address this shortage, we developed SatHealth, a novel dataset combining multimodal spatiotemporal data, including environmental data, satellite images, all-disease prevalences estimated from medical claims, and social determinants of health (SDoH) indicators. We conducted experiments under two use cases with SatHealth: regional public health modeling and personal disease risk prediction. Experimental results show that living environmental information can significantly improve AI models' performance and temporal-spatial generalizability on various tasks. Finally, we deploy a web-based application to provide an exploration tool for SatHealth and one-click access to both our data and regional environmental embedding to facilitate plug-and-play utilization. SatHealth is now published with data in Ohio, and we will keep updating SatHealth to cover the other parts of the US. With the web application and published code pipeline, our work provides valuable angles and resources to include environmental data in healthcare research and establishes a foundational framework for future research in environmental health informatics.
Traditional evaluations of conversational AI have primarily focused on ''closed-ended'' interactions, where a human user queries the AI system, such as in customer support. However, many advanced real-world applications-such as job interviews, podcast hosting, and legal or healthcare intake discussions-require ''open-ended'' interactions in which the AI must take initiative by formulating questions to fully understand the human user's story. As AI assumes broader roles that demand greater autonomy, evaluating its performance in open-ended conversations becomes significantly more complex. This paper introduces a novel framework for rigorously assessing AI agents in such open-ended interviews. In this framework, a secondary AI agent (Agent B) simulates a human interviewee by strictly following a structured script that defines topic strengths and weaknesses, along with guidelines for when to hint at or deviate from these topics. Meanwhile, the primary AI agent (Agent A) engages in dynamic questioning to uncover the script's underlying facts and narrative. By comparing the final conversation transcript to the original script, we assess Agent A using metrics such as completeness, consistency, and investigative depth. This approach not only establishes a new benchmark for open-ended conversational skills but also provides insight into how effectively an AI agent can detect and navigate strategic diversions in scripted behavior.
Misinformation is a complex societal issue, and mitigating solutions are difficult to create due to data deficiencies. To address this, we have curated the largest collection of (mis)information datasets in the literature, totaling 75. From these, we evaluated the quality of 36 datasets that consist of statements or claims, as well as the 9 datasets that consist of data in purely paragraph form. We assess these datasets to identify those with solid foundations for empirical work and those with flaws that could result in misleading and non-generalizable results, such as spurious correlations, or examples that are ambiguous or otherwise impossible to assess for veracity. We find the latter issue is particularly severe and affects most datasets in the literature. We further provide state-of-the-art baselines on all these datasets, but show that regardless of label quality, categorical labels may no longer give an accurate evaluation of detection model performance. Finally, we propose and highlight Evaluation Quality Assurance (EQA) as a tool to guide the field toward systemic solutions rather than inadvertently propagating issues in evaluation. Overall, this guide aims to provide a roadmap for higher quality data and better grounded evaluations, ultimately improving research in misinformation detection. All datasets and other artifacts are available at misinfo-datasets.complexdatalab.com. The extended paper, including the appendices, can be accessed via arXiv at arxiv.org/abs/2411.05060.
Battery Life Prediction (BLP), which relies on time series data produced by battery degradation tests, is crucial for battery utilization, optimization, and production. Despite impressive advancements, this research area faces three key challenges. Firstly, the limited size of existing datasets impedes insights into modern battery life data. Secondly, most datasets are restricted to small-capacity lithium-ion batteries tested under a narrow range of diversity in labs, raising concerns about the generalizability of findings. Thirdly, inconsistent and limited benchmarks across studies obscure the effectiveness of baselines and leave it unclear if models popular in other time series fields are effective for BLP. To address these challenges, we propose BatteryLife, a comprehensive dataset and benchmark for BLP. BatteryLife integrates 16 datasets, offering a 2.5 times sample size compared to the previous largest dataset, and provides the most diverse battery life resource with batteries from 8 formats, 59 chemical systems, 9 operating temperatures, and 421 charge/discharge protocols, including both laboratory and industrial tests. Notably, BatteryLife is the first to release battery life datasets of zinc-ion batteries, sodium-ion batteries, and industry-tested large-capacity lithium-ion batteries. With the comprehensive dataset, we revisit the effectiveness of baselines popular in this and other time series fields. Furthermore, we propose CyclePatch, a plug-in technique that can be employed in various neural networks. Extensive benchmarking of 18 methods reveals that models popular in other time series fields can be unsuitable for BLP, and CyclePatch consistently improves model performance establishing state-of-the-art benchmarks. Moreover, BatteryLife evaluates model performance across aging conditions and domains. BatteryLife is available at https://github.com/Ruifeng-Tan/BatteryLife.
Knowledge Graph Completion (KGC) attempts to predict missing facts in a Knowledge Graph (KG). Recently, there's been an increased focus on designing KGC methods that can excel in the inductive setting, where a portion or all of the entities and relations seen in inference are unobserved during training. Numerous benchmark datasets have been proposed for inductive KGC, all of which are subsets of existing KGs used for transductive KGC. However, we find that the current procedure for constructing inductive KGC datasets inadvertently creates a shortcut that can be exploited even while disregarding the relational information. Specifically, we observe that the Personalized PageRank (PPR) score can achieve strong or near SOTA performance on most datasets. In this paper, we study the root cause of this problem. Using these insights, we propose an alternative strategy for constructing inductive KGC datasets that helps mitigate the PPR shortcut. We then benchmark multiple popular methods using the newly constructed datasets and analyze their performance. The new benchmark datasets help promote a better understanding of the capabilities and challenges of inductive KGC by removing any shortcuts that obfuscate performance. The code and dataset and can be found https://github.com/HarryShomer/Better-Inductive-KGC.
State-of-the-art link prediction (LP) models demonstrate impressive benchmark results. However, popular benchmark datasets often assume that training, validation, and testing samples are represen- tative of the overall dataset distribution. In real-world situations, this assumption is often incorrect; uncontrolled factors lead new dataset samples to come from a different distribution than train- ing samples. Additionally, the majority of recent work with graph dataset shift focuses on node- and graph-level tasks, largely ignor- ing link-level tasks. To bridge this gap, we introduce a novel splitting strategy, known as LPShift, which utilizes structural properties to induce a controlled distribution shift. We verify LPShift's effect through empirical evaluation of SOTA LP models on 16 LPShift variants of original dataset splits, with results indicating drastic changes to model performance. Additional experiments demon- strate graph structure has a strong influence on the success of current generalization methods. The source code is available here https://github.com/revolins/LPShift.
In recent years, the rapid advancement of Artificial Intelligence (AI) technologies, particularly Large Language Models (LLMs), has revolutionized the paradigm of scientific discovery, establishing AI-for-Science (AI4Science) as a dynamic and evolving field. However, there is still a lack of an effective framework for the overall assessment of AI4Science, particularly from a holistic perspective on data quality and model capability. Therefore, in this study, we propose SciHorizon, a comprehensive assessment framework designed to benchmark the readiness of AI4Science from both scientific data and LLM perspectives. First, we introduce a generalizable framework for assessing AI-ready scientific data, encompassing four key dimensions-Quality, FAIRness, Explainability, and Compliance-which are subdivided into 15 sub-dimensions. Drawing on data resource papers published between 2018 and 2023 in peer-reviewed journals, we present recommendation lists of AI-ready datasets for Earth, Life, and Materials Sciences, making a novel and original contribution to the field. Concurrently, to assess the capabilities of LLMs across multiple scientific disciplines, we establish 16 assessment dimensions based on five core indicators-Knowledge, Understanding, Reasoning, Multimodality, and Values-spanning Mathematics, Physics, Chemistry, Life Sciences, and Earth and Space Sciences. Using the developed benchmark datasets, we have conducted a comprehensive evaluation of over 50 representative open-source and closed-source LLMs. All the results are publicly available and can be accessed online at www.scihorizon.cn/en.
Evaluating generative foundation models on open-ended multimodal understanding (MMU) and generation (MMG) tasks across diverse modalities (e.g., images, audio, video) poses significant challenges due to the complexity of cross-modal interactions. To this end, the concept of utilizing Multimodal LLMs (MLLMs) as automated judges has emerged, yielding encouraging results in assessing vision-language understanding tasks. Building upon this, this paper extends the MLLM-as-a-Judge paradigm to unify evaluation across modalities. We introduce TaskAnything and JudgeAnything, to respectively evaluate the overall performance and judging capabilities of MLLMs across any-to-any modality tasks. Specifically, TaskAnything evaluates the MMU and MMG capabilities across 15 any-to-any modality categories, employing 1,500 queries curated from well-established benchmarks. Furthermore, JudgeAnything evaluates the judging capabilities of 8 advanced models (e.g., Gemini-1.5-Pro and Qwen2.5-Omni series) from the perspectives of Pair Comparison and Score Evaluation, providing a standardized testbed that incorporates human judgments and detailed rubrics. Our extensive experiments reveal that while these MLLMs show promise in assessing MMU (i.e., achieving an average of 66.55% under the Pair Comparison setting and 42.79% under the Score Evaluation setting), they encounter significant challenges with MMG tasks (i.e., averaging only 53.37% under the Pair Comparison setting and 30.05% under the Score Evaluation setting), exposing cross-modality biases and hallucination issues. To address this, we present OmniArena, an automated platform for evaluating omni-models and multimodal reward models. Our work highlights the need for fairer evaluation protocols and stronger alignment with human preferences. The source code and dataset are publicly available at: https://urrealhero.github.io/judgeanythingweb/.
Event Sequences (EvS ) refer to sequential data characterized by irregular sampling intervals and a mix of categorical and numerical features. Accurate classification of these sequences is crucial for various real-life applications, including healthcare, finance, and user interaction. Despite the popularity of the EvS classification task, there is currently no standardized benchmark or rigorous evaluation protocol. This lack of standardization makes it difficult to compare results across studies, which can result in unreliable conclusions and hinder progress in the field. To address this gap, we present EBES, a comprehensive benchmark for EvS classification with sequence-level targets. EBES features standardized evaluation scenarios and protocols, along with an open-source PyTorch library. Code is available at https://github.com/On-Point-RND/EBES. Preprocessed data is available at https://huggingface.co/datasets/On-Point-Rnd/ebes that implements 9 modern models. Additionally, it includes the largest collection of EvS datasets, featuring 10 curated datasets, including a novel synthetic dataset and real-world data with the largest publicly available banking dataset. The library offers user-friendly interfaces for integrating new methods and datasets. Our benchmarking results highlight the unique properties of EvS compared to other sequential data types, provide a performance ranking of modern models-with GRU-based models achieving the best results-and reveal the challenges associated with robust EvS learning. The goal of EBES is to facilitate reproducible research, expedite progress in the field, and increase the real-world impact of EvS classification techniques.
Capillary Dataset: A Dataset of Nail-fold Capillaries Captured by Microscopy for Diabetes Detection
PDF ↗Diabetes mellitus is a chronic condition marked by insufficient insulin utilization or production, causing metabolic dysregulation. If not controlled, it can cause serious complications that affect major organ systems such as the cardiovascular and ocular systems. Early diagnosis is essential for timely interventions to ensure effective glycemic control and lower the risk of complications. The present study introduced a robust and comprehensive dataset derived from nail-fold capillaroscopy. This dataset, which includes imaging and some video data from 126 individuals diagnosed with type 2 diabetes mellitus (T2DM) alongside 76 healthy controls, consisted of 3283 images obtained from diabetic subjects and 3412 images from non-diabetic participants. All images were acquired at 390× high magnification with a high resolution of 640×480 pixels, including corresponding video data, ensuring a thorough and detailed dataset for our study. The dataset was organized into two analytical tracks. The first track focused on nail-fold morphology, classifying images into four types: hairpin, crossing, tortuous, and bushy. After filtering out duplicates and low-quality images, 1279 images were selected for analysis. The second track involved binary classification for the detection of diabetes, differentiating healthy individuals from people with diabetes. This utilized the complete dataset and the augmented versions created by combining multiple images into composite formats to enhance feature representation. Additionally, we evaluated several benchmark deep-learning models such as Vision Transformers (ViT) and Convolutional Neural Networks (CNNs) for morphological classification and diabetes detection tasks. This analysis illuminated current model performance, highlighted challenges, and paved the way for future research opportunities. Importantly, this dataset is critical and holds significant potential for advancing non-invasive automated diagnostic methodologies in diabetes-related nail-fold capillary research, offering a promising future for diabetes research. The dataset is available at: https://huggingface.co/datasets/HanaNguyen/Capillary-Dataset. The github: https://github.com/urgonguyen/Capillarydataset.git.
Simulating a recommendation system in a controlled environment, to identify specific behaviors and user preferences, requires highly flexible synthetic data generation models capable of mimicking the patterns and trends of real datasets. In this context, we propose HYDRA, a novel preferences data generation model driven by three main factors: user-item interaction level, item popularity, and user engagement level. The key innovations of the proposed process include the ability to generate user communities characterized by similar item adoptions, reflecting real-world social influences and trends. Additionally, HYDRA considers item popularity and user engagement as mixtures of different probability distributions, allowing for a more realistic simulation of diverse scenarios. This approach enhances the model's capacity to simulate a wide range of real-world cases, capturing the complexity and variability found in actual user behavior. We demonstrate the effectiveness of HYDRA through extensive experiments on well-known benchmark datasets. The results highlight its capability to replicate real-world data patterns, offering valuable insights for developing and testing recommendation systems in a controlled and realistic manner. The code used to perform the experiments is publicly available: https://github.com/flexibledatageneration/HYDRA.
We introduce IrrMap, the first large-scale dataset (1.1 million patches) for irrigation method mapping across regions. IrrMap consists of multi-resolution satellite imagery from LandSat and Sentinel, along with key auxiliary data such as crop type, land use, and vegetation indices. The dataset spans 1,668,899 farms and 11,443,492 acres across multiple western U.S. states from 2013 to 2023, providing a rich and diverse foundation for irrigation analysis and ensuring geospatial alignment and quality control. The dataset is ML-ready, with standardized 224×224 GeoTIFF patches, the multiple input modalities, carefully chosen train-test-split data, and accompanying dataloaders for seamless deep learning model training and benchmarking in irrigation mapping. The dataset is also accompanied by a complete pipeline for dataset generation, enabling researchers to extend IrrMap to new regions for irrigation data collection or adapt it with minimal effort for other similar applications in agricultural and geospatial analysis. We also analyze the irrigation method distribution across crop groups, spatial irrigation patterns (using Shannon diversity indices), and irrigated area variations for both LandSat and Sentinel, providing insights into regional and resolution-based differences. To promote further exploration, we openly release IrrMap, along with the derived datasets, benchmark models, and pipeline code, through a GitHub repository: https://github.com/Nibir088/IrrMap and Data repository: https://huggingface.co/Nibir/IrrMap, providing comprehensive documentation and implementation details.
T3Set: A Multimodal Dataset with Targeted Suggestions for LLM-based Virtual Coach in Table Tennis Training
PDF ↗Coaching is critical for learning table tennis skills. However, amateur table tennis players often lack access to professional coaches due to high costs and a limited number of coaches. While recent multimodal large language models show promise as virtual coaches, most of the existing approaches merely rely on video analysis, which is not comprehensive enough. In table tennis, many important kinematic details (e.g., strength, acceleration) cannot be captured by videos. They can only be tracked using sensors. To address this gap, we present T3Set (Table Tennis Training Set), a multimodal dataset that synchronizes inertial measurement unit (IMU) data from sensors mounted on 32 players' rackets with video recordings. The sensor data has 16 dimensions and a sample rate of 100Hz. This dataset covers 7 fundamental techniques across 380 training rounds, totaling 8655 annotated strokes, with 8395 targeted suggestions from coaches. The key features of T3Set include (1) temporal alignment between sensor data, video data, and text data. (2) high-quality targeted suggestions which are consistent with predefined suggestion taxonomy. Based on T3Set, we propose a novel two-stage framework that effectively integrates motion perception with generative reasoning as a virtual coach. Our method quantitatively outperforms baseline methods. The dataset, code, and documentation are available at https://github.com/jima-cs/T3Set.
On the Generalization and Adaptation Ability of Machine-Generated Text Detectors in Academic Writing
PDF ↗The rising popularity of large language models (LLMs) has raised concerns about potential abuse and harmful content. As a result, developing a highly generalizable and adaptable machine-generated text (MGT) detection system has become an urgent priority. Given that LLMs are most commonly misused in academic writing, this work investigates the generalization and adaptation capabilities of MGT detectors in three key aspects specific to academic writing: First, we construct MGT-Academic, a large-scale dataset comprising over 336M tokens and 749K samples. MGT-Academic focuses on academic writing, featuring human-written texts (HWTs) and MGTs across STEM, Humanities, and Social Sciences, paired with an extensible code framework for efficient benchmarking. Second, we benchmark the performance of various detectors for binary classification and text attribution tasks in both in-domain and cross-domain settings. This benchmark reveals the often-overlooked challenges of text attribution tasks. Third, we introduce a novel text attribution task in which models must adapt to new classes over time, with little or no access to prior training data, spanning both few-shot and many-shot scenarios. We implement a range of adaptation techniques to enhance performance across these settings. Our findings provide new insights into the generalization ability of MGT detectors and lay the foundation for building robust, adaptive detection systems. The code framework is available at https://github.com/Y-L-LIU/MGTBench-2.0.
Natural Language to SQL (i.e., NL2SQL) translation is crucial for democratizing database access, but even state-of-the-art models frequently generate semantically incorrect SQL queries, hindering the widespread adoption of these techniques by database vendors. While existing NL2SQL benchmarks primarily focus on correct query translation, we argue that a benchmark dedicated to identifying common errors in NL2SQL translations is equally important, as accurately detecting these errors is a prerequisite for any subsequent correction - whether performed by humans or models. To address this gap, we propose NL2SQL-BUGs, the first benchmark dedicated to detecting and categorizing semantic errors in NL2SQL translation. NL2SQL-BUGs adopts a two-level taxonomy to systematically classify semantic errors, covering 9 main categories and 31 subcategories. The benchmark consists of 2,018 expert-annotated instances, each containing a natural language query, database schema, and SQL query, with detailed error annotations for semantically incorrect queries. Through comprehensive experiments, we demonstrate that current large language models exhibit significant limitations in semantic error detection, achieving an average detection accuracy of 75.16%. Specifically, our method successfully detected 106 errors (accounting for 6.91% ) in BIRD, a widely-used NL2SQL dataset, which were previously undetected annotation errors. This highlights the importance of semantic error detection in NL2SQL systems. The benchmark is publicly available at https://nl2sql-bugs.github.io/.
Large language models (LLMs) achieve remarkable success in natural language processing (NLP). In practical scenarios like recommendations, as users increasingly seek personalized experiences, it becomes crucial to incorporate user interaction history into the context of LLMs to enhance personalization. However, from a practical utility perspective, user interactions' extensive length and noise present challenges when used directly as text prompts. A promising solution is to compress and distill interactions into compact embeddings, serving as soft prompts to assist LLMs in generating personalized responses. Although this approach brings efficiency, a critical concern emerges: Can user embeddings adequately capture valuable information and prompt LLMs? To address this concern, we propose UQABench, a benchmark designed to evaluate the effectiveness of user embeddings in prompting LLMs for personalization. We establish a fair and standardized evaluation process, encompassing pre-training, fine-tuning, and evaluation stages. To thoroughly evaluate user embeddings, we design three dimensions of tasks: sequence understanding, action prediction, and interest perception. These evaluation tasks cover the industry's demands in traditional recommendation tasks, such as improving prediction accuracy, and its aspirations for LLM-based methods, such as accurately understanding user interests and enhancing the user experience. We conduct extensive experiments on various state-of-the-art methods for modeling user embeddings. Additionally, we reveal the scaling laws of leveraging user embeddings to prompt LLMs. The benchmark is available online at https://github.com/OpenStellarTeam/UQABench.
The detection of methane emissions has drawn significant attention from researchers, as methane is one of the main contributors to global warming. Due to their high accessibility and extensive coverage, satellite-based remote sensing data have become a widely used tool for studying methane emissions. The multispectral satellite Sentinel-2 is particularly prominent for studies related to methane, as it provides shortwave infrared (SWIR) spectral band data that align with the methane absorption line. Previous studies on methane emission detection using Sentinel-2 data rely mainly on synthetic data, which involves simulated methane emission plumes overlaid on real Sentinel-2 backgrounds. However, our analysis reveals that Sentinel-2 images with simulated plumes exhibit different data distributions and visual characteristics from those of real-world data. Moreover, these studies typically use only a single reference image as a clean background along with the plume image, which we find to be insufficient. Finally, the lack of publicly available benchmarks hinders research and development in this domain. To address these challenges, we introduce the MethaneS2CM dataset for methane emission detection, collected from Sentinel-2 multispectral data and the CarbonMapper platform. Our dataset consists of two main products (L1C and L2A) of Sentinel-2, including over 4000 plumes from 43 countries, ranging from 2016 to 2024. The dataset incorporates both long-term and short-term variations in Sentinel-2 images, with each sample consisting of images captured at different time periods. Furthermore, we conduct extensive experiments to evaluate a wide range of existing methane emission detection methods and models based on this dataset. We subsequently propose a deep methane emission classification model, MEECNet. Experimental results demonstrate that MEECNet outperforms all baseline methods. Our dataset is publicly available at https://huggingface.co/datasets/H1deaki/MethaneS2CM.