Battery Life Prediction (BLP), which relies on time series data produced by battery degradation tests, is crucial for battery utilization, optimization, and production. Despite impressive advancements, this research area faces three key challenges. Firstly, the limited size of existing datasets impedes insights into modern battery life data. Secondly, most datasets are restricted to small-capacity lithium-ion batteries tested under a narrow range of diversity in labs, raising concerns about the generalizability of findings. Thirdly, inconsistent and limited benchmarks across studies obscure the effectiveness of baselines and leave it unclear if models popular in other time series fields are effective for BLP. To address these challenges, we propose BatteryLife, a comprehensive dataset and benchmark for BLP. BatteryLife integrates 16 datasets, offering a 2.5 times sample size compared to the previous largest dataset, and provides the most diverse battery life resource with batteries from 8 formats, 59 chemical systems, 9 operating temperatures, and 421 charge/discharge protocols, including both laboratory and industrial tests. Notably, BatteryLife is the first to release battery life datasets of zinc-ion batteries, sodium-ion batteries, and industry-tested large-capacity lithium-ion batteries. With the comprehensive dataset, we revisit the effectiveness of baselines popular in this and other time series fields. Furthermore, we propose CyclePatch, a plug-in technique that can be employed in various neural networks. Extensive benchmarking of 18 methods reveals that models popular in other time series fields can be unsuitable for BLP, and CyclePatch consistently improves model performance establishing state-of-the-art benchmarks. Moreover, BatteryLife evaluates model performance across aging conditions and domains. BatteryLife is available at https://github.com/Ruifeng-Tan/BatteryLife.
论文检索
输入标题、作者或关键词，从 2,893 篇学术成果中精准定位
Recent advances in graph machine learning (ML) with the introduction of Graph Neural Networks (GNNs) have led to a widespread interest in applying these approaches to business applications at scale. GNNs enable differentiable end-to-end (E2E) learning of model parameters given graph structure which enables optimization towards popular node, edge (link) and graph-level tasks. While the research innovation in new GNN layers and training strategies has been rapid, industrial adoption and utility of GNNs has lagged considerably due to the unique scale challenges that large-scale graph ML problems create. In this work, we share our approach to training, inference, and utilization of GNNs at Snapchat. To this end, we present GiGL (Gigantic Graph Learning), an open-source library to enable large-scale distributed graph ML to the benefit of researchers, ML engineers, and practitioners. We use GiGL internally at Snapchat to manage the heavy lifting of GNN workflows, including graph data preprocessing from relational DBs, subgraph sampling, distributed training, inference, and orchestration. GiGL is designed to interface cleanly with open-source GNN modeling libraries prominent in academia like PyTorch Geometric (PyG), while handling scaling and productionization challenges that make it easier for internal practitioners to focus on modeling. GiGL is used in multiple production settings, and has powered over 35 launches across multiple business domains in the last 2 years in the contexts of friend recommendation, content recommendation and advertising. This work details high-level design and tools the library provides, scaling properties, case studies in diverse business settings with large-scale graphs up to hundreds of millions of nodes, tens of billions of edges, and hundreds of node and edge features, and several key lessons learned in employing graph ML at scale on large social data. GiGL is open-sourced at https://github.com/Snapchat/GiGL.
Graph neural networks (GNNs) have emerged as state-of-the-art methods to learn from graph-structured data for recommendation. However, most existing GNN-based recommendation methods focus on the optimization of model structures and learning strategies based on pre-defined graphs, neglecting the importance of the graph construction stage. Earlier works for graph construction usually rely on specific rules or crowdsourcing, which are either too simplistic or too labor-intensive. Recent works start to utilize large language models (LLMs) to automate the graph construction, in view of their abundant open-world knowledge and remarkable reasoning capabilities. Nevertheless, they generally suffer from two limitations: (1) invisibility of global view (e.g., overlooking contextual information) and (2) construction inefficiency. To this end, we introduce AutoGraph, an automatic graph construction framework based on LLMs for recommendation. Specifically, we first use LLMs to infer the user preference and item knowledge, which is encoded as semantic vectors. Next, we employ vector quantization to extract the latent factors from the semantic vectors. The latent factors are then incorporated as extra nodes to link the user/item nodes, resulting in a graph with in-depth global-view semantics. We further design metapath-based message aggregation to effectively aggregate the semantic and collaborative information. The framework is model-agnostic and compatible with different backbone models. Extensive experiments on three real-world datasets demonstrate the efficacy and efficiency of AutoGraph compared to existing baseline methods. We have deployed AutoGraph in Huawei advertising platform, and gain a 2.69% improvement on RPM and a 7.31% improvement on eCPM in the online A/B test. Currently AutoGraph has been used as the main traffic model, serving hundreds of millions of people.
Gene selection in high-dimensional genomic data is essential for understanding disease mechanisms and improving therapeutic outcomes. Traditional feature selection methods effectively identify predictive genes but often ignore complex biological pathways and regulatory networks, leading to unstable and biologically irrelevant signatures. Prior approaches, such as Lasso-based methods and statistical filtering, either focus solely on individual gene-outcome associations or fail to capture pathway-level interactions, presenting a key challenge: how to integrate biological pathway knowledge while maintaining statistical rigor in gene selection? To address this gap, we propose a novel two-stage framework that integrates statistical selection with biological pathway knowledge using multi-agent reinforcement learning (MARL). First, we introduce a pathway-guided pre-filtering strategy that leverages multiple statistical methods alongside KEGG pathway information for initial dimensionality reduction. Next, for refined selection, we model genes as collaborative agents in a MARL framework, where each agent optimizes both predictive power and biological relevance. Our framework incorporates pathway knowledge through Graph Neural Network-based state representations, a reward mechanism combining prediction performance with gene centrality and pathway coverage, and collaborative learning strategies using shared memory and a centralized critic component. Extensive experiments on multiple gene expression datasets demonstrate that our approach significantly improves both prediction accuracy and biological interpretability compared to traditional methods.
Managing evolving graph data presents substantial challenges in storage and privacy, and training graph neural networks (GNNs) on such data often leads to catastrophic forgetting, impairing performance on earlier tasks. Despite existing continual graph learning (CGL) methods mitigating this to some extent, they rely on centralized architectures and ignore the potential of distributed graph databases to leverage collective intelligence. To this end, we propose Federated Continual Graph Learning (FCGL) to adapt GNNs across multiple evolving graphs under storage and privacy constraints. Our empirical study highlights two core challenges: local graph forgetting (LGF), where clients lose prior knowledge when adapting to new tasks, and global expertise conflict (GEC), where the global GNN exhibits sub-optimal performance in both adapting to new tasks and retaining old ones, arising from inconsistent client expertise during server-side parameter aggregation. To address these, we introduce POWER, a framework that preserves experience nodes with maximum local-global coverage locally to mitigate LGF, and leverages pseudo-prototype reconstruction with trajectory-aware knowledge transfer to resolve GEC. Experiments on various graph datasets demonstrate POWER's superiority over federated adaptations of CGL baselines and vision-centric federated continual learning approaches.
Loss or Gain: Hierarchical Conditional Information Bottleneck Approach for Incomplete Time Series Classification
PDF ↗Incomplete time series classification is both practically valuable and challenging as missing values in time series data are prevalent in real-world scenarios. Current approaches suffer from two major limitations. First, they overemphasize the consistency of data reconstruction during missing value imputation while neglecting the task-effectiveness of the imputed results for the classification. Second, they fail to systematically establish a synergistic optimization mechanism between data imputation and feature representation. To address these challenges, we propose a Hierarchical Conditional Information Bottleneck (HCIB) framework, which achieves incomplete time series classification through end-to-end joint optimization. Specifically, at the data imputation level, we re-examine the dual effects of missing data: the loss of critical information (Loss) versus the gain in interference suppression (Gain), elucidating this duality through bias-variance trade-off theory. Building on this analysis, we propose a task-information sufficiency criterion and extend the information bottleneck theory into a task-driven imputation framework by incorporating label information as a conditional constraint. At the feature representation level, we construct a hierarchical information bottleneck architecture to learn compressed yet informative temporal representations from the task-oriented imputed data. Furthermore, we derive the optimizable objective function for HCIB and design specialized neural network architectures for time series. Comprehensive experiments on multivariate and univariate time series datasets across multiple domains consistently demonstrate that the proposed method achieves significant improvements in classification performance compared to SOTA approaches.
Network interdiction problems, arising in critical applications from military strategy to disease control, involve a complex attacker-defender dynamic: one player optimizes a network-based objective, while the other strategically modifies the network to impede that objective. The inherent bi-level optimization and combinatorial nature of these problems pose a significant computational challenge, often rendering traditional exact solvers impractical and hindering the development of effective heuristics. While Graph Neural Networks (GNNs) have demonstrated promise in solving single-level combinatorial optimization problems on graphs, their direct application to bi-level interdiction problems remains limited. In this paper, we bridge this gap by introducing a novel approach that leverages the power of GNNs to learn Mixed-Integer Linear Programming (MILP) formulations of network interdiction problems. By representing the problem in this structured mathematical form, we empower a multipartite GNN with the representational capacity to effectively capture the complex interplay between the two players. This approach aligns the neural network with the underlying mathematical structure of interdiction problems, leading to improved performance. Through extensive experiments on two network interdiction tasks, we demonstrate the superiority of our proposed method over both baseline GNN models and traditional exact solvers, showcasing its potential for real-world applications.
Research on Graph Structure Learning (GSL) provides key insights for graph-based clustering, yet current methods like Graph Neural Networks (GNNs), Graph Attention Networks (GATs), and contrastive learning often rely heavily on the original graph structure. Their performance deteriorates when the original graph's adjacency matrix is too sparse or contains noisy edges unrelated to clustering. Moreover, these methods depend on learning node embeddings and using traditional techniques like k-means to form clusters, which may not fully capture the underlying graph structure between nodes. To address these limitations, this paper introduces DeSE, a novel unsupervised graph clustering framework incorporating Deep Structural Entropy. It enhances the original graph with quantified structural information and deep neural networks to form clusters. Specifically, we first propose a method for calculating structural entropy with soft assignment, which quantifies structure in a differentiable form. Next, we design a Structural Learning layer (SLL) to generate an attributed graph from the original feature data, serving as a target to enhance and optimize the original structural graph, thereby mitigating the issue of sparse connections between graph nodes. Finally, our clustering assignment method (ASS), based on GNNs, learns node embeddings and a soft assignment matrix to cluster on the enhanced graph. The ASS layer can be stacked to meet downstream task requirements, minimizing structural entropy for stable clustering and maximizing node consistency with edge-based cross-entropy loss. Extensive comparative experiments are conducted on four benchmark datasets against eight representative unsupervised graph clustering baselines, demonstrating the superiority of the DeSE in both effectiveness and interpretability.
Although Graph Neural Networks (GNNs) have shown promising potential in fake news detection, they remain highly vulnerable to adversarial manipulations within social networks. Existing methods primarily establish connections between malicious accounts and individual target news to investigate the vulnerability of graph-based detectors, while they neglect the structural relationships surrounding targets, limiting their effectiveness in robustness evaluation. In this work, we propose a novel Structural Information principles-guided Adversarial Attack Framework, namely SI2AF, which effectively challenges graph-based detectors and further probes their detection robustness. Specifically, structural entropy is introduced to quantify the dynamic uncertainty in social engagements and identify hierarchical communities that encompass all user accounts and news posts. An influence metric is presented to measure each account's probability of engaging in random interactions, facilitating the design of multiple agents that manage distinct malicious accounts. For each target news, three attack strategies are developed through multi-agent collaboration within the associated subgraph to optimize evasion against black-box detectors. By incorporating the adversarial manipulations generated by SI2AF, we enrich the original network structure and refine graph-based detectors to improve their robustness against adversarial attacks. Extensive evaluations demonstrate that SI2AF significantly outperforms state-of-the-art baselines in attack effectiveness with an average improvement of 16.71%, and enhances GNN-based detection robustness by 41.54% on average.
Oldie but Goodie: Re-illuminating Label Propagation on Graphs with Partially Observed Features
PDF ↗In real-world graphs, we often encounter missing feature situations where a few or the majority of node features, e.g., sensitive information, are missed. In such scenarios, directly utilizing Graph Neural Networks (GNNs) would yield sub-optimal results in downstream tasks such as node classification. Despite the emergence of a few GNN-based methods attempting to mitigate its missing situation, when only a few features are available, they rather perform worse than traditional structure-based models. To this end, we propose a novel framework that further illuminates the potential of classical Label Propagation (Oldie), taking advantage of Feature Propagation, especially when only a partial feature is available. Now called by GOODIE, it takes a hybrid approach to obtain embeddings from the Label Propagation branch and Feature Propagation branch. To do so, we first design a GNN-based decoder that enables the Label Propagation branch to output hidden embeddings that align with those of the FP branch. Then, GOODIE automatically captures the significance of structure and feature information thanks to the newly designed Structure-Feature Attention. Followed by a novel Pseudo-Label contrastive learning that differentiates the contribution of each positive pair within pseudo-labels originating from the LP branch, GOODIE outputs the final prediction for the unlabeled nodes. Through extensive experiments, we demonstrate that our proposed model, GOODIE, outperforms the existing state-of-the-art methods not only when only a few features are available but also in abundantly available situations. Source code of GOODIE is available at: https://github.com/SukwonYun/GOODIE.
Graph adversarial attacks refer to a class of adversarial attack methods targeting graph-structured data, such as social networks, knowledge graphs, and molecular structures. The objective of these attacks is to introduce subtle yet carefully designed perturbations to graph data (e.g., nodes, edges, or features) to cause graph-based machine learning models, such as graph neural networks (GNNs), to produce incorrect predictions or classification results. The core idea behind graph adversarial attacks is to exploit the vulnerabilities of the model, maximizing the degradation of its performance with minimal perturbations. However, due to the incompleteness of attack strategies, graph adversarial attacks often exhibit redundancy, meaning that some perturbations are ineffective. These ineffective perturbations consume the attack budget and increase the likelihood of detection. To address this issue, this paper proposes Budget-Reduced Attack Filtering (BRAF in short) frameworks capable of eliminating ineffective perturbations produced by existing graph adversarial attack methods, thereby optimizing the attack budget. Extensive experiments validate the effectiveness and scalability of the proposed filtering frameworks. The code is available at https://github.com/Xiangchao-Wen/Filter-Graph-Attack.
Learning Reliable and Intuitive Temporal Logic Rules for Interpretable Time Series Classification
PDF ↗Recently, rule-based time series classification models are widely used in safety-critical scenarios demanding strong interpretability, through providing explicit and rigorous rationales. However, due to the discrepancy between discrete rules and continuous neural networks, the generated rules are not completely consistent with the actual decision-making process, rendering users hesitant to trust the model. Additionally, existing methods learn the role (weight) of each time point independently, which is not in line with human understanding and the consecutive nature of temporal properties. In this paper, we propose a novel neuro-symbolic model named TemporalRule, aiming to automatically learn Signal Temporal Logic (STL) rules for interpretable time series classification. Our model directly optimizes the neural networks representing discrete rules via gradient grafting, producing reliable rules that can exactly determine the classification results. Notably, a temporal logical layer is designed to simulate expressive temporal operators, including one 2-predicate operator (Until) and two 2-level composite operators (EA and AE), through adaptively learning the time bounds of involved intervals. That creates consistent weights for consecutive time points, enhancing the intuitiveness of derived rules. Extensive experiments on diverse real-world datasets in safety-critical domains show that TemporalRule achieves superior and stable classification accuracy compared to state-of-the-art rule-based approaches, and the learned rules can precisely and concisely explicate the classification results, leading to a trustable model.
Conformal prediction for graph neural networks (GNNs) offers a promising framework for quantifying uncertainty, enhancing GNN reliability in high-stakes applications. However, existing methods predominantly focus on static graphs, neglecting the evolving nature of real-world graphs. Temporal dependencies in graph structure, node attributes, and ground truth labels violate the fundamental exchangeability assumption of standard conformal prediction methods, limiting their applicability. To address these challenges, in this paper, we introduce NCPNET, a novel end-to-end conformal prediction framework tailored for temporal graphs. Our approach extends conformal prediction to dynamic settings, mitigating statistical coverage violations induced by temporal dependencies. To achieve this, we propose a diffusion-based non-conformity score that captures both topological and temporal uncertainties within evolving networks. Additionally, we develop an efficiency-aware optimization algorithm that improves the conformal prediction process, enhancing computational efficiency and reducing coverage violations. Extensive experiments on diverse real-world temporal graphs, including WIKI, REDDIT, DBLP, and IBM Anti-Money Laundering dataset, demonstrate NCPNET's capability to ensure guaranteed coverage in temporal graphs, achieving up to a 31% reduction in prediction set size on the WIKI dataset, significantly improving efficiency compared to state-of-the-art methods. Our data and code are available at https://github.com/ODYSSEYWT/NCPNET.
Graph Few-Shot Node Classification (GFSNC) has emerged as a promising approach to address the challenge of learning with limited labeled data in graph-structured networks. Despite the success of Graph Neural Networks (GNNs) in node classification tasks, their performance heavily depends on the availability of abundant labeled data, which is often impractical in real-world scenarios. To tackle this issue, GFSNC adopts the episodic meta-learning paradigm, where models are trained on a series of meta-tasks. However, existing methods face two critical limitations: (i) they focus on local distributions within individual meta-tasks, neglecting the global data distribution, and (ii) they optimize models to minimize intra-class distances without adequately addressing inter-class separability, leading to suboptimal performance. This paper presents TaskNS, a novel GFSNC framework that introduces task-negative samples into meta-training tasks to address these limitations. By incorporating samples from classes outside the current meta-task, our framework enables the model to gradually learn the global distribution of the graph data. Additionally, we design a novel loss function that enhances the model's ability to distinguish between different classes of query samples. This loss function not only ensures high intra-class compactness but also maximizes the inter-class separation by leveraging task-negative samples. To further enhance the quality of task-negative samples, we propose an h-hop-neighbors-based sampling method that leverages the topological structure of a graph. It selects task-negative samples that are structurally close to query samples, ensuring that they are informative and challenging for the model to classify. Extensive experiments on four benchmark datasets demonstrate the effectiveness of TaskNS, achieving average improvements of 4.6% in accuracy (ACC) and 4.9% in F1-score over state-of-the-art methods.
Combining Denoised Neural Network and Genetic Symbolic Regression for Memory Behavior Modeling via Dynamic Asynchronous Optimization
PDF ↗Memory behavior modeling is a key topic in cognitive psychology and education. Traditional approaches use experimental data to build memory equations, but these models often lack precision and are debated in form. Recently, data-driven methods have improved predictive accuracy but struggle with interpretability, limiting cognitive insights. Although knowledge-informed neural networks have succeeded in fields like physics, their use in behavior modeling is still limited. This paper proposes a Self-evolving Psychology-informed Neural Network (SPsyINN), which leverages classical memory equations as knowledge modules to constrain neural network training. To address challenges such as the difficulty in quantifying descriptors and the limited interpretability of classical memory equations, a genetic symbolic regression algorithm is introduced to conduct evolutionary searches for more optimal expressions based on classical memory equations, enabling the mutual progress of the knowledge module and the neural network module. Specifically, the proposed approach combines genetic symbolic regression and neural networks in a parallel training framework, with a dynamic joint optimization loss function ensuring effective knowledge alignment between the two modules. Then, for addressing the training efficiency differences arising from the distinct optimization methods and computational hardware requirements of genetic algorithms and neural networks, an asynchronous interaction mechanism mediated by proxy data is developed to facilitate effective communication between modules and improve optimization efficiency. Finally, a denoising module is integrated into the neural network to enhance robustness against data noise and improve generalization performance. Experimental results on five large-scale real-world memory behavior demonstrate that SPsyINN outperforms state-of-the-art methods in predictive accuracy. Ablation studies confirm the model's co-evolution capability, improving accuracy while discovering more interpretable memory equations, showing its potential for psychological research. Our code is released at: https://github.com/JiaqiDijon/SPsyINN
Towards Efficient Few-shot Graph Neural Architecture Search via Partitioning Gradient Contribution
PDF ↗To address the weight coupling problem, certain studies introduced few-shot Neural Architecture Search (NAS) methods, which partition the supernet into multiple sub-supernets. However, these methods often suffer from computational inefficiency and tend to provide suboptimal partitioning schemes. To address this problem more effectively, we analyze the weight coupling problem from a novel perspective, which primarily stems from distinct modules in succeeding layers imposing conflicting gradient directions on the preceding layer modules. Based on this perspective, we propose the Gradient Contribution (GC) method that efficiently computes the cosine similarity of gradient directions among modules by decomposing the Vector-Jacobian Product during supernet backpropagation. Subsequently, the modules with conflicting gradient directions are allocated to distinct sub-supernets while similar ones are grouped together. To assess the advantages of GC and address the limitations of existing Graph Neural Architecture Search methods, which are limited to searching a single type of Graph Neural Networks (Message Passing Neural Networks (MPNNs) or Graph Transformers (GTs)), we propose the Unified Graph Neural Architecture Search (UGAS) framework, which explores optimal combinations of MPNNs and GTs. The experimental results demonstrate that GC achieves state-of-the-art (SOTA) performance in supernet partitioning quality and time efficiency. In addition, the architectures searched by UGAS+GC outperform both the manually designed GNNs and those obtained by existing NAS methods. Finally, ablation studies further demonstrate the effectiveness of all proposed methods.
Wind farms generate electricity from wind, offering a sustainable and eco-friendly power source. In wind farms, determining the positions of wind turbines is crucial for high energy production due to complex wake interactions. However, optimizing wind farm layouts for given wind conditions remains challenging. While exist- ing methods try to solve the problem by training a surrogate model and employing optimization algorithms based on the model, these methods require a large number of simulations to obtain highly productive layouts, which is time-consuming in large-scale tasks. Furthermore, they mostly yield less diverse layouts, making the deployment of such layouts into real-world scenarios difficult. To address these challenges, we introduce a novel conditional genera- tive modeling approach to find wind farm layouts that maximize energy production. Our method consists of four stages. First, we collect datasets that consist of layouts and their corresponding an- nual energy production (AEP). Then, we train a diffusion model conditioned on AEP and wind scenarios using the collected dataset. Specifically, we use a Graph Neural Network as the backbone for the denoising network to ensure permutation invariance. Next, we sample promising layouts from the trained diffusion model by conditioning with a high AEP and given wind scenario. We also in- troduce a local search algorithm, which enables us to adjust layouts that violate design constraints. Finally, we evaluate the generated layouts and augment the dataset. We repeat these processes itera- tively to further optimize the layouts. Throughout these processes, we effectively generate highly productive and diverse wind farm layouts in a sample-efficient manner. Experimental results demon- strate that our approach outperforms state-of-the-art methods in maximizing energy production across various scenarios.
The Quadratic Assignment Problem (QAP) has attracted lasting attention for its wide applications and computational challenges. Despite the rich literature in machine learning for QAP, most works often address the problem in the setting of image matching, whereby deep networks could play a vital role in extracting useful features for the subsequent matching. While its power on pure numerical QAP instances is limited in node embedding, often with a vanilla graph neural network. This paper tries to tap the potential of deep nets for QAP, specifically by modifying the input instance, which is orthogonal to previous efforts. Specifically, we develop a bi-level unsupervised framework, where the inner optimization involves trying to solve the modified instance with entropic regularization that can be solved iteratively using the Sinkhorn algorithm without affecting backpropagation by truncating gradients during training. The outer minimization deals with the quadratic objective function of the original QAP. In particular, seeing the intractable scale of the most general form, i.e., Lawler's QAP and the practical utility of the more efficient Koopmans-Beckmann QAP (KBQAP) form for solving other graph and combinatorial problems like TSP and graph edit distance, we embody our network on the KBQAP, and show its strong performance on various benchmarks in our experiments.
Graph Neural Networks (GNNs) have achieved significant success in various graph learning tasks. However, their success relies heavily on the availability of adequate high-quality labels, which requires high annotation costs. Recently, Large Language Models (LLMs) became known to the community for their strong zero-shot performance on diverse textual tasks. To utilize LLM's zero-shot abilities, existing work substitutes the oracle in the graph active learning setting with an LLM and achieves low-cost annotation compared to consulting human experts. However, annotations from LLMs inevitably contain noise, and we find through an empirical analysis that LLMs possess complex and dataset-specific annotation noise distributions. This intricacy of LLMs' annotation noise is neglected by existing work, leading to sub-optimal annotation data quality under the same budget. In this paper, we propose a Dataset- and LLM-Aware graph active learning framework called DMA that explicitly models LLM's annotation noise distributions on the given dataset. Concretely, DMA queries the oracle LLM with manually constructed prompts for each class's pseudo samples that are then used to measure the class-wise annotation noise rates. We then equip DMA with a new noise-aware graph active learning algorithm to utilize this fine-grained annotation noise distribution. Extensive performance optimizations are applied to the implementation to boost DMA's scalability and runtime efficiency on large-scale datasets. Evaluations on five text-attributed graph datasets show that DMA consistently outperforms all the baselines in terms of annotation data quality, which illustrates the importance of careful handling of LLM's intricate annotation noise.
Differentially private stochastic gradient descent (DP-SGD) is broadly considered to be the gold standard for training and fine-tuning neural networks under differential privacy (DP). With the increasing availability of high-quality pre-trained model checkpoints (e.g., vision and language models), fine-tuning has become a popular strategy. However, despite recent progress in understanding and applying DP-SGD for private transfer learning tasks, significant challenges remain - most notably, the performance gap between models fine-tuned with DP-SGD and their non-private counterparts. Sparse fine-tuning on private data has emerged as an alternative to full-model fine-tuning -- recent work has shown that privately fine-tuning only a small subset of model weights and keeping the rest of the weights fixed can lead to better performance. In this work, we propose a new approach for sparse fine-tuning of neural networks under DP. Existing work on private sparse finetuning often used fixed choice of trainable weights (e.g., updating only the last layer), or relied on public model's weights to choose the subset of weights to modify. Such choice of weights remains suboptimal. In contrast, we explore an optimization-based approach, where our selection method makes use of the private gradient information, while using off the shelf privacy accounting techniques. Our numerical experiments on several computer vision models and datasets show that our parameter selection method leads to better prediction accuracy, compared to full-model private fine-tuning or existing private sparse fine-tuning approaches. Our code is available here: https://github.com/mazumder-lab/SPARTA/tree/main