Anomaly detection on text-rich graphs is widely prevalent in real life, such as detecting incorrectly assigned academic papers to authors and detecting bots in social networks. The remarkable capabilities of large language models (LLMs) pave a new revenue by utilizing rich-text information for effective anomaly detection. However, simply introducing rich texts into LLMs can obscure essential detection cues and introduce high fine-tuning costs. Moreover, LLMs often overlook the intrinsic structural bias of graphs which is vital for distinguishing normal from abnormal node patterns. To this end, this paper introduces GuARD, a text-rich and graph-informed language model that combines key structural features from graph-based methods with fine-grained semantic attributes extracted via small language models for effective anomaly detection on text-rich graphs. GuARD is optimized with the progressive multimodal multi-turn instruction tuning framework in the task-guided instruction tuning regime tailed to incorporate both rich-text and structural modalities. Extensive experiments on four datasets reveal that GuARD outperforms graph-based and LLM-based anomaly detection methods, while offering up to 5× speedup in training and 10× speedup in inference over vanilla long-context LLMs on the large-scale WhoIsWho dataset.
论文检索
输入标题、作者或关键词，从 844 篇学术成果中精准定位
Despite the advancements in quantum convolution or quanvolution, challenges persist in making quanvolution scalable, efficient, and applicable to multi-dimensional data. Existing quanvolutional networks heavily rely on classical layers, with minimal quantum involvement due to inherent limitations in current quanvolution algorithms. Moreover, the application of quanvolution in the domain of 1D data remains largely unexplored. To address these limitations, we propose a new quanvolution algorithm-Quanv1D-capable of processing arbitrary-channel 1D data, handling variable kernel sizes, and generating a customizable number of feature maps, along with a classification network-fully quanvolutional network (FQN)-built solely using Quanv1D layers. Quanv1D is inspired by the classical Conv1D and stands out from the quanvolution literature by being fully trainable, modular, and freely scalable with a self-regularizing feature. To evaluate FQN, we tested it on 20 UEA and UCR time series datasets, both univariate and multivariate, and benchmarked its performance against state-of-the-art convolutional models (both quantum and classical). We found FQN to outperform all compared models in terms of average accuracy while using significantly fewer parameters. Additionally, to assess the viability of FQN on real hardware, we conducted a shot-based analysis across all the datasets to simulate statistical quantum noise and found our model robust and equally efficient.
Robust and Explainable Detector of Time Series Anomaly via Augmenting Multiclass Pseudo-Anomalies
PDF ↗Unsupervised anomaly detection in time series has been a pivotal research area for decades. Current mainstream approaches focus on learning normality, on the assumption that all or most of the samples in the training set are normal. However, anomalies in the training set (i.e., anomaly contamination) can be misleading. Recent studies employ data augmentation to generate pseudo-anomalies and learn the boundary separating the training samples from the augmented samples. Although this approach mitigates anomaly contamination if augmented samples mimic unseen real anomalies, it suffers from several limitations. (1) Covering a wide range of time series anomalies is challenging. (2) It disregards augmented samples that resemble normal samples (i.e., false anomalies). (3) It places too much trust in the labels of training and augmented samples. In response, we propose RedLamp, which employs diverse data augmentations to generate multiclass pseudo-anomalies and learns the multiclass boundary. Such multiclass pseudo-anomalies cover a wide variety of time series anomalies. We conduct multiclass classification using soft labels, which prevents the model from being overconfident and ensures its robustness against contaminated/false anomalies. The learned latent space is inherently explainable as it is trained to separate pseudo-anomalies into multiclasses. Extensive experiments demonstrate the effectiveness of RedLamp in anomaly detection and its robustness against anomaly contamination.
Utilizing Strategic Pre-training to Reduce Overfitting: Baguan - A Pre-trained Weather Forecasting Model
PDF ↗Weather forecasting has long posed a significant challenge for humanity. While recent AI-based models have surpassed traditional numerical weather prediction (NWP) methods in global forecasting tasks, overfitting remains a critical issue due to the limited availability of real-world weather data spanning only a few decades. Unlike fields like computer vision or natural language processing, where data abundance can mitigate overfitting, weather forecasting demands innovative strategies to address this challenge with existing data. In this paper, we explore pre-training methods for weather forecasting, finding that selecting an appropriately challenging pre-training task introduces locality bias, effectively mitigating overfitting and enhancing performance. We introduce Baguan, a novel data-driven model for medium-range weather forecasting, built on a Siamese Autoencoder pre-trained in a self-supervised manner and fine-tuned for different lead times. Experimental results show that Baguan outperforms traditional methods, delivering more accurate forecasts. Additionally, the pre-trained Baguan demonstrates robust overfitting control and excels in downstream tasks, such as subseasonal-to-seasonal (S2S) modeling and regional forecasting, after fine-tuning.
Bias evaluation is fundamental to trustworthy AI, both in terms of checking data quality and in terms of checking the outputs of AI systems. In testing data quality, for example, one may study the distance of a given dataset, viewed as a distribution, to a given ground-truth reference dataset. However, classical metrics, such as the Total Variation and the Wasserstein distances, are known to have high sample complexities and, therefore, may fail to provide a meaningful distinction in many practical scenarios. In this paper, we propose a new notion of distance, the Maximum Subgroup Discrepancy (MSD). In this metric, two distributions are close if, roughly, discrepancies are low for all feature subgroups. While the number of subgroups may be exponential, we show that the sample complexity is linear in the number of features, thus making it feasible for practical applications. Moreover, we provide a practical algorithm for evaluating the distance based on Mixed-integer optimization (MIO). We also note that the proposed distance is easily interpretable, thus providing clearer paths to fixing the biases once they have been identified. Finally, we describe a natural general bias detection framework, termed MSDD distances, and show that MSD aligns well with this framework. We empirically evaluate MSD by comparing it with other metrics and by demonstrating the above properties of MSD on real-world datasets.
Understanding human mobility patterns is a complex challenge that requires modeling both node-oriented time series (e.g., population) and edge-oriented time series (e.g., population flows) within graph topologies across time. While previous methods have focused on either node-oriented time series or interactions, the synergistic integration of these two modalities has proven difficult to achieve. In this paper, we propose BINTS (BI-modal learning for Networked Time Series), a novel bi-modal learning framework that employs soft contrastive learning along the temporal axis. BINTS captures modality similarities and temporal patterns by simultaneously learning from evolving node-oriented time series and interactions, solving the limitations of single-modality approaches. To evaluate our method, we curate comprehensive multi-modal human mobility datasets spanning diverse locations and times. Our experimental results demonstrate that BINTS significantly outperforms existing forecasting models by capturing synergies across different data modalities. Overall, we establish BINTS as a powerful technique for holistically understanding and forecasting complex mobility dynamics. For reproducibility, the source code of our framework is available at https://github.com/kaist-dmlab/BINTS.
Time-series unsupervised domain adaptation (TS-UDA) is essential in fields such as healthcare and manufacturing, where data often consists of distinct entities, such as individual patients or machinery. This heterogeneity leads to discrepancies not only in feature distributions but also in label distributions, posing a significant challenge for domain adaptation. However, prior studies have mostly focused on alleviating covariate shifts, resulting in predicted target labels that are often biased toward the source domain's label distribution. To address this issue, we propose Time-series domain Adaptation for mitigating Label Shifts (TA4LS), a novel label refinement approach. TA4LS leverages the consistency between predicted labels and clustering information obtained from the unique characteristics that differentiate each label in the target domain. Furthermore, our approach as a plug-in module achieves performance improvements across diverse existing unsupervised domain adaptation methods, particularly in scenarios with significant discrepancies between source and target label distributions. In experiments on four benchmark datasets with label shifts, TA4LS demonstrates superior performance across six unsupervised domain adaptation methods and six label shift handling modules.
Optimising Budget Management via Primal-Dual Approximation with Constrained Polynomial Weights Update
PDF ↗Budget management is an essential capability in many online applications, including running advertising campaigns in sponsored search and allocating promotional content in recommender systems (RS). Most existing approaches to optimising budget spendings rely on improving worst-case approximation guarantees in the respective online knapsack packing problem or leveraging online learning techniques to improve an average system performance. However, worst-case approaches often underperform in practice, as extreme scenarios are uncommon, while online learning methods may lack robustness in highly non-stationary environments. In our work, we bridge these two approaches by developing an online budget pacing algorithm that preserves worst-case guarantees while improving the average allocative efficiency of budget management systems. Specifically, we frame the optimal budget allocation problem within an online learning perspective, where strategies chosen by the online learner are constrained by a primal-dual approximation algorithm defining the environment. We identify a feedback loop between the online learner's strategy and environment's response and propose a method to break this loop, optimising average allocative efficiency without compromising worst-case guarantees. To evaluate our approach, we conduct extensive offline experiments using both synthetic and real-world data, covering 320M promotional impressions across 20 different markets of a major industrial RS. Our results demonstrate that our method significantly enhances total allocated value and ROI across all markets.
Although large language models (LLMs) have shown great potential in recommender systems, the prohibitive computational costs for fine-tuning LLMs on entire datasets hinder their successful deployment in real-world scenarios. To develop affordable and effective LLM-based recommender systems, we focus on the task of coreset selection which identifies a small subset of fine-tuning data to optimize the test loss, thereby facilitating efficient LLMs' fine-tuning. Although there exist some intuitive solutions of subset selection, including distribution-based and importance-based approaches, they often lead to suboptimal performance due to the misalignment with downstream fine-tuning objectives or weak generalization ability caused by individual-level sample selection. To overcome these challenges, we propose GORACS, which is a novel Group-level Optimal tRAnsport-guided Coreset Selection framework for LLM-based recommender systems. GORACS is designed based on two key principles for coreset selection: 1) selecting the subsets that minimize the test loss to align with fine-tuning objectives, and 2) enhancing model generalization through group-level data selection. Corresponding to these two principles, GORACS has two key components: 1) a Proxy Optimization Objective (POO) leveraging optimal transport and gradient information to bound the intractable test loss, thus reducing computational costs by avoiding repeated LLM retraining, and 2) a two-stage Initialization-Then-Refinement Algorithm (ITRA) for efficient group-level selection. Our extensive experiments across diverse recommendation datasets and tasks validate that GORACS significantly reduces fine-tuning costs of LLMs while achieving superior performance over the state-of-the-art baselines and full data training.
Recently, we have been inundated with dynamic, time-evolving activity data streams generated by various individual IoT/sensor devices (e.g., healthcare devices and factory equipment). What would be the ideal mechanism for adaptively summarizing and modeling individual and local activities directly on small IoT devices with limited computing resources? In this paper, we present MicroAdapt, which comprises scalable and effective algorithms for modeling and forecasting time-evolving data streams. Our proposed method has the following advantages: (a) Adaptivity: it continuously monitors the latest time-series patterns in data streams, recognizes any sudden changes, updates model parameters, and enables real-time, long-range forecasting; (b) Any-time processing: it operates on a large collection of data streams, requiring a constant time (i.e., O(1)) for model updates and future predictions, at any point in time; (c) Lightweight computing: it does not require high-performance computing machines equipped with powerful GPUs, such as those used in deep learning, and can run on lightweight computers or even ''edge'' computing devices, such as Raspberry Pi. Intuitively, our method is inspired by the concept of an evolutionary adaptation mechanism of microorganisms, and thus it enables adaptive, any-time, lightweight model estimation and summarization of non-stationary data streams whose time-series patterns are constantly changing over time. We conduct extensive experiments on real datasets and demonstrate that MicroAdapt effectively captures important time-series patterns in the data streams, makes long-range forecasts, and consistently outperforms the existing state-of-the-art methods (e.g., TSMixer) as regards accuracy and execution speed. MicroAdapt improves accuracy by about 60% for MSE and 30% for MAE, while reducing computation time by up to six orders of magnitude (more than 100,000 times). We also applied our analytics to experiments implemented on Raspberry Pi with real IoT data streams, thus demonstrating the practicality and effectiveness of our approach. In fact, our proposed method required less than 1.95GB of memory and consumed less than 1.69W of power.
The rapid expansion in the size of new datasets has created a need for fast and efficient parameter-learning techniques. Compressive learning is a framework that enables efficient processing by using random, nonlinear features to project large-scale databases onto compact, information-preserving representations whose dimensionality is independent of the number of samples and can be easily stored, transferred, and processed. These database-level summaries are then used to decode parameters of interest from the underlying data distribution without requiring access to the original samples, offering an efficient and privacy-friendly learning framework. However, both the encoding and decoding techniques are typically randomized and data-independent, failing to exploit the underlying structure of the data. In this work, we propose a framework that meta-learns both the encoding and decoding stages of compressive learning methods by using neural networks that provide faster and more accurate systems than the current state-of-the-art approaches. To demonstrate the potential of the presented Compressive Meta-Learning framework, we explore multiple applications--including neural network-based compressive PCA, compressive ridge regression, compressive k-means, and autoencoders.
Differentially private stochastic gradient descent (DP-SGD) is broadly considered to be the gold standard for training and fine-tuning neural networks under differential privacy (DP). With the increasing availability of high-quality pre-trained model checkpoints (e.g., vision and language models), fine-tuning has become a popular strategy. However, despite recent progress in understanding and applying DP-SGD for private transfer learning tasks, significant challenges remain - most notably, the performance gap between models fine-tuned with DP-SGD and their non-private counterparts. Sparse fine-tuning on private data has emerged as an alternative to full-model fine-tuning -- recent work has shown that privately fine-tuning only a small subset of model weights and keeping the rest of the weights fixed can lead to better performance. In this work, we propose a new approach for sparse fine-tuning of neural networks under DP. Existing work on private sparse finetuning often used fixed choice of trainable weights (e.g., updating only the last layer), or relied on public model's weights to choose the subset of weights to modify. Such choice of weights remains suboptimal. In contrast, we explore an optimization-based approach, where our selection method makes use of the private gradient information, while using off the shelf privacy accounting techniques. Our numerical experiments on several computer vision models and datasets show that our parameter selection method leads to better prediction accuracy, compared to full-model private fine-tuning or existing private sparse fine-tuning approaches. Our code is available here: https://github.com/mazumder-lab/SPARTA/tree/main
Empowering large language models (LLMs) with effective tool utilization capabilities is crucial for enabling AI agents to solve complex problems. However, current models face two major limitations: (1) unreliable tool planning and invocation due to low-quality instruction datasets (e.g., widespread hallucinated API calls), and (2) weak tool reflection abilities (over 90% of errors cannot be corrected) resulting from static imitation learning. To address these critical limitations, we propose Tool-MVR, a novel Tool-Augmented LLM that achieves comprehensive System 2 reasoning through two key innovations. Specifically, we first introduce Multi-Agent Meta-Verification (MAMV), a systematic pipeline that rigorously validates APIs, queries, and reasoning trajectories to construct ToolBench-V, a new high-quality instruction dataset that addresses the limitation of unreliable tool planning and invocation. Second, we propose Exploration-based Reflection Learning (EXPLORE), which enhances tool reflection capabilities by leveraging tool feedback through a dynamic "Error → Reflection → Correction" learning paradigm, resulting in our reflection dataset ToolBench-R and addressing the critical weakness in tool reflection. Finally, we obtain Tool-MVR by finetuning open-source LLMs (e.g., Qwen-7B) on both ToolBench-V and ToolBench-R. Our experiments demonstrate that Tool-MVR achieves state-of-the-art performance on StableToolBench, surpassing both ToolLLM (by 23.9%) and GPT-4 (by 15.3%) while reducing API calls by 31.4%, with strong generalization capabilities across unseen tools and scenarios. Additionally, on our proposed RefineToolBench, the first benchmark specifically designed to evaluate tool reflection capabilities. Tool-MVR achieves a 58.9% error correction rate, significantly outperforming ToolLLM's 9.1%.
In medicine, treatments often influence multiple, interdependent outcomes, such as primary endpoints, complications, adverse events, or other secondary endpoints. Hence, to make optimal treatment decisions, clinicians are interested in learning the distribution of multi-dimensional treatment outcomes. However, the vast majority of machine learning methods for predicting treatment effects focus on single-outcome settings, despite the fact that medical data often include multiple, interdependent outcomes. To address this limitation, we propose a novel diffusion-based method called DIME to learn the joint distribution of multiple outcomes of medical treatments. Our DIME method addresses three challenges relevant in medical practice: (i) our method is tailored to learn the joint interventional distribution of multiple medical outcomes, which enables reliable decision-making with uncertainty quantification rather than relying solely on point estimates; (ii) our method explicitly captures the dependence structure between outcomes; and (iii) our method can handle outcomes of mixed type, including binary, categorical, and continuous variables. In our method, we take into account the fundamental problem of causal inference, where only outcomes for the assigned treatment are observed, through causal masking. For training, our method decomposes the joint distribution into a series of conditional distributions with a customized conditional masking to account for the dependence structure across outcomes. For inference, our method auto-regressively generates predictions. This allows our method to move beyond point estimates of causal quantities and thus learn the joint interventional distribution. To the best of our knowledge, DIME is the first neural method tailored to learn the joint, multi-outcome distribution of medical treatments. Across various experiments, we demonstrate that our method effectively learns the joint distribution and captures shared information among multiple outcomes.
Beyond Fixed Variables: Expanding-variate Time Series Forecasting via Flat Scheme and Spatio-temporal Focal Learning
PDF ↗Multivariate Time Series Forecasting (MTSF) has long been a key research focus. Traditionally, these studies assume a fixed number of variables, but in real-world applications, Cyber-Physical Systems often expand as new sensors are deployed, increasing variables in MTSF. In light of this, we introduce a novel task, Expanding-variate Time Series Forecasting (EVTSF). This task presents unique challenges, specifically (1) handling inconsistent data shapes caused by adding new variables, and (2) addressing imbalanced spatio-temporal learning, where expanding variables have limited observed data due to the necessity for timely operation. To address these challenges, we propose STEV, a flexible spatio-temporal forecasting framework. STEV includes a new Flat Scheme to tackle the inconsistent data shape issue, which extends the graph-based spatio-temporal modeling architecture into 1D space by flattening the 2D samples along the variable dimension, making the model variable-scale-agnostic while still preserving dynamic spatial correlations through a holistic graph. Additionally, we introduce a novel Spatio-temporal Focal Learning strategy that incorporates a negative filter to resolve potential conflicts between contrastive learning and graph representation, and a focal contrastive loss as its core to guide the framework to focus on optimizing the expanding variables. To evaluate the effectiveness of STEV, we benchmark EVTSF performance on three real-world datasets from various domains and compare it against three potential solutions employing state-of-the-art (SOTA) MTSF models tailored for EVSTF. Experimental results show that STEV significantly outperforms its competitors, especially in handling expanding variables. Notably, STEV, with only 5% of observations during the expanding period, is on par with SOTA MTSF models trained with complete data. Further exploration of various expanding scenarios underscores the generalizability of STEV in real-world applications.
The remarkable prompting capability of large language models (LLMs) offers substantial convenience to users across diverse backgrounds. Nevertheless, as the sensitive information within prompts is inevitably exposed to LLMs, caution must be exercised to preserve privacy. Among various studies, text anonymization is considered an effective approach to preventing privacy leakage in prompts through text substitution. However, existing works overemphasize privacy while overlooks preserving contextual integrity, degrading semantic consistency. To address these concerns, this paper introduces a context-sensitive prompt privacy-preserving framework, namely Adaptive Linguistic Sanitization and Anonymization (ALSA). In specific, ALSA incorporates a three-dimensional scoring mechanism to dynamically quantify the substitutability of each word within a prompt by integrating the Privacy Leakage Risk Score (PLRS), the Contextual Information Importance Score (CIIS), and the Task Relevance Score (TRS). Subsequently, a clustering technique is adopted to dynamically determine the threshold for assigning an anonymization action (i.e., Retain, Replace, Encrypt, or Delete) by balancing privacy, semantics, and task relevance. Extensive experiments on five benchmark datasets validate the superiority of ALSA over state-of-the-art baselines in terms of accuracy, privacy preservation, and semantic integrity.
Knowledge tracing (KT), which is instrumental in monitoring and forecasting students' knowledge states throughout their learning trajectory in online learning environments, has over the past decade garnered widespread attention due to its pivotal role in facilitating personalized education. Existing KT approaches were mainly invented from the model-centric perspective to overcome the sequence modeling difficulty while not exploiting the potential information of sparsity, thereby limiting their performance. To make full use of the information in the dataset, this paper proposes a data-centric knowledge tracing paradigm, termed LPDG, aiming to generate interaction data between students and exercises by revealing students' Learning Patterns and facilitating the Generation of ideal training Data. Specifically, we propose a learning patterns-guided exercise sequence regenerator, which incorporates Transformer and a tailor-made pattern enhancer, thereby aiding in the extraction of valuable information for generating high-quality training data. Moreover, we devise a learning patterns-guided pseudo-label generator, which leverages the diffusion process to construct pseudo-labels for the regenerated sequences. Afterwards, the fully generated ideal data is incorporated into the training data, and we integrate this framework with various model-centric approaches in KT. Finally, experimental results across datasets clearly demonstrate the efficacy of our proposed LPDG framework.
This paper studies grid-free point process modeling under varying fluid parameters. Existing methods rely on grid-based approaches or fixed parameters, making it challenging to handle complex nonlinear dynamics and out-of-distribution (OOD) scenarios. To address this, we propose Adaptive Perturbation Graph ODE (AGODE), a novel framework that integrates three key innovations: (1) an adaptive conditioning mechanism for physical parameter adaptation(2) a continuous graph neural ODE for spatiotemporal evolution modeling, and (3) a perturbation module with mutual information maximization for uncertainty quantification. AGODE employs graph neural networks to encode unstructured point cloud data into latent dynamics governed by neural ODEs, where physical parameters are injected through context-aware conditioning vectors. The perturbation module generates diverse trajectory samples by introducing stochastic noise during ODE integration, while contrastive learning aligns predictions with physical contexts to filter implausible outcomes. Extensive experiments across five fluid dynamics benchmarks (Prometheus, Navier-Stokes, Spherical-SWE, 3D Reaction-Diffusion, ERA5) demonstrate AGODE's state-of-the-art performance. Specifically, AGODE achieves MSE of 0.0302/0.0312 (in-domain/OOD) on Prometheus, outperforming PURE by 6.5%/5.0%, and reduces Navier-Stokes errors by 28.1% compared to physics-informed NMO. Notably, AGODE maintains superior OOD generalization with only 2.9% average error increase versus 7.8% for baselines, while its uncertainty quantification improves prediction reliability by 41% (95% confidence interval coverage). These results validate AGODE's capabilities in continuous spatiotemporal modeling, multi-parameter adaptation, and robust uncertainty estimation for complex fluid systems.
Weather forecasting is essential but remains computationally intensive and physically incomplete in traditional numerical weather prediction (NWP) methods. Deep learning (DL) models offer efficiency and accuracy but often ignore physical laws, limiting interpretability and generalization. We propose PhyDL-NWP, a physics-guided deep learning framework that integrates physical equations with latent force parameterization into data-driven models. It predicts weather variables from arbitrary spatiotemporal coordinates, computes physical terms via automatic differentiation, and uses a physics-informed loss to align predictions with governing dynamics. PhyDL-NWP enables resolution-free downscaling by modeling weather as a continuous function and fine-tunes pre-trained models with minimal overhead, achieving up to 170× faster inference with only 55K parameters. Experiments show that PhyDL-NWP improves both forecasting performance and physical consistency.
Gradients as An Action: Towards Communication-Efficient Federated Recommender Systems via Adaptive Action Sharing
PDF ↗As a promising privacy-aware collaborative model training paradigm, Federated Learning (FL) is becoming popular in the design of distributed recommender systems. However, Federated Recommender Systems (FedRecs) greatly suffer from two major problems: i) extremely high communication overhead due to massive item embeddings involved in recommendation systems, and ii) intolerably low training efficiency caused by the entanglement of both heterogeneous network environments and client devices. Although existing methods attempt to employ various compression techniques to reduce communication overhead, due to the parameter errors introduced by model compression, they inevitably suffer from model performance degradation. To simultaneously address the above problems, this paper presents a communication-efficient FedRec framework named FedRAS, which adopts an action-sharing strategy to cluster the gradients of item embedding into a specific number of model updating actions for communication rather than directly compressing the item embeddings. In this way, the cloud server can use the limited actions from clients to update all the items. Since gradient values are significantly smaller than item embeddings, constraining the directions of gradients (i.e., the action space) introduces smaller errors compared to compressing the entire item embedding matrix into a reduced space. To accommodate heterogeneous devices and network environments, FedRAS incorporates an adaptive clustering mechanism that dynamically adjusts the number of actions. Comprehensive experiments on well-known datasets demonstrate that FedRAS can reduce the size of communication payloads by up to 96.88%, while not sacrificing recommendation performance within various heterogeneous scenarios. We have open-sourced FedRAS at https://github.com/mastlab-T3S/FedRAS.