Forecasting complex time series is an important yet challenging problem that involves various industrial applications. Recently, masked time-series modeling has been proposed to effectively model temporal dependencies for forecasting by reconstructing masked segments from unmasked ones. However, since the semantic information in time series is involved in intricate temporal variations generated by multiple time series components, simply masking a raw time series ignores the inherent semantic structure, which may cause MTM to learn spurious temporal patterns present in the raw data. To capture distinct temporal semantics, we show that masked modeling techniques should address entangled patterns through a decomposition approach. Specifically, we propose ST-MTM, a masked time-series modeling framework with seasonal-trend decomposition, which includes a novel masking method for the seasonal-trend components that incorporates different temporal variations from each component. ST-MTM uses a period masking strategy for seasonal components to produce multiple masked seasonal series based on inherent multi-periodicity and a sub-series masking strategy for trend components to mask temporal regions that share similar variations. The proposed masking method presents an effective pre-training task for learning intricate temporal variations and dependencies. Additionally, ST-MTM introduces a contrastive learning task to support masked modeling by enhancing contextual consistency among multiple masked seasonal representations. Experimental results show that our proposed ST-MTM achieves consistently superior forecasting performance compared to existing masked modeling, contrastive learning, and supervised forecasting methods.
论文检索
输入标题、作者或关键词，从 844 篇学术成果中精准定位
Score-based causal discovery methods can effectively identify causal relationships by evaluating candidate graphs and selecting the one with the highest score. One popular class of scores is kernel-based generalized score functions, which can adapt to a wide range of scenarios and work well in practice because they circumvent assumptions about causal mechanisms and data distributions. Despite these advantages, kernel-based generalized score functions pose serious computational challenges in time and space, with a time complexity of O (n3) and a memory complexity of O (n2), where n is the sample size. In this paper, we propose an approximate kernel-based generalized score function with O (n) time and space complexities by using low-rank technique and designing a set of rules to handle the complex composite matrix operations required to calculate the score, as well as developing sampling algorithms for different data types to benefit the handling of diverse data types efficiently. Our extensive causal discovery experiments on both synthetic and real-world data demonstrate that compared to the state-of-the-art method, our method can not only significantly reduce computational costs, but also achieve comparable accuracy, especially for large datasets.
Multivariate time series forecasting is crucial for various applications, such as financial investment, energy management, weather forecasting, and traffic optimization. However, accurate forecasting is challenging due to two main factors. First, real-world time series often show heterogeneous temporal patterns caused by distribution shifts over time. Second, correlations among channels are complex and intertwined, making it hard to model the interactions among channels precisely and flexibly. In this study, we address these challenges by proposing a general framework called DUET, which introduces DU al clustering on the temporal and channel dimensions to Enhance multivariate Time series forecasting. First, we design a Temporal Clustering Module (TCM) that clusters time series into fine-grained distributions to handle heterogeneous temporal patterns. For different distribution clusters, we design various pattern extractors to capture their intrinsic temporal patterns, thus modeling the heterogeneity. Second, we introduce a novel Channel-Soft-Clustering strategy and design a Channel Clustering Module (CCM), which captures the relationships among channels in the frequency domain through metric learning and applies sparsification to mitigate the adverse effects of noisy channels. Finally, DUET combines TCM and CCM to incorporate both the temporal and channel dimensions. Extensive experiments on 25 real-world datasets from 10 application domains, demonstrate the state-of-the-art performance of DUET.
The time-index models are a class of time series forecasting models that map time-index features to forecasts in continuous space. Compared to the historical-value models, the time-index models can avoid the effect of data sampling frequency and are usually more expressive. However, the vanilla deep time-index model is weak in modeling the high-frequency components of time series and often requires the introduction of many parameters to enhance the modeling capability. Moreover, the time-index model learns only a mapping relationship and ignores the sequence relationship between temporal features, leading to a weak extrapolation capability in the forecast horizon. In this paper, inspired by the ability of quantum implicit neural representations to model the high-frequency components of signals with fewer parameters, we propose Quantum Time-Index Models with Reservoir (QuantumTime). Specifically, we introduce variational quantum circuits to address the challenge of representing high-frequency components in time series. Then, we introduce a reservoir that empowers QuantumTime with powerful extrapolation capabilities by exploiting the rich dynamical properties of reservoir computing. Ultimately, experiments conducted on chaotic datasets and various real-world datasets demonstrate that QuantumTime achieves highly competitive results compared to the state-of-the-art deep time-index model while reducing training parameters by at least 95%. Our approach provides a paradigm for utilizing potential quantum advantage in practical tasks.
The problem of online label shift, where label distribution evolves over time while the label-conditional density remains unchanged, has attracted increasing attentions. Although existing approaches have achieved sound theoretical guarantees and encouraging performance, the assumption of an unchanged conditional distribution may limit its application in broader tasks. In this paper, we investigate an extended variant named generalized online label shift (GOLS) problem, in which we relax the label shift assumption on the raw feature space and instead assume the existence of an unknown invariant representation such that conditional distribution of this representation given the label remains constant. To handle GOLS, our main idea involves capturing the inherently stable information from non-stationary streams, in the form of learning an invariant representation. Specifically, we design a novel objective to learn the invariant representation, which exploits the unique structure in GOLS. To optimize this objective, we propose an algorithm employing online ensemble paradigm to perform multi-resolution updates using various historical data windows, thereby enhancing the stability of the representation. This approach is theoretically guaranteed to achieve an optimal convergence rate. To improve the efficiency of the ensemble framework, we further propose a mask-based implementation for ensembling with DNNs. Experiments on benchmarks and real-world tasks validate the effectiveness of our approach.
Knowledge-intensive tasks often require complex reasoning and contextual understanding over long contexts. However, the learning and deployment of long-LLMs remains a challenging problem despite recent progresses. In this work, we propose that the short LLMs have great potentiality for solving knowledge-intensive tasks that have long context, i.e. they can be solved by purely working with oracle short-contexts within the input long-context. On top of this argument, we propose a framework called DCISO DynamiC knowledge-Intensive task S>Olver), which enables a short-LLM to address the knowledge-intensive tasks with long context via dynamic context browsing. In our framework, the short-LLM prompts itself to reason for two critical decisions: 1) how to access to the appropriate part of context within the input, 2) how to make effective use of the accessed context. By adaptively accessing and utilizing the context based on the presented tasks, DCISO can serve as a general framework to handle diversified knowledge-intensive long-context problems. We comprehensively evaluate different types of tasks from popular long-context benchmarks, where DCISO is able to achieve a substantially improved performance. Our codes will be released at this repository.
In recent years, there has been a surge in research on dynamic graph representation learning, primarily focusing on modeling the evolution of temporal-spatial patterns in real-world applications. However, within the domain of discrete-time dynamic graphs, the exploration of temporal edges remains underexplored. Existing approaches often rely on additional sequential models to capture dynamics, leading to high computational and memory costs, particularly for large-scale graphs. To address this limitation, we propose the Input Snapshots Fusion based Dynamic Graph Neural Network (SFDyG), which combines Hawkes processes with graph neural networks to capture temporal and structural patterns in dynamic graphs effectively. By fusing multiple snapshots into a single temporal graph, SFDyG decouples computational complexity from the number of snapshots, enabling efficient full-batch and mini-batch training. Experimental evaluations on eight diverse dynamic graph datasets for future link prediction tasks demonstrate that SFDyG consistently outperforms existing methods.
The two-tower model has become prevalent in recommender systems for its computational efficiency and robust predictive capabilities. The model usually employs two independent neural networks to encode user and item data separately, and predicts the similarity score with inner product or cosine functions, depending on which the Top-k ranked item list is generated. The optimization process typically involves a multi-label classification objective, often guided by surrogate loss functions like Softmax and One-vs-All (OvA), to enhance the recommendation performance. Despite both Softmax and OvA losses being Bayes-consistent, empirical observations reveal a significant performance gap in evaluation metrics, suggesting limitations in Bayes-consistency for analyzing loss effectiveness. To address this, we introduce ℋ-consistency into the discussion, which provides non-asymptotic and hypothesis-specific guarantees for Top-k classification within the two-tower model's hypothesis space. Through theoretical analysis, we demonstrate that Softmax and Cosine Contrastive Loss exhibit ℋ-consistency, while the OvA loss does not, explaining the observed performance discrepancies. Our findings bridge the gap between theoretical properties and practical outcomes, offering deeper insights into the optimization of two-tower models and contributing to the development of more effective recommendation systems.
Extreme Classification (XC) aims to map a query to the most relevant documents from a very large document set. XC algorithms used in real-world applications typically learn this mapping from datasets curated from implicit feedback, such as user clicks. However, these datasets often suffer from missing labels. In this work, we observe that systematic missing labels lead to missing knowledge, which is critical for modelling relevance between queries and documents. We formally show that this absence of knowledge is hard to recover using existing methods such as propensity weighting and data imputation strategies that solely rely on the training dataset. While Large Language Models (LLMs) provide an attractive solution to augment the missing knowledge, leveraging them in applications with low latency requirements and large document sets is challenging. To mitigate missing knowledge at scale, we propose SKIM (Scalable Knowledge Infusion for Missing Labels), an algorithm that leverages a combination of Small Language Models or SLMs, e.g., Llama2-7b, and abundant unstructured meta-data to effectively address the missing label problem. We show the efficacy of our method on large-scale public datasets through a combination of unbiased evaluation strategies, such as exhaustive human annotations and simulation-based evaluation benchmarks. SKIM outperforms existing methods on Recall@100 by more than 10 absolute points. Additionally, SKIM scales to proprietary query-ad retrieval datasets containing 10 million documents, outperforming baseline methods by 12% in offline evaluations and increasing ad click-yield by 1.23% in an online A/B test conducted on Bing Search. We release the code and trained models at: github.com/bicycleman15/skim
Out-of-distribution (OOD) detection has emerged as a pivotal approach for enhancing the reliability of machine learning models, considering the potential for test data to be sampled from classes disparate from in-distribution (ID) data employed during model training. Detecting those OOD data is typically realized as a distance measurement problem, where those deviating far away from the training distribution in the learned feature space are considered OOD samples. Advanced works have shown great success in learning with prototypes for feature-based OOD detection methods, where each ID class is represented with single or multiple prototypes. However, modeling with a finite number of prototypes would fail to maximally capture intra-class variations. In view of this, this paper extends the existing prototype-based learning paradigm to an infinite setting. This motivates us to design two feasible formulations for the Distributional Prototype Learning (DPL) objective, where, to avoid intractable computation and exploding parameters caused by the infinity nature, our key idea is to model an infinite number of discrete prototypes of each ID class with a class-wise continuous distribution. We theoretically analyze both alternatives, identifying the more stable-converging version of the learning objective. We show that, by sampling prototypes from a mixture of class-conditioned Gaussian distributions, the objective can be efficiently computed in a closed form without resorting to the computationally expensive Monte-Carlo approximation of the involved expectation terms. Extensive evaluations across mainstream OOD detection benchmarks empirically manifest that our proposed DPL has established a new state-of-the-art in various OOD settings.
Path enumeration is a fundamental problem and has been extensively studied in the literature. Given two query vertices and a weight threshold, the problem aims to identify all simple paths with weight not exceeding the threshold. Existing studies on path enumeration include DFS-based solutions and join-based solutions, where the join-based solutions only work for unweighted graphs. In this paper, we are the first to propose a join-based framework for weighted graphs. By observing the characteristics of DFS, we design a series of novel data structures and operations based on the join-based framework. In this way, our final solution combines the advantages of both join and DFS. We conduct experiments on several real large graphs. For weighted graphs, our method is much more efficient than existing algorithms. For unweighted graphs, our method is still competitive compared with the state-of-the-art solution which only works for unweighted graphs.
Electron-Informed Coarse-Graining Molecular Representation Learning for Real-World Molecular Physics
PDF ↗Various representation learning methods for molecular structures have been devised to accelerate data-driven chemistry. However, the representation capabilities of existing methods are essentially limited to atom-level information, which is not sufficient to describe real-world molecular physics. Although electron-level information can provide fundamental knowledge about chemical compounds beyond the atom-level information, obtaining the electron-level information in real-world molecules is computationally impractical and sometimes infeasible. We propose a method for learning electron-informed molecular representations without additional computation costs by transferring readily accessible electron-level information about small molecules to large molecules of our interest. The proposed method achieved state-of-the-art prediction accuracy on extensive benchmark datasets containing experimentally observed molecular physics. The source code for HEDMoL is available at https://github.com/ngs00/HEDMoL.
We address the problem of detecting data glitches in ML training sets, specifically mislabeled and anomalous samples. Detection of data glitches provides insights into the quality of the data sampling. Their repair may improve the reliability and the performance of the model. The proposed methodology is based on exploiting influence functions that estimate how much the loss of the model (or a given sample) is affected when a sample is removed from the training set. We introduce three novel signals for detecting, characterizing, and repairing data glitches in a training set based on sample influences. Influence-based signals form an explainable-by-design data glitch detection framework, producing intuitively explainable signals of the actual predictive model built. In contrast, specialized algorithms that are agnostic to the target ML model (e.g., anomaly detectors) replicate the work of fitting the data distribution and may detect glitches that are inconsistent with the decision boundary of the predictive model. Computational experiments on tabular and image data modalities demonstrate that the proposed signals outperform, in some cases up to a factor of 6, all existing influence-based signals, and generalize across different datasets and ML models. In addition, they often outperform specialized glitch detectors (e.g., mislabeled and anomaly detectors) and provide accurate label repairs for mislabeled samples.
Data-centric methods have shown great potential in understanding and predicting spatiotemporal dynamics, enabling better design and control of the object system. However, deep learning models often lack interpretability, fail to obey intrinsic physics, and struggle to cope with the various domains. While geometry-based methods, e.g., graph neural networks (GNNs), have been proposed to further tackle these challenges, they still need to find the implicit physical laws from large datasets and rely excessively on rich labeled data. In this paper, we herein introduce the conservation-informed GNN (CiGNN), an end-to-end explainable learning framework, to learn spatiotemporal dynamics based on limited training data. The network is designed to conform to the general conservation law via symmetry, where conservative and non-conservative information passes over a multiscale space enhanced by a latent temporal marching strategy. The efficacy of our model has been verified in various spatiotemporal systems based on synthetic and real-world datasets, showing superiority over baseline models. Results demonstrate that CiGNN exhibits remarkable accuracy and generalizability, and is readily applicable to learning for prediction of various spatiotemporal dynamics in a spatial domain with complex geometry.
Link prediction is a fundamental problem in graph data. In its most realistic setting, the problem consists of predicting missing or future links between random pairs of nodes from the set of disconnected pairs. Graph Neural Networks (GNNs) have become the predominant framework for link prediction. GNN-based methods treat link prediction as a binary classification problem and handle the extreme class imbalance---real graphs are very sparse---by sampling (uniformly at random) a balanced number of disconnected pairs not only for training but also for evaluation. However, we show that the reported performance of GNNs for link prediction in the balanced setting does not translate to the more realistic imbalanced setting and that simpler topology-based approaches are often better at handling sparsity. These findings motivate Gelato, a similarity-based link-prediction method that applies (1) graph learning based on node attributes to enhance a topological heuristic, (2) a ranking loss for addressing class imbalance, and (3) a negative sampling scheme that efficiently selects hard training pairs via graph partitioning. Experiments show that Gelato outperforms existing GNN-based alternatives.
Black-box attacks have emerged as a significant threat to deep neural networks. This challenge is particularly difficult in discrete sequential data compared to continuous data. Recently, the Blockwise Bayesian Attack (BBA) leveraging discrete Bayesian optimization with an adapted RBF kernel has gained prominence as a cutting-edge solution. However, it relies solely on alignment information (i.e., positional differences) within the RBF kernel, which may not fully capture the information (such as statistical, structural, and semantic information) inherent in discrete sequential data and potentially lacks the desired inductive bias necessary to approximate the target function accurately. To overcome this limitation, this paper proposes a novel bilevel Bayesian optimization approach to adaptively learn a hybrid space that better captures the similarity between discrete sequences. Specifically, we introduce a multi-kernel mechanism that incorporates multiple types of information, creating a more comprehensive similarity measure. Moreover, we develop a bilevel Bayesian optimization algorithm, where the outer-level objective determines the optimal weights of the multiple kernels, while the inner-level objective identifies the optimal adversarial sequence. Extensive experiments conducted on discrete sequential data demonstrate that our approach ensures secure multi-kernel selection and achieves a higher attack success rate with only a few additional queries, compared to BBA and other traditional optimization strategies.
In Deep Neural Networks (DNNs), manipulating gradients is central to various algorithms, including data subset selection and instance attribution. For better tractability, practitioners often resort to using only the gradients of the last layer as a heuristic, instead of the full gradient across all model parameters, which we show is detrimental due to the Support Vector Effect (SVE). We introduce SVE, a max-margin-like behavior in the last layer(s) of DNNs and employ it to thoroughly scrutinize prevalent data selection and attribution methods relying on last layer gradients. Our investigation exposes limitations in these techniques and not only provides explanations for previously observed pitfalls, like lack of diversity and temporal performance degradation, but also offers fresh insights, including the vulnerability of existing methods to basic poisoning attacks and the potential for competitive performance using much simpler alternatives. Based on insights from SVE, we craft new methods RandE and PAE for data subset selection and instance attribution, respectively, which often outperform the purported state-of-the-art at a fraction of the cost, emphasizing the practical advantages of more efficient and less complex approaches.
We focus on the dueling bandits problem, which has recently drawn significant attention due to its wide-ranging applications in online recommendation systems and the alignment of large language models (LLMs), considers an online preference learning scenario where the learner iteratively selects arms based on pairwise comparison feedback to infer user preferences. Two primary objectives are typically considered in dueling bandits: Regret Minimization (RM), which aims to improve the overall quality of selected arms over time, and Best Arm Identification (BAI), which seeks to efficiently identify the best item with minimal user feedback. For instance, RM is exemplified by the objective of consistently providing high-quality items, while BAI reduces the required human feedback by minimizing the number of necessary comparisons. Conventional research treats RM and BAI as two conflicting objectives, optimizing one at the expense of the other. In this paper, we propose a novel framework that demonstrates the near-consistency of RM and BAI in dueling bandits by reducing the BAI in dueling bandits into a sequential noisy identification problem. Based on our formulation, we propose a black-box reduction technique that transforms any RM algorithm into a BAI algorithm, and prove that such reduction with optimal RM algorithm achieves optimal sample complexity and nearly-optimal cumulative weak regret simultaneously. Our proposed algorithm acheives a nearly-optimal BAI sample complexity and attains a cumulative weak regret that is order-wise equivalent to the best-known result simultaneously. Experiments on both synthetic benchmarks and real-world online recommendation tasks validate the effectiveness of the proposed method, providing empirical evidences for our theoretical findings.
Computerized Adaptive Testing (CAT) aims to select the most appropriate questions based on the examinee's ability and is widely used in online education. However, existing CAT systems often lack initial understanding of the examinee's ability, requiring random probing questions. This can lead to poorly matched questions, extending the test duration and negatively impacting the examinee's mindset, a phenomenon referred to as the Cold Start with Insufficient Prior (CSIP) task. This issue occurs because CAT systems do not effectively utilize the abundant prior information about the examinee available from other courses on online platforms. These response records, due to the commonality of cognitive states across different knowledge domains, can provide valuable prior information for the target domain. However, no prior work has explored solutions for the CSIP task. In response to this gap, we propose Diffusion Cognitive States TransfeR Framework (DCSR), a novel domain transfer framework based on Diffusion Models (DMs) to address the CSIP task. Specifically, we construct a cognitive state transition bridge between domains, guided by the common cognitive states of examinees, encouraging the model to reconstruct the initial ability state in the target domain. To enrich the expressive power of the generated data, we analyze the causal relationships in the generation process from a causal perspective. Redundant and extraneous cognitive states can lead to limited transfer and negative transfer effects. Therefore, we designed three decoupling strategies to control confounding variables, thereby blocking backdoor paths that hinder causal discovery. Given that excessive uncertainty can affect the applicability of generated results to the CAT system, we propose consistency constraint and task-oriented constraint to control the randomness of the generated results and their relevance to the CAT task, respectively. Our DCSR can seamlessly apply the generated initial ability states in the target domain to existing question selection algorithms, thus improving the cold start performance of the CAT sys- tem. Extensive experiments conducted on five real-world datasets demonstrate that DCSR significantly outperforms existing baseline methods in addressing the CSIP task.
Spatio-temporal forecasting is a critical component of various smart city applications, such as transportation optimization, energy management, and socio-economic analysis. Recently, several automated spatio-temporal forecasting methods have been proposed to automatically search the optimal neural network architecture for capturing complex spatio-temporal dependencies. However, the existing automated approaches suffer from expensive neural architecture search overhead, which hinders their practical use and the further exploration of diverse spatio-temporal operators in a finer granularity. In this paper, we propose AutoSTF, a decoupled automatic neural architecture search framework for cost-effective automated spatio-temporal forecasting. From the efficiency perspective, we first decouple the mixed search space into temporal space and spatial space and respectively devise representation compression and parameter-sharing schemes to mitigate the parameter explosion. The decoupled spatio-temporal search not only expedites the model optimization process but also leaves new room for more effective spatio-temporal dependency modeling. From the effectiveness perspective, we propose a multi-patch transfer module to jointly capture multi-granularity temporal dependencies and extend the spatial search space to enable finer-grained layer-wise spatial dependency search. Extensive experiments on eight datasets demonstrate the superiority of AutoSTF in terms of both accuracy and efficiency. Specifically, our proposed method achieves up to 13.48x speed-up compared to state-of-the-art automatic spatio-temporal forecasting methods while maintaining the best forecasting accuracy. The source code and data are available at https://github.com/usail-hkust/AutoSTF.