With the rapid proliferation of short-video platforms and content-driven social networks, sequential recommendation models capable of accurately capturing user interests have become increasingly crucial. Among these, Transformer-based sequential recommendation models have gained widespread adoption due to their superior ability. The positional encoding (PE) in Transformer architectures serves to incorporate positional information into sequences. However, relying solely on original absolute positional information may be insufficient for sequential recommendation models. In contrast, the dwell time after interactions (i.e., the time intervals between consecutive user interactions) provides a more accurate reflection of users' emotional responses and evolving interests. Despite its significance, this aspect has often been overlooked in existing works. To fully utilize this information, our work introduces an adaptive PE method, termed TSAPE (Temporal-Series-Aware Positional Encoding). This approach introduces an innovative modeling of the sequence of time intervals between user interactions, rather than the numerical values of the intervals themselves, thereby capturing real-time feedback on user interests and integrating it with conventional PE mechanisms. Furthermore, we employ multiple layers of one-dimensional convolutional networks and attention mechanisms to endow the features with adaptive capabilities across various time interval scenarios. This enables TSAPE to more accurately capture sequential positional information at any given moment. By enhancing the sequential order information of interactions, TSAPE significantly improves the accuracy of next-item recommendations. We seamlessly integrated our method into several Transformer-based sequential recommendation models and conducted comparisons with state-of-the-art sequential recommendation approaches and widely-used PE methods. The results demonstrate that the integration of TSAPE consistently outperforms the original backbone models and other SOTA methods. The SASRec model integrated with TSAPE achieves an average improvement of 15.61% across three evaluation metrics on four benchmark datasets. Our code has been made publicly available at https://github.com/rongbo-qi/TSAPE_Rec.
论文检索
输入标题、作者或关键词，从 361 篇学术成果中精准定位
Accurate forecasting of sequential data streams is a cornerstone of modern Web services, supporting applications such as traffic management, user behavior modeling, and online anomaly prevention. However, in many Web environments, new domains emerge rapidly and labeled history data is scarce, which makes zero-shot forecasting particularly challenging. Existing time-series pre-trained models (TSPMs) show promise but they lack the ability to dynamically incorporate external knowledge, while conventional retrieval-augmented generation (RAG) methods are rarely extended beyond text. In this work, we present QuiZSF, a retrieval-augmented forecasting framework that integrates search and forecasting for time series data. The framework performs search by retrieving structurally similar sequences from a large-scale time-series database, and it performs forecasting by integrating the retrieved knowledge into the target sequence. Specifically, QuiZSF introduces a ChronoRAG Base, a hierarchical tree-structured database that enables scalable and domain-aware retrieval, a Multi-grained Series Interaction Learner that captures fine- and coarse-grained dependencies between target and retrieved sequences, and a Model Cooperation Coherer that adapts retrieved knowledge to TSPMs. This design teaches models to actively perform search, align auxiliary information across modalities, and leverage it for more accurate forecasting. Extensive experiments on five public benchmarks demonstrate that QuiZSF consistently outperforms strong baselines, ranking first in up to 87.5% of zero-shot forecasting settings while maintaining high efficiency.
Deep learning has achieved strong results in modeling sequential data, including event sequences, temporal point processes, and irregular time series. Recently, transformers have largely replaced recurrent networks in these tasks. However, transformers often underperform recurrent networks in classification tasks that aim to predict future targets, such as churn, user reactions, or treatment response. The reason behind this performance gap remains largely underexplored. In this paper, we identify a key limitation of transformers: the lack of a single vector representation that compactly summarizes the evolving state of a sequence. We further show that commonly used contrastive embeddings are poorly suited to capturing the local context needed for accurate forward-looking prediction. To address these challenges, we introduce history tokens, a novel concept that enables the accumulation of historical information during next-token prediction pretraining. Our approach significantly improves transformer-based models, achieving impressive results in finance, e-commerce, and healthcare tasks. The code is publicly available: https://github.com/ivan-chai/pretpp.
Venous thromboembolism (VTE) is a common and life-threatening complication in cancer patients after treatment. Early risk assessment and detection of VTE primarily rely on clinical indicators, such as blood test results. However, existing studies are limited to static or snapshot-based models, failing to capture the evolving dynamics of disease progression, as deep time-series modeling is hindered by the lack of longitudinal clinical data. To address this gap, we introduce CliTsVTE, a large-scale clinical time-series dataset curated for VTE modeling, comprising 501,063 samples from 26,022 patients over seven years across nine cancer types. The dataset contains continuous time gaps between consecutive time points. Unlike many benchmarks, CliTsVTE reflects real-world clinical settings and presents unique challenges in continuous irregular time-series modeling with long-term irregularity and varying data granularity, which makes missingness significantly consequential. To tackle this, we propose a deep learning framework integrating multiple sequential backbones with an adversarially regularized autoencoder (ARAE) that learns latent representations to eliminate missingness. Experiments on CliTsVTE show that our best model achieves 88.7% accuracy and an AUC of 0.952, significantly outperforming traditional time-point models and regular time-series benchmarks. These results establish a strong benchmark for deep modeling of continuous irregularity in clinical time-series data and highlight the potential of AI-driven large-scale clinical datasets in solving real-world medical research challenges.
Emerging foundation models have spurred growing interest in task-unspecific time series modeling, which can accommodate data from diverse domains and support various tasks. However, most existing methods still suffer from poor adaptability and generalization across cross-domain time series with varying feature dimensionalities. This study achieves unified time series modeling to transcend task and data boundaries, inspired by the recent advances in visual foundation models. We propose a universal cross-modal Time series modeling method named TimeIG, featuring Tailored Temporal Imaging and Dependent Channel Grouping. Specifically, TimeIG adopts the novel dependent channel grouping method to project heterogeneous time series into hierarchical spaces to identify representative features and learn inter-dependencies across variables. Then, a tailored temporal imaging method is proposed to generate carefully hand-crafted images from time series, facilitating complementary temporal-spatial feature extraction. Finally, a dual-branch module is designed to simultaneously model the sequential and visual data, enabling cross-modal alignment via the proposed Cross-Modal Attention and Dynamic Weighted-Averaging. Extensive experiments on real-world cross-domain time series datasets show that TimeIG consistently outperforms existing SOTA methods. The code of TimeIG is publicly available at https://github.com/missCmj/TimeIG.
Stochastic sequential decision-making systems — such as Markov decision processes and their variants — are increasingly used in areas such as transportation, healthcare, and communication. However, the ability to explain these systems’ outputs to non-technical end users has not kept pace with their widespread adoption. This paper addresses that gap by extending prior work and presenting a unified framework for generating causal explanations of agent behavior in sequential decision-making settings, grounded in the structural causal model (SCM) paradigm. Our framework supports the generation of multiple, semantically distinct explanations for agent actions — capabilities that were previously unattainable. In addition to introducing a novel taxonomy of explanations for MDPs to guide empirical investigation, we develop both exact and approximate causal inference methods within the SCM framework. We analyze their applicability and derive run-time bounds for each. This leads to the proposed algorithm, MeanRESP, which operates flexibly across a spectrum of approximations tailored to external constraints. We further analyze the sample complexity and error rates of approximate MeanRESP, and provide a detailed comparison of its outputs — under varying definitions of responsibility — with popular Shapley-value-based methods. Empirically, we performed a series of experiments to evaluate the practicality and effectiveness of the proposed system, focusing on real-world computational demands and the validity and reliability of metrics for comparing approximate and exact causal methods. Finally, we present two user studies that reveal user preferences for certain types of explanations and demonstrate a strong preference for explanations generated by our framework compared to those from other state-of-the-art systems.
To achieve successful assistance with long-horizon web-based tasks, AI agents must be able to sequentially follow real-world user instructions over a long period. Unlike existing web-based agent benchmarks, sequential instruction following in the real world poses significant challenges beyond performing a single, clearly defined task. For instance, real-world human instructions can be ambiguous, require different levels of AI assistance, and may evolve over time, reflecting changes in the user's mental state. To address this gap, we introduce RealWebAssist, a novel benchmark designed to evaluate sequential instruction-following in realistic scenarios involving long-horizon interactions with the web, visual GUI grounding, and understanding ambiguous real-world user instructions. RealWebAssist includes a dataset of sequential instructions collected from real-world human users. Each user instructs a web-based assistant to perform a series of tasks on multiple websites. A successful agent must reason about the true intent behind each instruction, keep track of the mental state of the user, understand user-specific routines, and ground the intended tasks to actions on the correct GUI elements. Our experimental results show that state-of-the-art models struggle to understand and ground user instructions, posing critical challenges in following real-world user instructions for long-horizon web assistance.
IdealTSF: Can Non-Ideal Data Contribute to Enhancing the Performance of Time Series Forecasting Models?
PDF ↗Deep learning has shown strong performance in time series forecasting tasks. However, issues such as missing values and anomalies in sequential data hinder its further development in prediction tasks. Previous research has primarily focused on extracting feature information from sequence data or addressing these suboptimal data as positive samples for knowledge transfer. A more effective approach would be to leverage these non-ideal negative samples to enhance event prediction. In response, this study highlights the advantages of non-ideal negative samples and proposes the IdealTSF framework, which integrates both ideal positive and negative samples for time series forecasting. IdealTSF consists of three progressive steps: pretraining, training, and optimization. It first pretrains the model by extracting knowledge from negative sample data, then transforms the sequence data into ideal positive samples during training. Additionally, a negative optimization mechanism with adversarial disturbances is applied. Extensive experiments demonstrate that negative sample data unlocks significant potential within the basic attention architecture for time series forecasting. Therefore, IdealTSF is particularly well-suited for applications with noisy samples or low-quality data.
Artificial intelligence systems in critical fields like autonomous driving and medical imaging analysis often continually learn new tasks using a shared stream of input data. For instance, after learning to detect traffic signs, a model may later need to learn to classify traffic lights or different types of vehicles using the same camera feed. This scenario introduces a challenging setting we term Continual Multitask Learning (CMTL), where a model sequentially learns new tasks on an underlying data distribution without forgetting previously learned abilities. Existing continual learning methods often fail in this setting because they learn fragmented, task-specific features that interfere with one another. To address this, we introduce Learning with Preserving (LwP), a novel framework that shifts the focus from preserving task outputs to maintaining the geometric structure of the shared representation space. The core of LwP is a Dynamically Weighted Distance Preservation (DWDP) loss that prevents representation drift by regularizing the pairwise distances between latent data representations. This mechanism of preserving the underlying geometric structure allows the model to retain implicit knowledge and support diverse tasks without requiring a replay buffer, making it suitable for privacy-conscious applications. Extensive evaluations on time-series and image benchmarks show that LwP not only mitigates catastrophic forgetting but also consistently outperforms state-of-the-art baselines in CMTL tasks.
Multivariable time series forecasting methods can integrate information from exogenous variables, leading to significant prediction accuracy gains. The transformer architecture has been widely applied in various time series forecasting models due to its ability to capture long-range sequential dependencies. However, a naïve application of transformers often struggles to effectively model complex relationships among variables over time. To mitigate against this, we propose a novel architecture, termed Spectral Operator Neural Network (Sonnet). Sonnet applies learnable wavelet transformations to the input and incorporates spectral analysis using the Koopman operator. Its predictive skill relies on the Multivariable Coherence Attention (MVCA), an operation that leverages spectral coherence to model variable dependencies. Our empirical analysis shows that Sonnet yields the best performance on 34 out of 47 forecasting tasks with an average mean absolute error (MAE) reduction of 2.2% against the most competitive baseline. We further show that MVCA can remedy the deficiencies of naïve attention in various deep learning models, reducing MAE by 10.7% on average in the most challenging forecasting tasks.
SVGL: Scale-Variable Graph Learning in Model Space for Multivariate Time Series Classification
PDF ↗Multivariate time series classification (MTSC) has broad applications in numerous domains. Existing MTSC methods typically focus on either temporal dynamics or variable interactions of the data, often overlooking cross-scale couplings among different variables. To bridge this gap, we propose Scale-Variable Graph Learning (SVGL), a novel framework that effectively captures data-inherent scale-variable interactions for MTSC. SVGL begins with spectral analysis to adaptively identify key periodic scales for each variable. A period-aware reservoir computing network is then incorporated to fit the variable at these scales, encoding the sequential and periodic dynamics into multi-scale dynamic representations. Subsequently, we construct a scale-variable graph to model interactions of the encoded temporal dynamics, where nodes represent scale-variable pairs and edges denote their correlations. After sparsely initializing the graph via nearest neighbors, a parallel graph learning architecture is integrated in SVGL, combining global graph convolutional and sample-specific graph attention to aggregate effective features for classification. Extensive experiments on 30 UEA datasets demonstrate that SVGL outperforms state-of-the-art baselines in accuracy and maintains low training overhead.
Deep Learning · Sequential Models, Time series
Sequence learning is dominated by Transformers and parallelizable recurrent neural networks such as state-space models, yet learning long-term dependencies remains challenging, and state-of-the-art designs trade power consumption for performance. The Bistable Memory Recurrent Unit (BMRU) was introduced to enable hardware–software co-design of ultra-low power RNNs: quantized states with hysteresis provide persistent memory while mapping directly to analog primitives. However, BMRU performance lags behind parallelizable RNNs on complex sequential tasks. In this paper, we identify gradient blocking during state updates as a key limitation and propose a cumulative update formulation that restores gradient flow while preserving persistent memory, creating skip-connections through time. This leads to the Cumulative Memory Recurrent Unit (CMRU) and its relaxed variant, the $\alpha$CMRU. Experiments show that the cumulative formulation dramatically improves convergence stability and reduces initialization sensitivity. The CMRU and $\alpha$CMRU match the performance of Linear Recurrent Units (LRUs) and minimal Gated Recurrent Units (minGRUs) on standard benchmarks at small model sizes, while the CMRU retains quantized states, persistent memory, and noise-resilient dynamics essential for analog implementation.
Deep Learning · Sequential Models, Time series
Representing the past in a compressed, efficient, and informative manner is a central problem for systems trained on sequential data. The *HiPPO* framework, originally proposed by Gu & Dao et al., provides a principled approach to sequential compression by projecting signals onto orthogonal polynomial (OP) bases via structured linear ordinary differential equations. Subsequent works have embedded these dynamics in state space models (SSMs), where HiPPO structure serves as an initialization. Nonlinear successors of these SSM methods such as Mamba are state of the art for many tasks with long-range dependencies, but the mechanisms by which they represent and prioritize history remain largely implicit. In this work, we revisit the HiPPO framework with the goal of making these mechanisms explicit. We show how polynomial representations of history can be extended to support capabilities of modern SSMs such as adaptive allocation of memory, and input-dependent state updates, and associative memory, while retaining direct interpretability in the OP basis. We introduce a unified framework comprising five such extensions, which we collectively refer to as a "HiPPO zoo." Each extension exposes a specific modeling capability as an explicit modification of the underlying measure or dynamics governing the polynomial coefficients, rather than as an opaque learned transformation. The resulting models adapt their memory online and train in streaming settings with efficient updates. We illustrate the behaviors and advantages of these extensions through a range of synthetic sequence modeling tasks, highlighting how explicit polynomial memories can recover and clarify mechanisms implicit in SSMs.
Deep Learning · Sequential Models, Time series
Irregular Multivariate Time Series (IMTS) arise naturally in many real-world domains, yet their irregular sampling patterns pose significant challenges for effective modeling. Existing approaches for IMTS fall into two categories: architecture-based and data-based methods. Architecture-based methods require specialized modeling for IMTS, limiting reuse of established Multivariate Time Series (MTS) models, data-based methods convert IMTS into regular time series through imputation or interpolation, often introducing artificial values that distort temporal dynamics. In this work, we propose a novel input-embedding-based approach for modeling the IMTS. Our method preserves the original MTS backbone and operates directly on IMTS. We introduce QuITE (Query-based Irregular Time-series Embedding), a simple yet effective, backbone-agnostic embedding module that enables MTS models to directly process IMTS. QuITE leverages a set of learnable query tokens to aggregate irregular observations via a single self-attention layer, producing fixed-dimensional latent representations. Extensive experiments on real-world benchmarks demonstrate that QuITE consistently improves the performance of existing MTS models, achieving average relative performance gains up to 45.9% across diverse datasets and backbone architectures.
Deep Learning · Sequential Models, Time series
The AI community is rapidly converging on generalist foundation models trained on web-scale data. While this paradigm has yielded impressive gains, in this paper we argue that this objective needs to shift for enabling AI that reliably helps people in their daily activities. The most valuable systems will not be those that attempt do everything for everyone, but those that do the right things for a specific individual over a long period of time. We take the position that \emph{specialist models}---defined not by narrow task taxonomies, but by a tight coupling to an individual user and their local environment---represent the endgame for high-impact assistive AI. We substantiate this argument through three case studies: (i) AI agents that need to help humans automate daily web activities; (ii) wearable assistants that must predict actions in-context from continuous egocentric streams; and (iii) home robots that require helping humans in daily tasks with safety and compliance guarantees. In these settings, standard scaling assumptions are inverted: the most critical data is generated \emph{after deployment} as a streaming, on-policy interaction trace. We outline research directions for building specialists that learn from organic observational data, avoid self-reinforcing errors, and improve safely over long horizons.
Deep Learning · Sequential Models, Time series
In dynamical systems reconstruction (DSR) we aim to recover the dynamical system (DS) underlying observed time series. Specifically, we aim to learn a generative surrogate model which approximates the underlying, data-generating DS, and recreates its long-term properties (`climate statistics'). In scientific and medical areas, in particular, these models need to be mechanistically tractable -- through their mathematical analysis we would like to obtain insight into the recovered system's workings. Piecewise-linear (PL), ReLU-based RNNs (PLRNNs) have a strong track-record in this regard, representing SOTA DSR models while allowing mathematical insight by virtue of their PL design. However, all current PLRNN variants are *discrete-time maps*. This is in disaccord with the assumed continuous-time nature of most physical and biological processes, and makes it hard to accommodate data arriving at *irregular* temporal intervals. Neural ODEs are one solution, but they do not reach the DSR performance of PLRNNs and often lack their tractability. Here we develop theory for *continuous-time* PLRNNs (cPLRNNs): We present a novel algorithm for training and simulating such models, bypassing numerical integration by efficiently exploiting their PL structure. We further demonstrate how important topological objects like equilibria or limit cycles can be determined semi-analytically in trained models. We compare cPLRNNs to both their discrete-time cousins as well as Neural ODEs on DSR benchmarks, including systems with discontinuities which come with hard thresholds.
Deep Learning · Sequential Models, Time series
While Mamba2's expanded state dimension enhances temporal modeling, it incurs substantial inference overhead that saturates bandwidth during autoregressive generation. Standard pruning methods fail to address this bottleneck: unstructured sparsity leaves activations dense, magnitude-based selection ignores runtime dynamics, and gradient-based methods impose prohibitive costs. We introduce GHOST (Grouped Hidden-state Output-aware Selection and Truncation), a structured pruning framework that approximates control-theoretic balanced truncation using only forward-pass statistics. By jointly measuring controllability and observability, GHOST rivals the fidelity of gradient-based methods without requiring backpropagation. As a highlight, on models ranging from 130M to 2.7B parameters, our approach achieves a 50% state-dimension reduction with approximately 1 perplexity point increase on WikiText-2. Code is available at https://anonymous.4open.science/r/mamba2_ghost-7BCB/.
Deep Learning · Sequential Models, Time series
Recent years have seen a surge in data-driven surrogates for dynamical systems that can be orders of magnitude faster than numerical solvers. However, many machine learning-based models such as neural operators exhibit spectral bias, attenuating high-frequency components that often encode small-scale structure. This limitation is particularly damaging in applications such as weather forecasting, where misrepresented high frequencies can induce long-horizon instability. To address this issue, we propose multi-scale wavelet transformers (MSWTs), which learn system dynamics in a tokenized wavelet domain. The wavelet transform explicitly separates low- and high-frequency content across scales. MSWTs leverage a wavelet-preserving downsampling scheme that retains high-frequency features and employ wavelet-based attention to capture dependencies across scales and frequency bands. Experiments on chaotic dynamical systems show substantial error reductions and improved long-horizon spectral fidelity. On the ERA5 climate reanalysis, MSWTs further reduce climatological bias, demonstrating their effectiveness in a real-world forecasting setting.
General Machine Learning · Sequential, Network, and Time Series Modeling
This paper studies the problem of recovering a hidden vertex correspondence between two correlated graphs when both edge weights and node features are observed. While most existing work on graph alignment relies primarily on edge information, many real-world applications provide informative node features in addition to graph topology. To capture this setting, we introduce the featured correlated Gaussian Wigner model, where two graphs are coupled through an unknown vertex permutation, and the node features are correlated under the same permutation. We characterize the optimal information-theoretic thresholds for exact recovery and partial recovery of the latent mapping. On the algorithmic side, we propose QPAlign, an efficient method based on a quadratic programming relaxation, and demonstrate its strong empirical performance on both synthetic and real datasets. Moreover, we also derive theoretical guarantees for the proposed procedure, supporting its reliability and providing convergence guarantees.
General Machine Learning · Sequential, Network, and Time Series Modeling
Integrating Large Language Models (LLMs) into time series tasks has yielded impressive performance. While some works aim to enhance accuracy by explicitly designing step-by-step reasoning into prompts, such explicit Chain-of-Thought (CoT) approaches are difficult to generalize to time series. This is because it is difficult to clearly define the reasoning trajectories of time series. In addition, the high heterogeneity across time series often requires specialized prompt designs, limiting the model's scalability. To address these challenges, we propose **Time-CoT** (**Time** Series **C**hain-**o**f-**T**hought), a hierarchical reasoning framework based on temporal semantic codes for multivariate time series classification. This framework automatically constructs scenario-specific reasoning trajectories based on the characteristics of time series, thereby better eliciting the LLM's reasoning capability for time-series data. Specifically, Time-CoT, we first perform temporal representation pre-training with a multi-view temporal representation fusion to acquire high-quality temporal embeddings. We then discretize these temporal embeddings into hierarchical temporal semantic codes as the reasoning trajectory. Finally, the LLM predicts temporal semantic codes in a stepwise manner and then infers the final labels, thereby establishing a coarse-to-fine decision process. Experiments on ten public multivariate time series datasets demonstrate that the Time-CoT effectively adapts to diverse datasets and outperforms state-of-the-art methods. Our code is available at .