Continual Learning (CL) primarily aims to retain knowledge to prevent catastrophic forgetting and transfer knowledge to facilitate learning new tasks. Unlike traditional methods, we propose a novel perspective: CL not only needs to prevent forgetting, but also requires intentional forgetting. This arises from existing CL methods ignoring biases in real-world data, leading the model to learn spurious correlations that transfer and amplify across tasks. From feature extraction and prediction results, we find that data biases simultaneously reduce CL's ability to retain and transfer knowledge. To address this, we propose ErrorEraser, a universal plugin that removes erroneous memories caused by biases in CL, enhancing performance in both new and old tasks. ErrorEraser consists of two modules: Error Identification and Error Erasure. The former learns the probability density distribution of task data in the feature space without prior knowledge, enabling accurate identification of potentially biased samples. The latter ensures only erroneous knowledge is erased by shifting the decision space of representative outlier samples. Additionally, an incremental feature distribution learning strategy is designed to reduce the resource overhead during error identification in downstream tasks. Extensive experimental results show that ErrorEraser significantly mitigates the negative impact of data biases, achieving higher accuracy and lower forgetting rates across three types of CL methods. The code is available at https://github.com/diadai/ErrorEraser.
论文检索
输入标题、作者或关键词，从 844 篇学术成果中精准定位
Text-attributed graphs (TAGs) have emerged as a powerful representation for modeling complex relationships across diverse domains. With the rise of large language models (LLMs), there is growing interest in leveraging their capabilities for graph learning. However, current approaches face significant challenges in embedding structural information into LLM-compatible formats, requiring either computationally expensive alignment mechanisms or manual graph verbalization techniques that often lose critical structural details. Moreover, these methods typically require labeled data from source domains for effective transfer learning, significantly constraining their adaptability. We propose STAG, a novel self-supervised framework that directly quantizes graph structural information into discrete tokens using a frozen codebook. Unlike traditional quantization approaches, our method employs soft assignment and KL divergence guided quantization to address the unique challenges of graph data, which lacks natural tokenization structures. Our framework enables both LLM-based and traditional learning approaches, supporting true zero-shot transfer learning without requiring labeled data even in the source domain. Extensive experiments demonstrate state-of-the-art performance across multiple node classification benchmarks while maintaining compatibility with different LLM architectures, offering an elegant solution to bridging graph learning with LLMs.
Motif counting is fundamental in graph analytic tasks (e.g., clustering and recommendation) but #P-hard. Recent research has focused on exploring and applying deep learning-based solutions to tackle this problem. However, these solutions assume a deterministic graph where edge existence is certain, which may not hold due to the measurement and statistical prediction errors. Meanwhile, existing methods for uncertain graphs still face considerable time costs. To address the above issues, we propose UnG-MoCha, a novel deep-learning approach to efficiently count motifs in uncertain graphs. UnG-MoCha extracts representative subgraphs via graph structure learning and learns graph and motif representations using hierarchical and classic graph neural networks, respectively. Canonical correlation analysis is used to exploit correlations between graph and motif representations, boosting accuracy. Experiments on real-world graphs demonstrate UnG-MoCha's superior performance for scalable motif counting on uncertain graphs.
Large-scale graph data has demonstrated significant success in graph representation learning, but the associated high computational cost and inefficiency hinder its widespread adoption across diverse applications. Graph condensation has emerged as a promising solution to reduce time and memory demands while preserving generalization performance comparable to the original graph. Although existing graph condensation methods have proven effective, they are constrained by their reliance on repeatedly optimizing a condensed graph at a fixed scale, which demands significant computational resources and lacks flexibility to accommodate varying training requirements. This motivates us to explore alternative approaches that incrementally refine and expand condensed graphs. To tackle above challenges in graph condensation, this paper presents Progressive Stacking (ProStack), a method that progressively constructs condensed graphs by incrementally expanding their structure and semantics. ProStack utilizes graph memory to retain global structural and feature information from the original graph, facilitating the reuse of essential features and patterns. The method employs a memory-guided dual optimization approach, focusing on structural consistency in the feature generation stage and optimizing the alignment of the label features in the label prediction stage. By stage-wise graph stacking, ProStack progressively refines the condensed graph, ensuring feature coherence and maintaining structural integrity throughout the process. Experimental results demonstrate that ProStack outperforms existing graph condensation techniques in efficiency and effectiveness.
One-Shot Federated Learning (OSFL) restricts communication between the server and clients to a single round, significantly reducing communication costs and minimizing privacy leakage risks compared to traditional Federated Learning (FL), which requires multiple rounds of communication. However, existing OSFL frameworks remain vulnerable to distributional heterogeneity, as they primarily focus on model heterogeneity while neglecting data heterogeneity. To bridge this gap, we propose FedHydra, a unified, data-free, OSFL framework designed to effectively address both model and data heterogeneity. Unlike existing OSFL approaches, FedHydra introduces a novel two-stage learning mechanism. Specifically, it incorporates model stratification and heterogeneity-aware stratified aggregation to mitigate the challenges posed by both model and data heterogeneity. By this design, the data and model heterogeneity issues are simultaneously monitored from different aspects during learning. Consequently, FedHydra can effectively mitigate both issues by minimizing their inherent conflicts. We compared FedHydra with five SOTA baselines on four benchmark datasets. Experimental results show that our method outperforms the previous OSFL methods in both homogeneous and heterogeneous settings. The code is available at https://github.com/Jun-B0518/FedHydra.
Recently, full-graph Graph Neural Networks (GNNs) have gained prominence by addressing complex problems such as weather forecasting and material discovery. Existing full-graph training methods do not fully manage intermediate data generated during training and rely on rigid inter-GPU communication, limiting both training speed and scale. We propose FlexGNN, which fully manages intermediate data and adaptively performs inter-GPU communication by generating and optimizing best-effort training execution plans. Extensive experiments demonstrate that FlexGNN significantly outperforms existing full-graph GNN methods in both training speed and scale. Specifically, it is up to 5.4X faster than HongTu and up to 95.5X faster than NeutronStar.
Cryptographic approaches, such as secure multiparty computation, can be used to securely compute a function of a distributed graph without centralizing the data of each participant. However, the output of the protocol can leak sensitive information about the structure of the original graph. In particular, we propose an approach by which an adversary observing the result of a private protocol for the computation of the number of common neighbors between all pairs of vertices, can reconstruct the adjacency matrix of the graph. In fact, this can only be done up to co-squareness, a notion we introduce, as two different graphs can have the same matrix of common neighbors. To realize this, we consider two adversary models, one who observes the common neighbors matrix only and a more informed one that has partial knowledge of the original graph. Our results demonstrate that, from their common neighbors matrix, graphs can be reconstructed with high accuracy (up to co-squareness). The proposed reconstruction is also interesting in itself from the point of view of graph theory.
Prompt learning has emerged as a powerful paradigm for adapting vision-language models such as CLIP to downstream tasks. However, existing methods often overfit to seen data, leading to significant performance degradation when generalizing to novel classes or unseen domains. To address this limitation, we propose DiSa, a Directional Saliency-Aware Prompt Learning framework that integrates two complementary regularization strategies to enhance generalization. First, our Cross-Interactive Regularization (CIR) fosters cross-modal alignment by enabling cooperative learning between prompted and frozen encoders. Within CIR, a saliency-aware masking strategy guides the image encoder to prioritize semantically critical image regions, reducing reliance on less informative patches. Second, we introduce a directional regularization strategy that aligns visual embeddings with class-wise prototype features in a directional manner to prioritize consistency in feature orientation over strict proximity. This approach ensures robust generalization by leveraging stable prototype directions derived from class-mean statistics. Extensive evaluations on 11 diverse image classification benchmarks demonstrate that DiSa consistently outperforms state-of-the-art prompt learning methods across various settings, including base-to-novel generalization, cross-dataset transfer, domain generalization, and few-shot learning.
We analyze the power and limitations of a family of graph generative models, including the original Stochastic Kronecker Graph (SKG) models, often used to ensure privacy of network data. The SKG model cannot always generate graphs with degree distribution that follows a power-law or log-normal distribution (Seshadhri, Pinar, and Kolda, J. ACM 2013). As a result, variants of the SKG model have been proposed to generate graphs which approximately follow degree distributions, without any significant oscillations. But all existing solutions either require significant additional parameterization or have no provable guarantees on important properties, such as the degree distribution. In this work, we analyze a novel and efficient set of models, Relative Prime Stochastic Kronecker Graphs (RPSKG), which can get rid of oscillations in the degree distribution by mixing initiator matrices of relative prime dimensions without additional parameters. The use of 3x3 initiator matrices alone cannot eliminate significant oscillations. We prove that such matrices result in degree distribution that is bounded above by an exponential tail and thus cannot result in a power-law or log-normal.
Diversity maximization problem is a well-studied problem where the goal is to find k diverse items. Fair diversity maximization aims to select a diverse subset of k items from a large dataset, while requiring that each group of items be well represented in the output. More formally, given a set of items with labels, our goal is to find k items that maximize the minimum pairwise distance in the set, while maintaining that each label is represented within some budget. In many cases, one is only interested in selecting a handful (say a constant) number of items from each group. In such scenario we show that a randomized algorithm based on padded decompositions improves the state-of-the-art approximation ratio to √log(m) /(3m), where m is the number of labels. The algorithms work in several stages: (i) a preprocessing pruning which ensures that points with the same label are far away from each other(ii) a decomposition phase, where points are randomly placed in clusters such that there is a feasible solution with maximum one point per cluster and that any feasible solution will be diverse(iii) assignment phase, where clusters are assigned to labels, and a representative point with the corresponding label is selected from each cluster. We experimentally verify the effectiveness of our algorithm on large datasets.
Biomedical knowledge graphs (KGs) encode rich, structured information critical for drug discovery tasks, but extracting meaningful insights from large-scale KGs remains challenging due to their complex structure. Existing biomedical subgraph retrieval methods are tailored for graph neural networks (GNNs), limiting compatibility with other paradigms, including large language models (LLMs). We introduce K-Paths, a model-agnostic retrieval framework that extracts structured, diverse, and biologically meaningful multi-hop paths from dense biomedical KGs. These paths enable prediction of unobserved drug-drug and drug-disease interactions, including those involving entities not seen during training, thus supporting inductive reasoning. K-Paths is training-free and employs a diversity-aware adaptation of Yen's algorithm to extract the K shortest loopless paths between entities in a query, prioritizing biologically relevant and relationally diverse connections. These paths serve as concise, interpretable reasoning chains that can be directly integrated with LLMs or GNNs to improve generalization, accuracy, and enable explainable inference. Experiments on benchmark datasets show that K-Paths improves zero-shot reasoning across state-of-the-art LLMs. For instance, Tx-Gemma 27B improves by 19.8 and 4.0 F1 points on interaction severity prediction and drug repurposing tasks, respectively. Llama 70B achieves gains of 8.5 and 6.2 points on the same tasks. K-Paths also boosts the training efficiency of EmerGNN, a state-of-the-art GNN, by reducing the KG size by 90% while maintaining predictive performance. Beyond efficiency, K-Paths bridges the gap between KGs and LLMs, enabling scalable and explainable LLM-augmented scientific discovery. We release our code and the retrieved paths as a benchmark for inductive reasoning.
Massive data collection holds the promise of a better understanding of complex phenomena and ultimately, of better decisions. Representation learning has become a key driver of deep learning applications, since it allows learning latent spaces that capture important properties of the data without requiring any supervised annotations. While representation learning has been hugely successful in predictive tasks, it can fail miserably in causal tasks including predicting the effect of an intervention. This calls for a marriage between representation learning and causal inference. An exciting opportunity in this regard stems from the growing availability of multi-modal and interventional data (in medicine, advertisement, education, etc.). However, these datasets are still miniscule compared to the action spaces of interest in these applications (e.g. interventions can take on continuous values like the dose of a drug or can be combinatorial as in combinatorial drug therapies). In this talk, we will present a statistical and computational framework for causal representation learning from multi-modal data and its application towards optimal intervention design.
The rapid progress made over the last few years in generating linguistically coherent natural language has blurred, in the mind of many, the difference between natural language generation, understanding, knowledge retrieval and use, and the ability to reason with respect to the world. Nevertheless, reliably and consistently supporting high-level decisions that depend on natural language understanding and heterogenous information retrieval is still difficult, mostly, but not only, since most of these tasks are computationally more complex than language models can support. I will discuss some of the challenges underlying reasoning and information access and argue that we should exploit what LLMs do well while delegating responsibility to special purpose models and solvers for decision making. I will present some of our work in this space, focusing on supporting reasoning and information access in a range of quantitative, visual, and spatial reasoning tasks.
Language Models have been around for decades but have suddenly taken the world by storm. In a surprising third act for anyone doing Natural Language Processing (NLP) in the 70s, 80s, 90s, or 2000s, in much of the popular media, artificial intelligence is now synonymous with language models. In this talk, I want to take a look backward at where language models came from and why they were so slow to emerge, a look inward to give a few thoughts on meaning, intelligence, and what language models understand and know, and a look forward at some topics of recent research: the possibilities for progress with new versions of Universal Transformers and using Large Language Models (LLMs) to build intelligent language-using agents for the digital world. I will argue that material beyond language is not necessary to having meaning and understanding, but it is very useful in most cases, and that composability, adaptability, and learning are vital to intelligence. Rather than simply continuing to build ever-larger LLMs from huge passive collections of text data, we need to explore developing better neural architectures and agents that can learn through interactions. I will discuss how web agents can learn through interactions and how such an interaction-first learning approach can work very effectively in web environments with relatively small language models.
Broad adoption of Online Travel Platforms (OTPs) has led to increasing interest in accurately predicting users' hotel purchase behavior, with price being a key influencer in user decision-making and receiving significant focus. In examining the hotel purchasing process, we identify a pervasive trend that users make extensive price comparisons before making decisions. Existing research primarily focuses on a hotel's own price, neglecting the complex dynamics of market-driven price competition. In this paper, we propose the concept of Marketplace-oriented Hotel Price Competitiveness (MHPC) to model a hotel's pricing competitiveness within the marketplace. Being independent of specific user preferences, MHPC can be applied to and improve various downstream operations in the online hotel industry, such as hotel ranking and pricing, ultimately benefiting hoteliers, users, and OTPs. Furthermore, a novel Hotel Price Competitiveness-aware Purchase Prediction Model (HP3M) is constructed by incorporating MHPC and demand dynamics into a multi-task learning framework, featuring three distinct submodules to encompass the tri-dimensional facets of MHPC. Extensive offline and online experiments demonstrate HP3M's effectiveness in predicting hotel purchase probability and enhancing the performance of hotel ranking and pricing compared to the state-of-the-art methods. HP3M has been fully deployed on Fliggy, a leading OTP in China, serving thousands of hoteliers and tens of millions of users.
The vast pre-existing slides serve as rich and important materials to carry lecture knowledge. However, effectively leveraging lecture slides to serve students is difficult due to the multi-modal nature of slide content and the heterogeneous teaching actions. We study the problem of discovering effective designs that convert a slide into an interactive lecture. We develop Slide2Lecture, a tuning-free and knowledge-regulated intelligent tutoring system that can (1) effectively convert an input lecture slide into a structured teaching agenda consisting of a set of heterogeneous teaching actions; (2) create and manage an interactive lecture that generates responsive interactions catering to student learning demands while regulating the interactions to follow teaching actions. Slide2Lecture contains a complete pipeline for learners to obtain an interactive classroom experience to learn the slide. For teachers and developers, Slide2Lecture enables customization to cater to personalized demands. Slide2Lecture's online deployment has made more than 200K interactions with students in the 3K lecture sessions. We release our implementation at https://github.com/NewEduAI/Release.
The prevalence of unhealthy eating habits has become a growing concern in the United States. However, popular food recommendation platforms, such as Yelp, tend to prioritize users' dietary preferences over the healthiness of their choices. While some efforts have focused on developing health-aware food recommendation systems, personalization based on specific health conditions remains underexplored. Additionally, the lack of interpretability in these systems prevents users from evaluating the reliability of recommendations, limiting their practical adoption. To address these issues, we introduce two large-scale personalized health-aware food recommendation benchmarks at the first attempt. Building on this, we propose a novel framework called the Multi-Objective Personalized Interpretable Health-aware Food Recommendation System (MOPI-HFRS). This system generates food recommendations by jointly optimizing three objectives: user preference, personalized healthiness, and nutritional diversity. It also incorporates a reasoning module enhanced by large language models (LLMs) to provide interpretable recommendations that promote healthy dietary knowledge. The framework integrates descriptive features and health data using two structure learning and pooling modules within a graph learning framework. Pareto optimization is applied to balance the multi-faceted objectives. To further enhance healthy dietary knowledge, the system leverages LLMs by infusing knowledge from the recommendation model, generating meaningful interpretations for the recommendations. Extensive experiments on the proposed benchmarks demonstrate that MOPI-HFRS outperforms state-of-the-art methods by delivering diverse, healthy food recommendations alongside reliable explanations.
Time series forecasting is important in finance domain. Financial time series (TS) patterns are influenced by both short-term public opinions and medium-/long-term policy and market trends. Hence, processing multi-period inputs becomes crucial for accurate financial time series forecasting (TSF). However, current TSF models either use only single-period input, or lack customized designs for addressing multi-period characteristics. In this paper, we propose a Multi-period Learning Framework (MLF) to enhance financial TSF performance. MLF considers both TSF's accuracy and efficiency requirements. Specifically, we design three new modules to better integrate the multi-period inputs for improving accuracy: (i) Inter-period Redundancy Filtering (IRF), that removes the information redundancy between periods for accurate self-attention modeling, (ii) Learnable Weighted-average Integration (LWI), that effectively integrates multi-period forecasts, (iii) Multi-period self-Adaptive Patching (MAP), that mitigates the bias towards certain periods by setting the same number of patches across all periods. Furthermore, we propose a Patch Squeeze module to reduce the number of patches in self-attention modeling for maximized efficiency. MLF incorporates multiple inputs with varying lengths (periods) to achieve better accuracy and reduces the costs of selecting input lengths during training. The codes and datasets are available at https://github.com/Meteor-Stars/MLF.
Predictive Maintenance (PDM) systems are essential for preemptive monitoring of sensor signals to detect potential machine component failures in industrial assets such as bearings in rotating machinery. Existing PDM systems face two primary challenges: 1) Irregular Signal Acquisition, where data collection from the sensors is intermittent, and 2) Signal Heterogeneity, where the full spectrum of sensor modalities is not effectively integrated. To address these challenges, we propose a Curriculum Learning Framework for Multi-Modal Predictive Maintenance - MentorPDM. MentorPDM consists of 1) a graph-augmented pretraining module that captures intrinsic and structured temporal correlations across time segments via a temporal contrastive learning objective and 2) a bi-level curriculum learning module that captures task complexities for weighing the importance of signal modalities and samples via modality and sample curricula. Empirical results from MentorPDM show promising performance with better generalizability in PDM tasks compared to existing benchmarks. The efficacy of the MentorPDM model will be further demonstrated in real industry testbeds and platforms.
Large-scale human mobility data contains rich spatial and temporal information for urban sensing, crowd flow modeling, and urban planning. However, it is usually difficult to access wide-coverage, long-term, and consistent-time human mobility data. Most of the publicly available datasets are actually only records of discontinuous trajectories of a very small portion of urban citizens in asynchronous time due to the limited usage of apps for location data collection or the limited number of volunteers. To address this problem and empower open urban research, this paper constructs a high-quality human mobility dataset by generating large-scale citizen trajectories based on massive cellular signaling data. Particularly, we first propose a heatmap diffusion module to generate a probability heatmap that produces plausible trajectories at both the individual and city scales. Then, we propose a masked trajectory AutoEncoder, which can generate individual trajectory embeddings from partially given or empty trajectories. Third, a flexible framework is provided to incorporate the heatmap diffusion module with the masked trajectory embeddings, demonstrating significant flexibility in handling both fully masked trajectories for city-wide analysis and partially masked trajectories for specific locations. We have conducted extensive experiments to validate the utility of the regenerated trajectories at both individual and region levels for various applications. Numerous case studies further illustrate that our model learns not only the distribution of the trajectories but also the semantics of different urban areas. In summary, this paper provides a Heatmap Diffusion framework based on a Masked Trajectory AutoEncoder to regenerate flexible trajectories for open urban research. Correspondingly, we will try to open a large-scale human mobility data service for open urban research. Further information can be found at https://github.com/Rising0321/FinalOpenUR.