The perceptions and decisions of individuals on social networks are deeply rooted in their intrinsic beliefs, which makes it possible to infer social beliefs from user behavior and message interactions. While existing research models these interactions as graphs and learns their representations, interpretability remains a significant challenge. In real-world scenarios, the interpretation of beliefs is nested within subject scopes of different granularity (such as topics and locations), posing additional challenges for belief discovery. In this paper, we introduce the Interpretable Graph Auto-Encoder Tree (IGAT), a novel end-to-end framework that jointly encodes hierarchical subject scopes and corresponding beliefs as a unified, interpretable hierarchical representation. IGAT integrates the interpretable hierarchy of Model Trees with disentangled representation learning models. We propose a differentiable Slice Mechanism to dynamically optimize internal node splitting and jointly train a leaf model to learn disentangled belief subspaces. The aggregation of these subspaces yields a unified representation, offering interpretations for both subjects and beliefs. Experimental evaluations on three real-world Twitter datasets show that IGAT achieves a consistent improvement of 1.49%-5.61% in F1-score, accuracy, and purity in the belief discovery task, as well as its effectiveness in various downstream analytical applications.
论文检索
输入标题、作者或关键词，从 2,578 篇学术成果中精准定位
Few-shot learning is crucial for tabular data, where the high cost of annotation often limits the availability of labeled samples. Despite its importance in real-world applications such as healthcare and finance, few-shot learning in tabular domains has received limited attention. To address this, we introduce range-limited augmentation, a novel augmentation strategy for contrastive learning that perturbs numerical features within predefined feature-specific ranges. Unlike conventional augmentations, which may result in false positive pairs during contrastive learning, our approach ensures semantic consistency by restricting augmentations to ranges. A quantitative analysis confirms that range-limited augmentation better preserves task-relevant information compared to existing augmentation techniques. Additionally, we propose FeSTa (Few-Shot Tabular classification benchmark), the first large-scale benchmark designed to systematically evaluate few-shot learning methods in tabular data. FeSTa includes 50 datasets and 32 algorithms spanning supervised, unsupervised, self-supervised, semi-supervised, and foundation models. Experiments on FeSTa show that range-limited augmentation consistently ranks among the top methods, achieving an average rank of 2.6 out of 32 in 1-shot classification, despite not relying on large-scale pretraining or complex architectures. The benchmark code is available in https://github.com/kyungeun-lee/festa.git.
Detecting social media bots is essential for maintaining the security and trustworthiness of social networks. While contemporary graph-based detection methods demonstrate promising results, their practical application is limited by label reliance and poor generalization capability across diverse communities. Generative Graph Self-Supervised Learning (GSL) presents a promising paradigm to overcome these limitations, yet existing approaches predominantly follow the homophily assumption and fail to capture the global patterns in the graph, which potentially diminishes their effectiveness when facing the challenges of interaction camouflage and distributed deployment in bot detection scenarios. To this end, we propose BotHP, a generative GSL framework tailored to boost graph-based bot detectors through heterophily-aware representation learning and prototype-guided cluster discovery. Specifically, BotHP leverages a dual-encoder architecture, consisting of a graph-aware encoder to capture node commonality and a graph-agnostic encoder to preserve node uniqueness. This enables the simultaneous modeling of both homophily and heterophily, effectively countering the interaction camouflage issue. Additionally, BotHP incorporates a prototype-guided cluster discovery pretext task to model the latent global consistency of bot clusters and identify spatially dispersed yet semantically aligned bot collectives. Extensive experiments on two real-world bot detection benchmarks demonstrate that BotHP consistently boosts graph-based bot detectors, improving detection performance, alleviating label reliance, and enhancing generalization capability.
Dynamic graphs offer more precise modeling of real-world applications compared to static graphs. Therefore, learning on dynamic graphs has garnered significant research attention in recent years. Unfortunately, the current approaches for learning dynamic graphs remain inadequate in capturing global spatial-temporal dependency. This issue arises from capturing biased spatial and temporal dependencies, thereby weakening the coupling between spatial and temporal dimensions. To overcome it, we propose a Self-Reliant Global Dependency Apprehending Framework on Dynamic Graphs, namely SGD-DyG. Specifically, we first design a frequency self-enhanced learning module that examines the global inherent interactions of the node features hidden in the frequency domain. Furthermore, we propose a global-local-mixed self-supervised learning module that maximizes spatial-temporal mutual information between local node and global graph embeddings. Extensive experiments on seven real-world dynamic graph datasets validate that the proposed SGD-DyG consistently exceeds state-of-the-art models.
Graph Alignment (GA) is an NP-hard combinatorial challenge. The existing methods usually work on small-scale graphs because their computational complexity grows as the square of the number of nodes. Recent techniques relied on alignment labels to divide the large source and target graphs into small ones respectively, which are then aligned separately. However, the alignment labels in expansive real-world graphs are often scarce, which makes GA even more challenging. To address these issues, we propose a novel self-supervised learning framework that is able to work on the graphs of million nodes without any alignment labels, where the existing methods usually fail to output an answer within a reasonable time. Our method also falls in the divide-and-conquer paradigm. We do not cluster the source and target graph separately by the existing graph clustering algorithms, because it is difficult to pair and then align the cluster-induced source and target subgraphs in the absence of alignment labels. We devise an Alignment-Aware Clustering (AAC) method to partition the source and target nodes jointly and keep the ought-to-be-aligned pairs of nodes within the same cluster as much as possible. Furthermore, we develop a Topology-Aware Repartition (TAR) that leverages pseudo-alignment labels as anchors for re-clustering, preserving detailed structural information within clusters. This enables graph neural networks to effectively enhance the graph representation learning for node alignments which are self-supervised by a graph-matching solver. Extensive experiments have demonstrated that our method greatly reduces computational demands and sustains high alignment accuracies in large-scale graph applications.
With the increasing computation of training graph neural networks (GNNs) on large-scale graphs, graph condensation (GC) has emerged as a promising solution to synthesize compact, substitute graphs of the large-scale original graphs for efficient GNN training. However, these condensed graphs are specifically designed for the node classification task, significantly limiting the versatility of the synthesized data across various downstream tasks. This limitation predominantly stems from the reliance of existing GC methods on classification as the surrogate task for optimization, which leads to an excessive dependence on node labels and restricts their utility in label-scarcity scenarios. More critically, this surrogate task tends to overfit class-specific information within the condensed graph, consequently restricting the generalization capabilities of GC for other downstream tasks. To address these challenges, we introduce Contrastive Graph Condensation (CTGC), which adopts a self-supervised surrogate task to extract critical, causal information from the original graph and enhance the cross-task generalizability of the condensed graph. Specifically, CTGC employs a dual-branch framework to disentangle the generation of the node attributes and graph structures, where a dedicated structural branch is designed to explicitly encode geometric information through nodes' positional embeddings. By implementing an alternating optimization scheme with contrastive loss terms, CTGC promotes the mutual enhancement of both branches and facilitates high-quality graph generation through the model inversion technique. Extensive experiments demonstrate that CTGC excels in handling various downstream tasks with a limited number of labels, consistently outperforming state-of-the-art GC methods.
Graph Contrastive Learning (GCL) emerged as a powerful paradigm in self-supervised graph representation learning. While earlier applications of GCL rely on homophily assumptions, spectral graph neural networks (GNNs) enhance the effectiveness of GCL on heterophilic graphs by incorporating both low-pass and high-pass filters. However, due to numerical considerations, existing approaches oversimplify low-pass and high-pass filters by modeling them as basic linear operations, failing to capture complex topological relationships. Here, we propose ArnoldiGCL, a novel algorithm that enables the application of complex spectral filters for Graph Contrastive Learning (GCL). Using Arnoldi orthonormalization-based Chebyshev interpolation, ArnoldiGCL overcomes the difficulties posed by ill-conditioned Vandermonde systems that arise in the modeling of complex filters. By introducing learnable filters, our method generates diverse spectral views and effectively captures nuanced graph structures. Theoretical analysis demonstrates that ArnoldiGCL accurately interpolates complex filters, thus forming a solid foundation for contrastive learning on graphs with complex structures. Extensive experiments on real-world datasets confirm that ArnoldiGCL significantly outperforms state-of-the-art GCL algorithms on both homophilic and heterophilic graphs, showcasing its robustness and versatility. The code for ArnoldiGCLis available as open source at: https://github.com/mustafaCoskunAgu/ArnoldiGCL
Recent advancements in time-series forecasting have highlighted the importance of frequency-domain modeling. However, deep learning models primarily operate in the time domain, limiting their ability to capture frequency-based patterns. Existing approaches normally introduce novel neural network architectures tailored to task-specific frequency properties, yet they often lack generalization and require extensive domain-specific adaptations. In this paper, we propose FAT, a novel pretraining framework that learns generalizable Frequency-Aware Time-series representations through self-supervised learning. The key idea of FAT is to pretrain any backbone model to directly extract generalizable frequency patterns from time-domain signals and encode them into robust representations-eliminating the need for architectural modifications or additional modules during inference. This is achieved via a frequency reformer that amplifies critical frequency components learned through self-supervision and enforces similarity between the original and frequency-reformed time-series representations produced by the encoder. In addition, recognizing that semantically equivalent time-series can exhibit different frequency expressions-analogous to how the same phrase is pronounced differently by different speakers-FAT introduces a Knowledge-Guided Frequency Reformer that unifies the expression of frequency patterns with the same underlying semantics and extends similarity constraints to frequency-invariant augmented samples to enhance robustness of learned representation. Experiments on 14 benchmark datasets across regression and classification tasks show that FAT consistently achieves state-of-the-art performance while maintaining robustness across diverse backbone models, significantly outperforming existing pretraining methods. Our code is available at https://github.com/JiaXiangfei/FAT.
Text-attributed graphs (TAGs) have emerged as a powerful representation for modeling complex relationships across diverse domains. With the rise of large language models (LLMs), there is growing interest in leveraging their capabilities for graph learning. However, current approaches face significant challenges in embedding structural information into LLM-compatible formats, requiring either computationally expensive alignment mechanisms or manual graph verbalization techniques that often lose critical structural details. Moreover, these methods typically require labeled data from source domains for effective transfer learning, significantly constraining their adaptability. We propose STAG, a novel self-supervised framework that directly quantizes graph structural information into discrete tokens using a frozen codebook. Unlike traditional quantization approaches, our method employs soft assignment and KL divergence guided quantization to address the unique challenges of graph data, which lacks natural tokenization structures. Our framework enables both LLM-based and traditional learning approaches, supporting true zero-shot transfer learning without requiring labeled data even in the source domain. Extensive experiments demonstrate state-of-the-art performance across multiple node classification benchmarks while maintaining compatibility with different LLM architectures, offering an elegant solution to bridging graph learning with LLMs.
Heterogeneous Information Networks (HINs), which consist of various types of nodes and edges, have recently witnessed excellent performance in graph mining. However, most existing heterogeneous graph neural networks (HGNNs) fail to simultaneously handle the problems of missing attributes, inaccurate attributes and scarce node labels, which limits their expressiveness. In this paper, we propose a generative self-supervised model GraMI to address these issues simultaneously. Specifically, GraMI first initializes all the nodes in the graph with a low-dimensional representation matrix. After that, based on the variational graph autoencoder framework, GraMI learns both node-level and attribute-level embeddings in the encoder, which can provide fine-grained semantic information to construct node attributes. In the decoder, GraMI reconstructs both links and attributes. Instead of directly reconstructing raw features for attributed nodes, GraMI generates the initial low-dimensional representation matrix for all the nodes, based on which raw features of attributed nodes are further reconstructed. In this way, GraMI can not only complete informative features for non-attributed nodes, but rectify inaccurate ones for attributed nodes. Finally, we conduct extensive experiments to show the superiority of GraMI in tackling HINs with missing and inaccurate attributes. Our code and data can be found here: https://github.com/See-r/GraMI.
Learning generalizable self-supervised graph representations for downstream tasks is challenging. To this end, Contrastive Learning (CL) has emerged as a leading approach. The embeddings of CL are arranged on a hypersphere where similarity is measured by the cosine distance. However, many real-world graphs, especially of hierarchical nature, cannot be embedded well in the Euclidean space. Although the hyperbolic embedding is suitable for hierarchical representation learning, naively applying CL to the hyperbolic space may result in the so-called dimension collapse, i.e., features will concentrate mostly within few density regions, leading to poor utilization of the whole feature space. Thus, we propose a novel contrastive learning framework to learn high-quality graph embeddings in hyperbolic space. Specifically, we design the alignment metric that effectively captures the hierarchical data-invariant information, as well as we propose a substitute of the uniformity metric to prevent the so-called dimensional collapse. We show that in the hyperbolic space one has to address the leaf- and height-level uniformity related to properties of trees. In the ambient space of the hyperbolic manifold these notions translate into imposing an isotropic ring density towards boundaries of Poincaré ball. Our experiments support the efficacy of our method.
Recent years have witnessed the perfect encounter of deep learning and quantitative trading has achieved great success in stock investment. Numerous deep learning-based models have been developed for forecasting stock returns, leveraging the powerful representation capabilities of neural networks to identify patterns and factors influencing stock prices. These models can effectively capture general patterns in the market, such as stock price trends, volume-price relationships, and time variations. However, the impact of special irrationality factors -- such as market sentiment, speculative behavior, market manipulation, and psychological biases -- has not been fully considered in existing deep stock forecasting models due to their relative abstraction as well as lack of explicit labels and data description. To fill this gap, we propose UMI, a Universal multi-level Market Irrationality factor model to enhance stock return forecasting. The UMI model learns factors that can reflect irrational behaviors in market from both individual stock and overall market levels. For the stock-level, UMI construct an estimated rational price for each stock, which is cointegrated with the stock's actual price. The discrepancy between the actual and the rational prices serves as a factor to indicate stock-level irrational events. Additionally, we define market-level irrational behaviors as anomalous synchronous fluctuations of stocks within a market. Using two self-supervised representation learning tasks, i.e., sub-market comparative learning and market synchronism prediction, the UMI model incorporates market-level irrationalities into a market representation vector, which is then used as the market-level irrationality factor. We also developed a forecasting model that captures both temporal and relational dependencies among stocks, accommodating the UMI factors. Extensive experiments on U.S. and Chinese stock markets with competitive baselines demonstrate our model's effectiveness and the universality of our factors in improving various forecasting models. We provide our code at https://github.com/lIcIIl/UMI.
With the explosive growth of online multimodal content, multimodal recommender systems(MRSs) have brought significant benefits to multimedia platforms. As MRSs evolve, many studies incorporate advanced technologies like graph neural networks(GNNs) and self-supervised learning(SSL), achieving remarkable results. However, these efforts still suffer from the quality disparity problem. It refers to the mixture of high and low quality across items' multiple modalities, owing to disparities in construction costs or design levels. These low-quality modalities often lack crucial details or introduce noise to the depiction of item, leading to insufficient or polluted item representation. Therefore, we propose a novel framework R2MR: Review and Rewrite Modality for Recommendation to tackle this issue. Specifically, R2MR is composed of two key components: Modality Reviewer and Modality Rewriter. The Modality Reviewer introduces a Consensus Review Mechanism. It performs perspective decomposition based on user representations and learns the consensus quality scores for modalities from diverse perspectives of multiple users. The Modality Rewriter proposes a Latent Mapping Model, which improves the quality of inferior modalities by learning various mapping patterns from high-quality modalities. Comprehensive experiments across three benchmark datasets reveal that R2MR substantially outperforms state-of-the-art methods, achieving an average improvement of 9.20%. The implementations are available at https://github.com/gutang-97/R2MR.
With the rise of e-commerce and short videos, online recommender systems that can capture users' interests and update new items in real-time play an increasingly important role. In both online and offline recommendation systems, the cold-start problem caused by interaction sparsity has been impacting the effectiveness of recommendations for cold-start items. Many cold-start scheme based on fine-tuning or knowledge transferring shows excellent performance on offline recommendation. Yet, these schemes are infeasible for online recommendation on streaming data pipelines due to different training method, computational overhead and time constraints. Inspired by the above questions, we propose a model-agnostic recommendation algorithm called Popularity-Aware Meta-learning (PAM), to address the item cold-start problem under streaming data settings. PAM divides the incoming data into different meta-learning tasks by predefined item popularity thresholds. The model can distinguish and reweight behavior-related and content-related features in each task based on their different roles in different popularity levels, thus adapting to recommendations for cold-start samples. These task-fixing design significantly reduces additional computation and storage costs compared to offline methods. Furthermore, PAM also introduced data augmentation and an additional self-supervised loss specifically designed for low-popularity tasks, leveraging insights from high-popularity samples. This approach effectively mitigates the issue of inadequate supervision due to the scarcity of cold-start samples. Experimental results across multiple public datasets demonstrate the superiority of our approach over other baseline methods in addressing cold-start challenges in online streaming data scenarios.
In the pursuit of enhancing software reusability and developer productivity, code search has emerged as a key area, aimed at retrieving code snippets relevant to functionalities based on natural language queries. Despite significant progress in self-supervised code pre-training utilizing the vast amount of code data in repositories, existing methods have primarily focused on leveraging contrastive learning to align natural language with function-level code snippets. These studies have overlooked the abundance of fine-grained (such as block-level and statement-level) code snippets prevalent within the function-level code snippets, which results in suboptimal performance across all levels of granularity. To address this problem, we first construct a multi-granularity code search dataset called MGCodeSearchNet, which contains 536K+ pairs of natural language and code snippets. Subsequently, we introduce a novel Multi-Granularity Self-Supervised contrastive learning code Search framework (MGS3). First, MGS3 features a Hierarchical Multi-Granularity Representation module (HMGR), which leverages syntactic structural relationships for hierarchical representation and aggregates fine-grained information into coarser-grained representations. Then, during the contrastive learning phase, we endeavor to construct positive samples of the same granularity for fine-grained code, and introduce in-function negative samples for fine-grained code. Finally, we conduct extensive experiments on code search benchmarks across various granularities, demonstrating that the framework exhibits outstanding performance in code search tasks of multiple granularities. These experiments also showcase its model-agnostic nature and compatibility with existing pre-trained code representation models.
Traffic prediction is essential for intelligent transportation systems and urban computing. It aims to establish a relationship between historical traffic data X and future traffic states Y by employing various statistical or deep learning methods. However, the relations of X → Y are often influenced by external confounders that simultaneously affect both X and Y, such as weather, accidents, and holidays. Existing deep-learning traffic prediction models adopt the classic front-door and back-door adjustments to address the confounder issue. However, these methods have limitations in addressing continuous or undefined confounders, as they depend on predefined discrete values that are often impractical in complex, real-world scenarios. To overcome this challenge, we propose the Spatial-Temporal sElf-superVised confoundEr learning (STEVE) model. This model introduces a basis vector approach, creating a base confounder bank to represent any confounder as a linear combination of a group of basis vectors. It also incorporates self-supervised auxiliary tasks to enhance the expressive power of the base confounder bank. Afterward, a confounder-irrelevant relation decoupling module is adopted to separate the confounder effects from direct X → Y relations. Extensive experiments across four large-scale datasets validate our model's superior performance in handling spatial and temporal distribution shifts and underscore its adaptability to unseen confounders. Our model implementation is available at https://github.com/bigscity/STEVE_CODE.
Foundation models like ChatGPT and GPT-4 have revolutionized artificial intelligence, exhibiting remarkable abilities to generalize across a wide array of tasks and applications beyond their initial training objectives. However, graph learning has predominantly focused on single-graph models, tailored to specific tasks or datasets, lacking the ability to transfer learned knowledge to different domains. This limitation stems from the inherent complexity and diversity of graph structures, along with the different feature and label spaces specific to graph data. In this paper, we recognize text as an effective unifying medium and employ Text-Attributed Graphs (TAGs) to leverage this potential. We present our UniGraph framework, designed to learn a foundation model for TAGs, which is capable of generalizing to unseen graphs and tasks across diverse domains. Unlike single-graph models that use pre-computed node features of varying dimensions as input, our approach leverages textual features for unifying node representations, even for graphs such as molecular graphs that do not naturally have textual features. We propose a novel cascaded architecture of Language Models (LMs) and Graph Neural Networks (GNNs) as backbone networks. Additionally, we propose the first pre-training algorithm specifically designed for large-scale self-supervised learning on TAGs, based on Masked Graph Modeling. We introduce graph instruction tuning using Large Language Models (LLMs) to enable zero-shot prediction ability. Our comprehensive experiments across various graph learning tasks and domains demonstrate the model's effectiveness in self-supervised representation learning on unseen graphs, few-shot in-context transfer, and zero-shot transfer, even surpassing or matching the performance of GNNs that have undergone supervised training on target datasets.
The domain of event sequences is widely applied in various industrial tasks in banking, healthcare, etc., where temporal tabular data processing is required. This paper introduces PyTorch-Lifestream, the first open-source library specially designed to handle event sequences. It supports scenarios with multimodal data and offers a variety of techniques for learning embeddings of event sequences and end-to-end model training. Furthermore, PyTorch-Lifestream efficiently implements state-of-the-art methods for event sequence analysis and adapts approaches from similar domains, thus enhancing the versatility and performance of sequence-based models for a wide range of applications, including financial risk scoring, campaigning, user ID matching, churn prediction, fraud detection, medical diagnostics, and recommender systems.
Self-supervised learning (SSL) has garnered increasing attention in electrocardiogram (ECG) analysis for its effectiveness in resource-limited settings. Existing state-of-the-art SSL methods rely on time-frequency detail reconstruction, but due to the inherent redundancy of ECG signals and individual variability, these approaches often yield suboptimal performance. In contrast, discrete label prediction becomes a superior pre-training objective by encouraging models to efficiently abstract ECG high-level semantics. However, the continuity and significant variability of ECG signals pose a challenge in generating semantically discrete labels. To address this issue, we propose an ECG pretraining framework with a self-distillation semantic tokenizer (ECG2TOK), which maps continuous ECG signals into discrete labels for self-supervised training. Specifically, the tokenizer extracts semantically aware embeddings of ECG by self-distillation and performs online clustering to generate semantically rich discrete labels. Subsequently, the SSL model is trained in conjunction with masking strategies and discrete label prediction to facilitate the abstraction of high-level semantic representations. We evaluate ECG2TOK in six downstream tasks, demonstrating that ECG2TOK efficiently achieves state-of-the-art performance and up to a 30.73% AUC increase in low-resource scenarios. Moreover, visualization experiments demonstrate that the discrete labels generated by ECG2TOK exhibit consistent semantics closely associated with clinical features. Our code is available on https://github.com/YXYanova/ECG2TOK.
SHIELD: A Self-supervised, Silicosis-focused Hierarchical Imaging Framework for Occupational Lung Disease Diagnosis
PDF ↗Silicosis is an irreversible lung disease caused by silica dust exposure in industrial settings. Early detection is crucial, but automatic diagnostic methods are hindered by limited data availability. We propose SHIELD - a self-supervised, Silicosis-focused Hierarchical Imaging framework for early occupational Lung disease Diagnosis. Our method leverages a multi-resolution jigsaw puzzle pretext task on CXR images to extract and preserve features for lung region analysis. By employing a pyramidal strategy to generate pretrained models at various resolutions, followed by fine-tuning and a two-level ensembling across diverse deep learning architectures, SHIELD achieves enhanced diagnostic accuracy. We validate our approach on a publicly collected CXR dataset of 3044 samples from public health centers in India. SHIELD achieves 72% accuracy, demonstrating up to 20% improvement over baseline approaches. This work advances medical image analysis and supports UN Sustainable Development Goal 3 by providing cost-effective early screening in resource-limited settings.