Contrastive learning (CL) has achieved remarkable success in addressing data sparsity issues in collaborative filtering (CF) for recommender systems (RSs). The key principle is to generate different augmented views given a user-item interaction graph. However, prior endeavors mainly focus on performing augmentation via stochastic functions, e.g., by injecting perturbations into different hidden dimensions uniformly. Without fine control, the hidden representations of augmentations may contain noisy dimensions that are harmful to CL and irrelevant to RSs. Removing dimension-specific noise is a challenging task due to the following two major bottlenecks. It is difficult to (i) distinguish different dimensions' efficacy for CL and (ii) bridge the semantic gap between CL and RSs. Overlooking these limitations may cause redundant, false-positive, and irrelevant noise in hidden dimensions of the augmented views. In this paper, we solve the above challenges from the perspective of robust learning and curriculum learning, and propose a novel Dimension-aware augmentation in Ceontrastive Leearning for recommendation (DimCL). In DimCL, we first theoretically analyze the easiness and hardness of different dimensions for CL and RSs. With thorough analysis, we propose two propositions, which reveal that the gradients of different dimensions of augmented views are potentially related to the learning difficulty of optimizing CL and RSs. The comparison of gradients can provide detectable signals to reflect the efficacy of different dimensions for CL and the semantic gap between CL and RSs. Based on the analysis results, we devise three denoising factors, which can help DimCL to identify hard-to-learn dimensions as redundant or false-positive noise and pinpoint dimensions in different augmented views with inconsistent difficulties of RSs as irrelevant noise without requiring additional supervised labels. After denoising, DimCL can remove dimension-level noise to reduce unnecessary difficulty, making CL easier and maintaining more consistent difficulty in RSs. Extensive experiments on four public datasets demonstrate DimCL's superiority and flexible applications over various traditional and CL-based CF methods. The source code is publicly available online at https://github.com/zc-97/DimCL.
论文检索
输入标题、作者或关键词，从 844 篇学术成果中精准定位
Multi-view clustering has attracted increasing attention in recent years. However, most existing multi-view clustering approaches are performed in a purely unsupervised manner, while ignoring the valuable weak supervision information that can be obtained (e.g., active query) in many real applications. This paper considers the weak pairwise constraints among samples to enhance the clustering performance, and proposes a Semi-supervised Multi-view Clustering method with Active Constraints, SMCAC for short. SMCAC consists of two stages, clustering (C-stage) and active query (A-stage). In the C-stage, we design a tensor based multi-view graph learning model equipped with sample pairwise constraints regularization to facilitate the discriminative graph learning and fusion. An effective optimization algorithm based on alternating direction minimization is devised to solve the clustering model. In the A-stage, the most uncertain or difficult sample pairs are actively selected to query the constraints, based on the divergence of multi-view similarities learned in the C-stage. The two processes alternate iteratively until the maximum number of queries is reached. Extensive experiments on several popular datasets well validate the effectiveness of the proposed method.
In real-world recommendation scenarios, users engage with items through various types of behaviors. Leveraging diversified user behavior information for learning can enhance the recommendation of target behaviors (e.g., buy), as demonstrated by recent multi-behavior methods. The mainstream multi-behavior recommendation framework consists of two steps: fusion and prediction. Recent approaches utilize graph neural networks for multi-behavior fusion and employ multi-task learning paradigms for joint optimization in the prediction step, achieving significant success. However, these methods have limited perspectives on multi-behavior fusion, which leads to inaccurate capture of user behavior patterns in the fusion step. Moreover, when using multi-task learning for prediction, the relationship between the target task and auxiliary tasks is not sufficiently coordinated, resulting in negative information transfer. To address these problems, we propose a novel multi-behavior recommendation framework based on the combinatorial optimization perspective, named COPF. Specifically, we treat multi-behavior fusion as a combinatorial optimization problem, imposing different constraints at various stages of each behavior to restrict the solution space, thus significantly enhancing fusion efficiency (COGCN). In the prediction step, we improve both forward and backward propagation during the generation and aggregation of multiple experts to mitigate negative transfer caused by differences in both feature and label distributions (DFME). Comprehensive experiments on three real-world datasets indicate the superiority of COPF. Further analyses also validate the effectiveness of the COGCN and DFME modules. Our code is available at https://github.com/1918190/COPF.
Graph Neural Networks (GNNs) have been proven effective in modeling graph data, mostly depending on the in-distribution assumption. While in the out-of-distribution (OOD) scenarios, especially for the more challenging node-level task, the feature and structure distribution shifts between training and test nodes lead to performance degradation. To improve node-level OOD generalization, typical approaches introduce graph augmentation to enrich the training environments and conduct invariant learning to learn stable representations across various augmented environments. However, their graph augmentations emphasize diversity but neglect the preservation of invariant patterns which are fundamental to invariant learning. Moreover, most of them simply conduct the classic invariant learning objective but lack the consideration of the graph-specific structure information. Therefore, to mitigate their weakness, we propose a Structure-aware Invariant learning framework for Node-level Graph OOD generalization (SING). Specifically, we develop the invariance constraint regularization terms during the optimization of augmentations. Additionally, we define the structure embedding to elucidate the structural property and design the structure embedding alignment loss to optimize the augmentations and the invariant representations. By introducing the structure information, we further integrate the unique structural property into invariant learning, thereby boosting the invariant message-passing GNNs. The extensive experiments on the transductive GOOD benchmark and the inductive datasets empirically validate our superior OOD generalization performance to baselines.
The widespread application of pre-trained language models (PLMs) in natural language processing (NLP) has led to increasing concerns about their explainability. Selective rationalization is a self-explanatory framework that selects human-intelligible input subsets as rationales for predictions. Recent studies have shown that applying existing rationalization frameworks to PLMs will result in severe degeneration and failure problems, producing sub-optimal or meaningless rationales. Such failures severely damage trust in rationalization methods and constrain the application of rationalization techniques on PLMs. In this paper, we find that the homogeneity of tokens in the sentences produced by PLMs is the primary contributor to these problems. To address these challenges, we propose a method named Pre-trained Language Model's Rationalization (PLMR), which splits PLMs into a generator and a predictor to deal with NLP tasks while providing interpretable rationales. The generator in PLMR also alleviates homogeneity by pruning irrelevant tokens, while the predictor uses full-text information to standardize predictions. Experiments conducted on two widely used datasets across multiple PLMs demonstrate the effectiveness of the proposed method PLMR in addressing the challenge of applying selective rationalization to PLMs. Codes: https://github.com/ylb777/PLMR.
The core challenge of de novo protein design lies in creating proteins with specific functions or properties, guided by certain conditions. Current models explore to generate protein using structural and evolutionary guidance, which only provide indirect conditions concerning functions and properties. However, textual annotations of proteins, especially the annotations for protein domains, which directly describe the protein's high-level functionalities, properties, and their correlation with target amino acid sequences, remain unexplored in the context of protein design tasks. In this paper, we propose Protein-Annotation Alignment Generation PAAG, a multi-modality protein design framework that integrates the textual annotations extracted from protein database for controllable generation in sequence space. Specifically, within a multi-level alignment module, PAAG can explicitly generate proteins containing specific domains conditioned on the corresponding domain annotations, and can even design novel proteins with flexible combinations of different kinds of annotations. Our experimental results underscore the superiority of the aligned protein representations from PAAG over 7 prediction tasks. Furthermore, PAAG demonstrates a significant increase in generation success rate (24.7% vs 4.7% in zinc finger, and 54.3% vs 22.0% in the immunoglobulin domain) in comparison to the existing model. We anticipate that PAAG will broaden the horizons of protein design by leveraging the knowledge from between textual annotation and proteins.
Graphs are ubiquitous for modeling complex relationships between objects across various fields. Graph neural networks (GNNs) have become a mainstream technique for graph-based applications, but their performance heavily relies on abundant labeled data. To reduce labeling requirement, pre-training and prompt learning has become a popular alternative. However, most existing prompt methods do not distinguish between homophilic and heterophilic characteristics in graphs. In particular, many real-world graphs are non-homophilic-neither strictly nor uniformly homophilic-as they exhibit varying homophilic and heterophilic patterns across graphs and nodes. In this paper, we propose ProNoG, a novel pre-training and prompt learning framework for such non-homophilic graphs. First, we examineexisting graph pre-training methods, providing insights into the choice of pre-training tasks. Second, recognizing that each node exhibits unique non-homophilic characteristics, we propose a conditional network to characterize node-specific patterns in downstream tasks. Finally, we thoroughly evaluate and analyze ProNoG through extensive experiments on ten public datasets.
Many modern recommender systems represent user and item attributes as embedding vectors, relying on them for accurate recommendations. However, entangled embeddings often capture not only intrinsic property factors (e.g., user interest in item property) but also popularity factors (e.g., user conformity to item popularity) indistinguishably. These embeddings, influenced by popularity distribution, may face challenges when the popularity distribution at test time differs from historical distribution. Existing remedies in the literature involve disentangled embedding learning, which aims to separately capture intrinsic and popularity factors, demonstrating plausible generalization during popularity distribution shifts. However, we highlight that these methods often overlook a crucial aspect of popularity shifts-their temporal nature-in both training and inference phases. To address this, we propose Temporal Popularity distribution shift generalizABle recommender system (TPAB), a novel disentanglement framework incorporating temporal popularity. TPAB introduce a new (1) temporal-aware embedding design for users and items. Within this design, (2) popularity coarsening and (3) popularity bootstrapping are proposed to enhance generalization further. We also provide theoretical analysis showing that the bootstrapping loss eliminates the effect of popularity on the learned model. During inference, we infer test-time popularity and corresponding embeddings, using them alongside property embeddings for prediction. Extensive experiments on real-world datasets validate TPAB, showcasing its outstanding generalization ability during temporal popularity distribution shifts.
N-ary relational facts represent semantic correlations among more than two entities. While recent studies have developed link prediction (LP) methods to infer missing relations for knowledge graphs (KGs) containing n-ary relational facts, they are generally limited to transductive settings. Fully inductive settings, where predictions are made on previously unseen entities, remain a significant challenge. As existing methods are mainly entity embedding-based, they struggle to capture entity-independent logical rules. To fill in this gap, we propose an n-ary subgraph reasoning framework for fully inductive link prediction (ILP) on n-ary relational facts. This framework reasons over local subgraphs and has a strong inductive inference ability to capture n-ary patterns. Specifically, we introduce a novel graph structure, the n-ary semantic hypergraph, to facilitate subgraph extraction. Moreover, we develop a subgraph aggregating network, NS-HART, to effectively mine complex semantic correlations within subgraphs. Theoretically, we provide a thorough analysis from the score function optimization perspective to shed light on NS-HART's effectiveness for n-ary ILP tasks. Empirically, we conduct extensive experiments on a series of inductive benchmarks, including transfer reasoning (with and without entity features) and pairwise subgraph reasoning. The results highlight the superiority of the n-ary subgraph reasoning framework and the exceptional inductive ability of NS-HART.
The integration of large language models (LLMs) with external content has enabled applications such as Microsoft Copilot but also introduced vulnerabilities to indirect prompt injection attacks. In these attacks, malicious instructions embedded within external content can manipulate LLM outputs, causing deviations from user expectations. To address this critical yet under-explored issue, we introduce the first benchmark for bindirect prompt injection attacks, named BIPIA, to assess the risk of such vulnerabilities. Using BIPIA, we evaluate existing LLMs and find them universally vulnerable. Our analysis identifies two key factors contributing to their success: LLMs' inability to distinguish between informational context and actionable instructions, and their lack of awareness in avoiding the execution of instructions within external content. Based on these findings, we propose two novel defense mechanisms -- boundary awareness and explicit reminder -- to address these vulnerabilities in both black-box and white-box settings. Extensive experiments demonstrate that our black-box defense provides substantial mitigation, while our white-box defense reduces the attack success rate to near-zero levels, all while preserving the output quality of LLMs. We hope this work inspires further research into securing LLM applications and fostering their safe and reliable use. Our code is available at https://github.com/microsoft/BIPIA.
Fairness in federated learning has emerged as a critical concern, aiming to develop an unbiased model among groups (e.g., male or female) of diverse sensitive features. However, there is a trade-off between model performance and fairness, i.e., improving model fairness will decrease model performance. Existing approaches have characterized such a trade-off by introducing hyperparameters to quantify client's preferences for model fairness and model performance. Nevertheless, these approaches are limited to scenarios where each client has only a single pre-defined preference, and fail to work in practical systems where each client generally has multiple preferences. To this end, we propose a Preference-aware scheme in Fair Federated Learning (called PraFFL) to generate preference-specific models in real time. PraFFL can adaptively adjust the model based on each client's preferences to meet their needs. We theoretically prove that PraFFL can offer the optimal model tailored to an arbitrary preference of each client, and show its linear convergence. Experimental results show that our proposed PraFFL outperforms six fair federated learning algorithms in terms of the model's capability of adapting to clients' different preferences. Our implementation is available at https://github.com/rG223/PraFFL.
Graph Neural Networks (GNNs) have demonstrated remarkable proficiency in handling a range of graph analytical tasks across various domains, such as e-commerce and social networks. Despite their versatility, GNNs face significant challenges in transferability, limiting their utility in real-world applications. Existing research in GNN transfer learning overlooks discrepancies in distribution among various graph datasets, facing challenges when transferring across different distributions. How to effectively adopt a well-trained GNN to new graphs with varying feature and structural distributions remains an under-explored problem. Taking inspiration from the success of Low-Rank Adaptation (LoRA) in adapting large language models to various domains, we propose GraphLoRA, an effective and parameter-efficient method for transferring well-trained GNNs to diverse graph domains. Specifically, we first propose a Structure-aware Maximum Mean Discrepancy (SMMD) to align divergent node feature distributions across source and target graphs. Moreover, we introduce low-rank adaptation by injecting a small trainable GNN alongside the pre-trained one, effectively bridging structural distribution gaps while mitigating the catastrophic forgetting. Additionally, a structure-aware regularization objective is proposed to enhance the adaptability of the pre-trained GNN to target graph with scarce supervision labels. Extensive experiments on eight real-world datasets demonstrate the effectiveness of GraphLoRA against fourteen baselines by tuning only 20% of parameters, even across disparate graph domains. The code is available at https://github.com/AllminerLab/GraphLoRA.
Large-scale human mobility exhibits spatial and temporal patterns that can assist policymakers in decision making. Although traditional prediction models attempt to capture these patterns, they are often affected by nonperiodic public events, such as disasters and occasional celebrations. Since regular human mobility patterns are affected by these events, estimating their causal effects is critical to accurate mobility predictions. News articles provide unique perspectives on these events, though processing them is a challenge. In this study, we propose a causality based prediction model, CausalMob, to analyze the causal effects of public events. We first utilize large language models (LLMs) to extract human intentions from news and transform them into features that act as causal treatments. Next, the model learns representations of spatio-temporal regional covariates from multiple data sources to serve as confounders for causal inference. Finally, we present a causal effect estimation framework to ensure that event features remain independent of confounders during prediction. Based on large-scale real-world data, the experimental results show that the proposed model excels in human mobility prediction, outperforming state-of-the-art models.
In partial label learning (PLL), every sample is associated with a candidate label set comprising the ground-truth label and several noisy labels. The conventional PLL assumes the noisy labels are randomly generated (instance-independent), while in practical scenarios, the noisy labels are always instance-dependent and are highly related to the sample features, leading to the instance-dependent partial label learning (IDPLL) problem. Instance-dependent noisy label is a double-edged sword. On one side, it may promote model training as the noisy labels can depict the sample to some extent. On the other side, it brings high label ambiguity as the noisy labels are quite undistinguishable from the ground-truth label. To leverage the nuances of IDPLL effectively, for the first time we create class-wise embeddings for each sample, which allow us to explore the relationship of instance-dependent noisy labels, i.e., the class-wise embeddings in the candidate label set should have high similarity, while the class-wise embeddings between the candidate label set and the non-candidate label set should have high dissimilarity. Moreover, to reduce the high label ambiguity, we introduce the concept of class prototypes containing global feature information to disambiguate the candidate label set. Extensive experimental comparisons with twelve methods on six benchmark data sets, including four fine-grained data sets, demonstrate the effectiveness of the proposed method. The code implementation is publicly available at https://github.com/Yangfc-ML/CEL.
A fundamental problem in many science domains is learning the causal structure of a system from observed data. The observed data canonically come from multiple environments (i.e. different times, locations, and measurements), and causal models may have unobserved shifts. Although the causal graphs can be identified by modeling the distribution changes among different environments, existing solutions can only learn causal structures when given environmental information. In contrast, we propose a causal discovery approach (CausalSME) which automatically identifies pseudo environments and unobserved distribution shifts. Specifically, CausalSME learns a causal model containing unobserved variables, which can correct the distribution shifts with mixed environments. The heart of CausalSME is a variational autoencoder that infers shifted causal effects of unobserved variables and guides the identification of environment information. It further divides the shifted samples by the identified environments to jointly learn an invariant causal model. We prove the structure identifiability of CausalSME with the causal additive model. In our extensive experiments we show that CausalSME achieves state-of-the-art performance.
Recent years have witnessed the perfect encounter of deep learning and quantitative trading has achieved great success in stock investment. Numerous deep learning-based models have been developed for forecasting stock returns, leveraging the powerful representation capabilities of neural networks to identify patterns and factors influencing stock prices. These models can effectively capture general patterns in the market, such as stock price trends, volume-price relationships, and time variations. However, the impact of special irrationality factors -- such as market sentiment, speculative behavior, market manipulation, and psychological biases -- has not been fully considered in existing deep stock forecasting models due to their relative abstraction as well as lack of explicit labels and data description. To fill this gap, we propose UMI, a Universal multi-level Market Irrationality factor model to enhance stock return forecasting. The UMI model learns factors that can reflect irrational behaviors in market from both individual stock and overall market levels. For the stock-level, UMI construct an estimated rational price for each stock, which is cointegrated with the stock's actual price. The discrepancy between the actual and the rational prices serves as a factor to indicate stock-level irrational events. Additionally, we define market-level irrational behaviors as anomalous synchronous fluctuations of stocks within a market. Using two self-supervised representation learning tasks, i.e., sub-market comparative learning and market synchronism prediction, the UMI model incorporates market-level irrationalities into a market representation vector, which is then used as the market-level irrationality factor. We also developed a forecasting model that captures both temporal and relational dependencies among stocks, accommodating the UMI factors. Extensive experiments on U.S. and Chinese stock markets with competitive baselines demonstrate our model's effectiveness and the universality of our factors in improving various forecasting models. We provide our code at https://github.com/lIcIIl/UMI.
Temporal hypergraph representation learning is a concept that integrates high-order structure learning with temporal dynamics, enabling more accurate analysis of temporal and high-order interactions. To enhance model expressiveness, the latest work samples multi-hop hyperedge-centric neighbors directly from temporal hypergraphs and encodes them for high-order structure learning, achieving promising performance. Such modeling, however, incurs prohibitive computational complexity, which increases exponentially with model depth and quadratically with average hyperedge cardinality, thereby limiting model scalability. In this paper, we propose FastHeP, a fast and accurate approach for temporal hyperedge prediction, which can handle large temporal hypergraphs. The key idea is to minimize computational complexity while maintaining model expressiveness. Concretely, we design an online hyperedge-centric neighbor store, which can store time-aware and redundancy-aware neighbors for nodes with rational theoretical guarantees. Upon the neighbor store, we propose a novel hybrid message passing to model temporal high-order structures, theoretically preserving strong expressive power. This explicitly learns local high-order structures for nodes of each hyperedge via graph attention, generating the node-wise structure features. These structure features are then fused into global correlations modeling among hyperedges, with a theoretical guarantee of permutation invariance. Last, FastHeP leverages local and global high-order semantics to generate temporal hyperedge embeddings, which is efficient in a linear complexity w.r.t. model depth and average hyperedge cardinality. Extensive experiments show that FastHeP achieves up to two orders of magnitude speed-up against baselines, with an average accuracy improvement of 5.1%.
Shapley values (Shap) are a popular approach to explaining decisions of black-box models by revealing the importance of individual features. Shap explanations are easy to interpret, but as they do not incorporate feature interactions, they are also incomplete and potentially misleading. Interaction-aware methods such as nShap report the additive importance of all subsets up to n features. These explanations are complete, but in practice excessively large and difficult to interpret. In this paper, we combine the best of both worlds. We partition the features into significantly interacting groups, and use these to compose a succinct, interpretable explanation. To determine which partitioning out of super-exponentially many explains a model best, we derive a criterion that weighs the complexity of an explanation against its representativeness for the model's behavior. To be able to find the best partitioning, we show how to prune sub-optimal solutions using a statistical test. This not only improves runtime but also helps to avoid explaining spurious interactions. Experiments show that iShap represents underlying modeling more accurately than Shap and nShap, and a user study suggests that iShap is perceived as more interpretable and trustworthy.
Developing effective path representations has become increasingly essential across various fields within intelligent transportation. Although pre-trained path representation learning models have shown improved performance, they predominantly focus on the topological structures from single modality data, i.e., road networks, overlooking the geometric and contextual features associated with path-related images, e.g., remote sensing images. Similar to human understanding, integrating information from multiple modalities can provide a more comprehensive view, enhancing both representation accuracy and generalization. However, variations in information granularity impede the semantic alignment of road network-based paths (road paths) and image-based paths (image paths), while the heterogeneity of multi-modal data poses substantial challenges for effective fusion and utilization. In this paper, we propose a novel Multi-modal, Multi-granularity Path Representation Learning Framework (MM-Path), which can learn a generic path representation by integrating modalities from both road paths and image paths. To enhance the alignment of multi-modal data, we develop a multi-granularity alignment strategy that systematically associates nodes, road sub-paths, and road paths with their corresponding image patches, ensuring the synchronization of both detailed local information and broader global contexts. To address the heterogeneity of multi-modal data effectively, we introduce a graph-based cross-modal residual fusion component designed to comprehensively fuse information across different modalities and granularities. Finally, we conduct extensive experiments on two large-scale real-world datasets under two downstream tasks, validating the effectiveness of the proposed MM-Path.
Model scaling laws have driven the development of neural networks with more-and-more parameters. As a result, there is a growing need for neural network pruning to enable the efficient deployment during inference. We take a deeper look at neural network pruning from the lens of invariance preservation. We argue that successfully pruned neural networks should be invariant to transformations which do not alter the data's underlying semantics (ex. translations). To this goal, we first show existing post-pruning algorithms do not perserve desired invariances. We then propose a framework to discover novel architectures that do capture desired invariances from data via pruning. Specifically, we show contrastive learning with small initialization can effectively transfer invariance preservation encoded in the model weights to the pruning mask. Our approach consistently outperform traditional pruning algorithms on fully-connected, convolutional, and transformer networks across 3 vision, 40 tabular, and 1 natural language datasets.