Deep neural networks often learn and rely on spurious correlations, i.e., superficial associations between non-causal features and the targets. For instance, an image classifier may identify camels based on the desert backgrounds. While it can yield high overall accuracy during training, it degrades generalization on more diverse scenarios where such correlations do not hold. This problem poses significant challenges for out-of-distribution robustness and trustworthiness. Existing methods typically mitigate this issue by using external group annotations or auxiliary deterministic models to learn unbiased representations. However, such information is costly to obtain, and deterministic models may fail to capture the full spectrum of biases learned by the models. To address these limitations, we propose Evidential Alignment, a novel framework that leverages uncertainty quantification to understand the behavior of the biased models without requiring group annotations. By quantifying the evidence of model prediction with second-order risk minimization and calibrating the biased models with the proposed evidential calibration technique, Evidential Alignment identifies and suppresses spurious correlations while preserving core features. We theoretically justify the effectiveness of our method as capable of learning the patterns of biased models and debiasing the model without requiring any spurious correlation annotations. Empirical results demonstrate that our method significantly improves group robustness across diverse architectures and data modalities, providing a scalable and principled solution to spurious correlations.
论文检索
输入标题、作者或关键词，从 844 篇学术成果中精准定位
Calibrating Pre-trained Language Classifiers on LLM-generated Noisy Labels via Iterative Refinement
PDF ↗The traditional process of creating labeled datasets is labor-intensive and expensive. Recent breakthroughs in open-source large language models (LLMs) have opened up a new avenue in generating labeled datasets automatically for various natural language processing (NLP) tasks, providing an alternative to such an expensive annotation process. However, the reliability of such auto-generated labels remains a significant concern due to inherent inaccuracies. When learning from noisy labels, the model's generalization is likely to be harmed as it is prone to overfit to those label noises. While previous studies in learning from noisy labels mainly focus on synthetic noise and real-world noise, LLM-generated label noise receives less attention. In this paper, we propose SiDyP: Simplex Label Diffusion with Dynamic Prior to calibrate the classifier's prediction, thus enhancing its robustness towards LLM-generated noisy labels. SiDyP retrieves potential true label candidates by neighborhood label distribution in text embedding space and iteratively refines noisy candidates using a simplex diffusion model. Our framework can increase the performance of the BERT classifier fine-tuned on both zero-shot and few-shot LLM-generated noisy label datasets by an average of 7.21% and 7.30% respectively. We demonstrate the effectiveness of SiDyP by conducting extensive benchmarking for different LLMs over a variety of NLP tasks. Our code is available on GitHub: https://github.com/gtfintechlab/SiDyP
Graph-level anomaly detection (GLAD) identifies graphs exhibiting abnormal properties within a graph dataset. Despite promising results in this task, the state-of-the-art methods cannot be fully trusted and deployed in realistic scenarios due to their black-box nature. To alleviate this, existing methods try to explain predictions by extracting important subgraphs from each graph, as instance-level explanations. However, instance-level explanations across all samples are costly to verify and insufficient to capture the model's general behaviors. Thus, we propose a global interpretable Graph-Level Anomaly Detection model via Prototype (GLADPro), which provides global-level explanations throughout the entire dataset, that is, the significant subgraph patterns that consistently influence the model's decisions. Specifically, GLADPro incorporates prototype learning with the information bottleneck principle, enabling prototypes to capture the most significant subgraph patterns as global-level explanations through persistent interactions with key subgraphs from input graphs. In addition, a regularization term is proposed to prevent the collapse traps with theoretical proof. Finally, we filter redundant prototypes using the maximum mean discrepancy metric. Extensive experiments demonstrate the superiority of GLADPro in anomaly detection and explainability; for instance, on the mutagen dataset, it reduces the number of explanations to verify from 1403 to only 6.
Searching for images using text has limitations because language has difficulties in expressing certain abstract intentions, e.g. artistic styles are difficult to describe for non-experts. As for the image search image model, images can convey abstract intentions, but cannot express the specific purpose, so many of the current graph search works only have a single function, such as content search and style search. Our work aims to combine the strengths of both, merging the ability of text to express specific ideas with the ability of images to convey abstract concepts, thus achieving a better capture of the user's intentions. To this end, we propose CCSR, a multimodal conditional content-style joint retrieval model. Our model is the first to apply contrastive learning to conditional retrieval and introduces a novel Mixture-of-Expert models (MOE) system to enable collaboration between multiple expert systems. We adopt a novel prompt learning strategy that allows the model to adaptively select specific prompts, thereby enhancing its focus on the current task. In addition, to evaluate the joint content-style retrieval capability of our model, we present a new dataset, StyleCoco, containing rich content categories and style categories. The experimental results indicate that CCSR has achieved state-of-the-art performance in conditional style retrieval, content retrieval, and style-content retrieval. The dataset and code will be publicly available on https://mccsr.github.io/.
Apprenticeship learning (AL) aims to induce decision-making policies by observing and imitating expert demonstrations. Existing AL approaches typically rely on online interactions and assume that the demonstrations follow a single reward function. Nevertheless, in real-world human-centric applications, policies are usually learned in an offline setting, with the demonstrations driven by multiple reward functions that evolve over time. To address these challenges, we introduce a novel AL framework: Time-aware Hierarchical EM Energy-based Sub-trajectory THEMES clustering. We evaluate the effectiveness of THEMES in two challenging human-centric domains -- healthcare and education. Our experimental results across multiple datasets demonstrate that THEMES can accurately induce policies, outperforming competitive baselines and ablations, demonstrating its potential for tackling a broad range of complex, real-world human-centric tasks.
Graph recommender (GR) is a type of graph neural network (GNN) encoder that is customized for extracting information from the user-item interaction graph. Due to its strong performance on the recommendation task, GR has gained significant attention recently. Graph contrastive learning (GCL) is also a popular research direction that aims to learn, often unsupervised, GNNs with certain contrastive objectives. As general graph representation learning methods, GCLs have been widely adopted with supervised recommendation loss for joint training of GRs. Despite the intersection of GR and GCL research, theoretical understanding of the relationship between the two fields is surprisingly sparse. This vacancy inevitably leads to inefficient scientific research. In this paper, we aim to bridge the gap between the fields of GR and GCL from the perspective of encoders and loss functions. We theoretically show an interesting fact that graph recommenders are equivalent to a commonly used single-view graph contrastive model. Specifically, we find that (1) the classic encoder in GR is essentially a linear graph convolutional network with one-hot inputs, and (2) the loss function in GR is well bounded by a single-view GCL loss with certain hyperparameters. The first observation enables us to explain crucial designs of GR models, e.g., the removal of self-loop and nonlinearity. The second finding leads to a remarkable result that the recommendation loss and the GCL loss can be used interchangeably. The fact that we can train GR models solely with the GCL loss is particularly insightful, since before this work, GCLs are typically viewed as unsupervised methods that need fine-tuning. For the completeness of our work, we further generalize our theory to accommodate other GCL losses, particularly a multi-view GCL method that explicitly utilizes structural information. We believe our theory holds significant potential to inspire novel cross-field research directions in the future.
Breaking the Top-K Barrier: Advancing Top-K Ranking Metrics Optimization in Recommender Systems
PDF ↗In the realm of recommender systems (RS), Top-K ranking metrics such as NDCG@K are the gold standard for evaluating recommendation performance. However, during the training of recommendation models, optimizing NDCG@K poses significant challenges due to its inherent discontinuous nature and the intricate Top-K truncation. Recent efforts to optimize NDCG@K have either overlooked the Top-K truncation or suffered from high computational costs and training instability. To overcome these limitations, we propose SoftmaxLoss@K (SL@K), a novel recommendation loss tailored for NDCG@K optimization. Specifically, we integrate the quantile technique to handle Top-K truncation and derive a smooth upper bound for optimizing NDCG@K to address discontinuity. The resulting SL@K loss has several desirable properties, including theoretical guarantees, ease of implementation, computational efficiency, gradient stability, and noise robustness. Extensive experiments on four real-world datasets and three recommendation backbones demonstrate that SL@K outperforms existing losses with a notable average improvement of 6.03%. The code is available at https://github.com/Tiny-Snow/IR-Benchmark.
Traditional knowledge distillation focuses on aligning the student's predicted probabilities with both ground-truth labels and the teacher's predicted probabilities. However, the transition to predicted probabilities from logits would obscure certain indispensable information. To address this issue, it is intuitive to additionally introduce a logit-level loss function as a supplement to the widely used probability-level loss function, for exploiting the latent information of logits. Unfortunately, we empirically find that the amalgamation of the newly introduced logit-level loss and the previous probability-level loss will lead to performance degeneration, even trailing behind the performance of employing either loss in isolation. We attribute this phenomenon to the collapse of the classification head, which is verified by our theoretical analysis based on the neural collapse theory. Specifically, the gradients of the two loss functions exhibit contradictions in the linear classifier yet display no such conflict within the backbone. Drawing from the theoretical analysis, we propose a novel method called dual-head knowledge distillation, which partitions the linear classifier into two classification heads responsible for different losses, thereby preserving the beneficial effects of both losses on the backbone while eliminating adverse influences on the classification head. Extensive experiments validate that our method can effectively exploit the information inside the logits and achieve superior performance against state-of-the-art counterparts
Hi-PNG: Efficient Interval-Filtering ANNS via Hierarchical Interval Partition Navigating Graph
PDF ↗Approximate nearest neighbor search (ANNS) is widely used to retrieve similar vectors from high-dimensional data. However, many real-world applications require additional interval filtering based on numerical constraints, such as stock price ranges. In this paper, we introduce interval-filtering ANNS (IF-ANNS), a novel yet general retrieval task where both base and query vectors are associated with numerical intervals. The goal is to retrieve nearest neighbors whose intervals are fully contained within the query interval. To efficiently address this problem, we propose Hierarchical Interval Partitioning Navigating Graph (Hi-PNG), a framework that hierarchically partitions the interval space to efficiently locate the search space, eliminating unnecessary computations and achieving significant speedups. We provide a theoretical analysis of the search space, demonstrating the superiority of our approach. Extensive experiments on eight datasets, including commonly used benchmarks and real-world stock price data, show that Hi-PNG outperforms four state-of-the-art graph-based ANNS baselines, achieving up to 15× acceleration while maintaining high precision. These results highlight Hi-PNG's effectiveness in solving the IF-ANNS problem. All codes and data generation are available at https://github.com/PUITAR/Hi-PNG.git.
When Interpretability Meets Generalization: Delta-GAM for Robust Extrapolation in Out-of-Distribution Settings
PDF ↗Out-of-Distribution (OOD) extrapolation, where test data feature values extend beyond the training range, poses significant challenges in machine learning. While existing solutions often sacrifice interpretability, resulting in limited applicability in high-stakes applications where interpretability is a a critical requirement. In this paper, we propose Delta-GAM, an interpretable Generalized Additive Model (GAM) that achieves robust extrapolation in OOD scenarios. Our method jointly learns (1) feature-target relationships and (2) functional adaptations for extrapolating beyond the training distribution by reformulating GAM fitting as a second-order interaction problem between features and their distributional offsets. We theoretically show that smooth GAM shape functions induce an approximately low-rank structure in these interactions, enabling efficient decomposition via a specialized neural network. Experiments on synthetic and real-world data demonstrate Delta-GAM's superior performance in OOD extrapolation tasks while preserving model interpretability, bridging a key gap in trustworthy machine learning.
In e-commerce, product descriptions and other forms of copywriting play a critical role in shaping consumer purchasing decisions. However, manually crafting such content is both time-consuming and costly, particularly given the vast and diverse item catalogs. Recent advances in large language models (LLMs) have transformed automated text generation, offering immense potential to streamline this process. Despite their capabilities, LLMs continue to face obstacles in e-commerce applications, including a lack of diversity and an inability to fully grasp the nuanced details of specific items. To address these limitations, we propose a novel framework that integrates graph-based knowledge into Retrieval-Augmented Generation (RAG) to enhance content generation. Our approach leverages user reviews to construct an item-feature graph, capturing both explicit and implicit connections between items and features. This structured representation enables the retrieval of diverse, contextually relevant, and factually grounded information, effectively addressing key deficiencies of existing methods. With the constructed graph, we design a graph traversal mechanism that explores a broader range of item-related features, augmenting the generation process with more varied and informative inputs. Extensive experiments demonstrate that our method significantly improves diversity while preserving fidelity, marking a major advancement in automated e-commerce content generation.
EARN: Efficient Inference Acceleration for LLM-based Generative Recommendation by Register Tokens
PDF ↗Large Language Model-based generative recommendation (LLMRec) has achieved notable success, but it suffers from high inference latency due to massive computational overhead and memory pressure of KV Cache. Existing KV Cache reduction methods face critical limitations: cache compression offers marginal acceleration given recommendation tasks' short decoding steps, while prompt compression risks discarding vital interaction history. Through systematic analysis of attention patterns in LLMRec, we uncover two pivotal insights: 1) layer-wise attention sparsity inversion where early layers retain dense informative patterns while later layers exhibit high redundancy, and 2) dual attention sinks phenomenon where attention scores concentrate on both head and tail tokens of input sequences. Motivated by these insights, we propose EARN, an efficient inference framework that leverages the early layers to compress information into register tokens placed at the input sequence boundaries, then focuses solely on these tokens in the subsequent layers. Extensive experiments on three datasets, two LLMRec methods and two LLM architectures demonstrate EARN's superiority, achieving up to 3.79x speedup and 80.8% KV Cache reduction with better accuracy than the general finetuning approach. Our work bridges the efficiency-effectiveness gap in LLMRec, offering practical deployment advantages for industrial scenarios.
Simulation of fluid flows is crucial for modeling physical phenomena like meteorology, aerodynamics, and biomedicine. Classical numerical solvers often require fine spatiotemporal grids to satisfy stability, consistency, and convergence conditions, leading to substantial computational costs. Although machine learning has demonstrated better efficiency, they typically suffer from issues of interpretability, generalizability, and data dependency. Hence, we propose a learnable and differentiable finite volume solver, called LDSolver, designed for efficient and accurate simulation of fluid flows on spatiotemporal coarse grids. LDSolver comprises two key components: (1) a differentiable finite volume solver, and (2) an learnable module providing equivalent approximation for fluxes (derivatives and interpolations), and temporal error correction on coarse grids. Even with limited training data (e.g., only a few trajectories), our model could accelerate the simulation while maintaining a high accuracy with superior generalizability. Experiments on different flow systems (e.g., Burgers, decaying, forced and shear flows) show that LDSolver achieves state-of-the-art performance, surpassing baseline models with notable margins.
FedDiAL: Adaptive Federated Learning with Hierarchical Discriminative Network for Large Pre-trained Models
PDF ↗Large pre-trained models have significantly advanced computer vision (CV) and natural language processing (NLP). However, their deployment is challenging in privacy-sensitive scenarios where data must remain decentralized. Federated Learning (FL) addresses this issue by enabling local model training without sharing raw data. Despite this advantage, integrating large models into FL introduces challenges such as training inefficiencies, label scarcity, and data heterogeneity. In this paper, we propose FedDiAL, a framework designed to effectively incorporate large pre-trained models into FL while addressing these challenges. At its core, FedDiAL, features the novel HDis-Net, which enhances the efficiency of large models in resource-constrained environments. We introduce a two-phase training strategy with probabilistic feature augmentation to improve feature discrimination. Additionally, we propose an adaptive pseudo-labeling method to generate high-confidence labels, mitigating label scarcity. To handle data heterogeneity, we develop a focused fine-tuning strategy that adapts HDis-Net to diverse client data distributions. Our theoretical analysis establishes the convergence of HDis-Net. Extensive experiments on four datasets, including Tiny ImageNet and AG News, demonstrate that FedDiAL outperforms state-of-the-art methods, achieving up to a 35.66% accuracy improvement in both CV and NLP tasks.
Learning from Text-Attributed Graphs (TAGs) has attracted significant attention due to its wide range of real-world applications. The rapid evolution of language models (LMs) has revolutionized the way we process textual data, which indicates a strong potential to replace shallow text embedding generally used in Graph Neural Networks (GNNs). However, we find that existing LM approaches that exploit text information in graphs suffer from inferior computation and data efficiency. In this study, we introduce LEADING, a novel and efficient approach for end-to-end fine-tuning of language models on TAGs. To enhance data efficiency, LEADING efficiently transfers rich knowledge from LMs to downstream graph learning tasks with limited labeled data by employing end-to-end training of LMs and GNNs in a semi-supervised learning setting. To address associated computation efficiency issues, it introduces two techniques: neighbor decoupling targeting LMs and implicit graph modeling targeting GNNs, respectively. Our proposed approach demonstrates superior performance, achieving state-of-the-art (SOTA) results on the ogbn-arxiv leaderboard, while maintaining computation cost and memory overhead comparable to graph-less fine-tuning of LMs. Through comprehensive experiments, we showcase its superior computation and data efficiency, presenting a promising solution for various LMs and graph learning tasks on TAGs.
Temporal graphs, widely used in social network modeling, are valuable for research but pose challenges due to data accessibility and privacy concerns. High-quality graph generation models can produce surrogate data for sharing and training, benefiting tasks such as behavior analysis, anomaly detection, and data augmentation. However, existing deep learning and probabilistic approaches often struggle to balance global statistical properties with local structural details. To overcome this limitation, we leverage motifs, small subgraphs that serve as the building blocks of complex networks, to encode local information. Based on a spectral analysis of motifs, we propose MoDiff, a novel motif-aware diffusion model for temporal graph generation. MoDiff integrates motifs into a diffusion framework by employing motif-enhanced Hermitian matrices that capture local structures and edge orientations, while the spectral diffusion model efficiently generates graphs. Moreover, MoDiff supports controllable graph generation by adjusting density parameters to simulate the evolution of temporal graphs. Experimental results demonstrate that MoDiff outperforms existing approaches, reducing degree discrepancies by 10-50% and clustering discrepancies by 50-90%, while better preserving higher-order structural features. Our code is available at: https://github.com/Yuwe1XU/MoDiff.
In this paper, we present the Proportional Payoff Allocation Game (PPA-Game), which characterizes situations where agents compete for divisible resources. In the PPA-game, agents select from available resources, and their payoffs are proportionately determined based on heterogeneous weights attributed to them. Such dynamics simulate content creators on online recommender systems like YouTube and TikTok, who compete for finite consumer attention, with content exposure reliant on inherent and distinct quality. We first conduct a game-theoretical analysis of the PPA-Game. While the PPA-Game does not always guarantee the existence of a pure Nash equilibrium (PNE), we identify prevalent scenarios ensuring its existence. Simulated experiments further prove that the cases where PNE does not exist rarely happen. Beyond analyzing static payoffs, we further discuss the agents' online learning about resource payoffs by integrating a multi-player multi-armed bandit framework. We propose an online algorithm facilitating each agent's maximization of cumulative payoffs over T rounds. Theoretically, we establish that the regret of any agent is bounded by O(log^1 + η T) for any η > 0. Empirical results further validate the effectiveness of our online learning approach.
Dual Structure-guided Contrastive Network for Incomplete Multi-view Partial Multi-label Classification
PDF ↗Incomplete multi-view partial multi-label classification (IMvPMLC), which tackles the combined challenges of incompleteness in both multi-view and multi-label problems, has drawn considerable attention. Existing IMvPMLC methods have made progress but still face several challenges: (i) They mainly focus on the consistency of representations across multiple views but overlook the relationships among instances, leading to suboptimal representations. (ii) They primarily utilize only the available labels for supervised learning, ignoring the missing label distribution and limiting their ability to capture label correlations. In this paper, we propose a novel model named Dual Structure-guided Contrastive Network (DSCN) for IMvPMLC. Specifically, we introduce a similarity-guided instance-level contrastive learning mechanism to achieve multi-view consistent and discriminative representations across instances by leveraging instance structures, while a multi-view attention-based fusion strategy dynamically facilitates the fusion of multi-view representations to derive a robust consensus representation. Then, we design a multi-view shared classifier integrated with a correlation-guided label-level contrastive learning mechanism to enhance predictions by leveraging complementary information across multiple views and capturing label structures, effectively exploiting missing label distribution. Extensive experiments on five benchmark datasets demonstrate that, DSCN yields a more than 13% accuracy, compared with the state-of-the-art approaches. The code and datasets are available at https://anonymous.4open.science/r/DSCN-D471.
A/B Test and Online Experiment Under Diminishing Marginal Effects: Regret Minimization and Statistical Inference
PDF ↗When large online platforms test a new strategy to implement with their user traffic, the phenomenon of diminishing marginal effects may arise. For example, when a strategy is implemented on 100% of the user traffic, the expected per-user effect can be lower compared to the expected per-user effect when a strategy is implemented on 10% of the user traffic, potentially due to limits of overall budget, resource, attention or content involved with that strategy. This diminishing marginal effect phenomenon brings an additional delicacy to online sequential experiments. In particular, for the classical goal of achieving the largest expected reward, the optimal decision may no longer be assigning 100% traffic to one strategy, but instead a mixture of strategies. We deliver two tasks for online sequential experiments in presence of diminishing marginal effect: (1) Adaptively identify the optimal traffic allocation to maximize the expected cumulative reward and (2) Construct valid central limit theorem (which is critically needed for A/B tests in online platforms) to perform reliable statistical inference for the expected reward under the optimal traffic allocation that is a priori unknown. We show that classical algorithms can fail to deliver the second task, especially because the statistical inference task presents its own difficulty. We develop a new online algorithm that leverages an additional smoothness condition on how the marginal effects change to achieve both tasks. We prove that this algorithm obtains the best achievable expected cumulative reward. Further, crucially for online platforms' need to do trustworthy statistical inference, the algorithm is proved to enjoy a valid central limit theorem. The theoretical findings are illustrated through numerical experiments.
Test-Time Training with Diversified Local Aggregation Consistency for Mortality Prediction using Clinical Time Series
PDF ↗Mortality prediction is necessary for patients in the Intensive Care Unit (ICU). Clinical time series provide essential insights for making accurate predictions. However, existing prediction models often struggle with domain shifts when applied across different domains. Privacy concern hinders model calibration due to the forbidden data sharing across domains. Test-Time Training (TTT) has been increasingly researched to tackle the above issues by updating a source model to each single target sample before inference. While massive vision-based TTT methods are proposed, deploying TTT in clinical time series still faces the unique challenge of temporal imbalance: Time points tend to cluster around specific periods in some patients. Neglecting the temporal imbalance in TTT can make the model biased toward dense local pattern, resulting in unsatisfactory prediction. To overcome this challenge, we propose a novel Test-Time Training method with Diversified Local Aggregation Consistency (DLAC-TTT). During the test-time update, DLAC-TTT focuses on the distinct temporal distributions within each patient, enforcing their local diversity and global consistency through aggregation. In this way, it can mitigate the over-reliance on specific local patterns and well integrate diverse local patterns for global learning. Extensive experiments show that DLAC-TTT can boost the generalization performance across real-world clinical datasets from different medical institutes.