Current Parameter-Efficient Fine-Tuning (PEFT) methods typically operate under an implicit assumption: once a target module is selected, every token passing through it contributes equally to the downstream task and requires a parameter update. In this paper, we challenge this convention by revealing a pervasive token-level redundancy in the fine-tuning of large models (LMs). We propose TS-PEFT, a theoretical framework utilizing proximal optimization that acts as a dynamic probe to identify token-level redundancy during the fine-tuning process. Extensive experiments demonstrate that indiscriminately updating all tokens is not only computationally superfluous but often introduces optimization noise. Surprisingly, by discarding 30%-70% of token updates, TS-PEFT consistently matches or exceeds the performance of dense baselines such as LoRA, DoRA. Our in-depth analysis shows that the learned token-level sparsity is a superior indicator of module importance compared to traditional weight criteria, providing a novel data-driven perspective on the intrinsic adaptation mechanism of LMs.
论文检索
输入标题、作者或关键词，从 990 篇学术成果中精准定位
Multi-label data often contain high-dimensional features, outlier instances, and noisy labels, all of which can lead to the curse of dimensionality and decreased performance in downstream tasks. Although numerous data reduction methods have been developed, existing approaches face two major limitations: 1) existing methods typically select features, instances, or labels independently, without considering how noise or redundancy in one dimension may negatively influence the selection of others; 2) there are very few feature and instance co-selection methods that commonly assume label annotations are free of noise, which is seldom true in practice. To address these issues, we propose Evidential Multi-Label Multi-Dimensional Selection (EMMS), which jointly performs feature, instance, and label selection on multi-label data. EMMS introduces a dual projection mechanism with sparsity constraints that transforms high-dimensional data first into a latent space and then into the label space. Simultaneously, projection residuals are explicitly modeled to facilitate the identification of representative instances, enabling unified selection across features, instances, and labels. Moreover, EMMS employs evidence theory to fuse instance-level and label-level evidence, thereby enhancing the reliability of the learned labels and reducing the influence of noisy labels, which in turn promotes multi-dimensional selection. Extensive experiments demonstrate that EMMS consistently outperforms state-of-the-art methods.
Online goal recognition in continuous domains poses two central challenges: efficiently encoding large trajectories and effectively comparing them. Recent work addresses these challenges by using custom state-space representations and metrics to compare observations against hypotheses. However, these approaches often overlook well-established encoding techniques used in other domains that offer substantial advantages. This paper introduces a novel method for online goal recognition that leverages path signatures, a compact, expressive representation of rough path theory that captures key semantic features of trajectories in an efficient way, enabling more meaningful comparisons between them. Experiments show that our method consistently outperforms the state of the art in predictive accuracy and online planning efficiency, while remaining competitive offline.
Continual test-time adaptation (CTTA) adapts a pre-trained medical segmentation model online to an unlabeled target stream whose distribution changes over time. However, most existing CTTA methods rely on pseudo-labeling and self-supervised objectives, which inevitably yield noisy supervision under domain shifts. To mitigate this limitation, we introduce off-the-shelf Vision Foundation Models (VFMs) as external knowledge sources. Zero-shot VFMs are insufficient for medical segmentation because they lack medical semantics, yet they contain rich and heterogeneous generic knowledge. To exploit such external knowledge, we propose Reciprocal Distillation with a Frozen Foundation Model (ReDi-FM), a novel framework with two core components: using structure-aware prompts from the source model to guide the frozen VFM to generate target-adapted supervision, and distilling the resulting knowledge back into the adapting model. For robust distillation, we introduce two complementary objectives: uncertainty-driven hard distillation for precise guidance in ambiguous regions, and hard class-balanced soft distillation for richer supervision of under-represented and challenging structures. A consensus-aware gate further stabilizes adaptation when the two teachers disagree. Extensive experiments on multi-domain medical segmentation benchmarks demonstrate that ReDi-FM outperforms state-of-the-art CTTA methods. Code is available at https://github.com/M4cheal/ReDi-FM.
Training critique models to provide useful feedback for actor models is an effective approach in scalable oversight, especially for complex tasks like math reasoning. However, current research lacks suitable datasets for effectively training critique models and integrating them in a principled way at both test time and training time. To bridge this gap, we first propose AutoMathCritique, an automated and scalable framework for collecting critique data. Using it, we create MathCritique-76k, a dataset of $76,321$ instances paired with step-level feedback, which is then used to train critique models for generating natural-language feedback. We show that critique models consistently improve the actor’s performance—particularly on difficult queries at test time—with larger gains as inference compute scales. Building on the findings, we propose a critique-in-the-loop self-improvement method that incorporates critique-based supervision into the actor’s self-training process. Extensive experiments demonstrate that this method improves the actor’s exploration efficiency and solution diversity, especially on challenging queries, leading to a stronger actor model. Our code and datasets are at https://mathcritique.github.io/.
In this work, we investigate the effect of lookahead branching strategies in neural network verification. We present a general recipe to integrate lookahead into any branch-and-bound verifier and demonstrate how one of the current state-of-the-art branching heuristics, FSB, can be viewed as a special instantiation of the lookahead branching strategy. We also describe how, in addition to improving the quality of branching decisions, lookahead can generate additional lemmas that accelerate verification. We instantiate the method in two representative branch-and-bound-based verifiers (Marabou and α-β-CROWN), and demonstrate that lookahead leads to consistent speedups in verification time and up to 57% more solved instances. Code is available at https://github.com/ai-ar-research/lookahead-branching.
Hyper-relational Knowledge Graphs (HKGs) extend traditional knowledge graphs by introducing high-order dependencies, where auxiliary qualifiers provide detailed information. However, modeling such intricate topologies (e.g., mixtures of hierarchies, cycles, and chains) is challenging. Existing methods suffer from structural distortion due to reliance on a single geometric space and overlook dynamic semantics across entity roles. To address these issues, we propose AdaGeM (Adaptive Geometric Learning on Manifolds), an innovative space-adaptive framework that integrates geometric learning with a role-aware Transformer. First, we propose an adaptive geometric hypergraph encoder that projects entities into a product manifold and design a topology-aware gating mechanism to dynamically select optimal geometric spaces for each entity. Second, we develop a role-aware Transformer equipped with role-specific projections and micro-structural bias injection to refine embeddings by distinguishing entity semantics across different roles and focusing on valid n-ary interactions, respectively. Extensive experiments on benchmark datasets demonstrate that AdaGeM outperforms state-of-the-art baselines in entity/relation prediction tasks, with ablation studies validating the necessity of multi-manifold modeling for heterogeneous HKG structures.
We present an extension of the Angluin-style learning algorithm for tree automata that incorporates deductive inference. The learning algorithm is provided with a term rewriting system that specifies properties of the target tree language (e.g., the order of subtrees under a symbol f is irrelevant). This term rewriting system is used to infer answers to some queries, which reduces the query complexity of the learning algorithm. We present examples of rewrite systems that express natural properties of tree-structured data, which yield a significant reduction in the number of queries.
Most of the existing decentralized multi-agent systems facilitate coordination by achieving consensus. However, reaching global consensus among a large number of agents in complex settings is challenging because resolving a local disagreement often triggers cascading adjustments across the system. This problem is further exacerbated under intermittent communication. In the field of transportation, human drivers usually rely on intentions (e.g., turn signals) in conjunction with traffic rules (such as "yielding to through traffic") to achieve efficient cooperation, instead of establishing consensus with all surrounding vehicles. Inspired by this observation, we propose the Intention \otimes Latent Rule paradigm to facilitate coordination beyond consensus, which is instantiated by DIRECT (Decentralized Intention and Latent Rule Emergence Cooperation). Within this framework, agents forecast their intentions based on current actions, and the latent rule inference module derives latent-rule representations from available information to regulate competing intentions, enabling cooperative action selection even when communication is disrupted. Extensive experiments on challenging StarCraft II micromanagement and Multi-Agent Particle Environments demonstrate that DIRECT consistently outperforms state-of-the-art methods, achieving more robust coordination across a range of communication-degraded scenarios. Our code is available at https://github.com/AAI-ZZU/DIRECT-master.
Network Traffic Anomaly Detection (NTAD), particularly under zero-positive settings, is a critical task in cybersecurity. Existing zero-positive NTAD approaches primarily rely on reconstruction-based pipelines. Nevertheless, these methods are susceptible to an identical shortcut issue, where models indiscriminately reconstruct both normal and anomalous inputs, leading to anomaly overgeneralization. To address this limitation, we propose BiPred, the first prediction-based detection paradigm for NTAD. BiPred symmetrically divides each network traffic sample into two segments and reformulates the NTAD task as a bidirectional prediction across these two segments. Unlike reconstruction-based methods, our BiPred not only eliminates shortcut learning by design but also establishes explicit bidirectional contextual dependencies, making it more sensitive to anomalous traffic. Moreover, we develop a novel Residual Scanning Mamba block that encodes multi-view contextual information to support bidirectional prediction. A residual fusion mechanism is proposed for the multi-view scanning Mamba to suppress the accumulation of inter-view redundancy. This design prevents representation degradation and provides multi-view contextual details for prediction. Extensive experiments demonstrate the superiority of our BiPred paradigm, highlighting a new research direction for NTAD. Code is available at https://github.com/ikun0124/BiPred.
Conventional threat detection methodologies encounter substantial limitations in real-world complex network settings, owing to their dependence on single-dimensional analysis, vulnerability to adversarial perturbations, and propensity for overfitting. This study introduces an adversarial training optimization framework that incorporates hierarchical label encoding and prompt learning, designed to enhance model robustness and generalization in threat detection. The framework first establishes a hierarchical structure of attack scenarios and types, leveraging a graph attention network to encode semantic and structural dependencies among labels. Subsequently, the classification task is reformulated as a masked language modeling problem through prompt learning, enabling effective semantic alignment. Furthermore, a novel adversarial training mechanism is proposed, which utilizes local hierarchical information as a potent regularization signal. Within a game-theoretic architecture comprising a generator, an encoder, and a discriminator, the encoder is steered to integrate authentic hierarchical priors and produce high-fidelity oracle representations. This process encourages the generator to implicitly assimilate sample-specific hierarchical knowledge during adversarial learning, thereby improving the model's resilience to noisy inputs and its capacity to detect infrequent attacks. Evaluation results show that this method achieves an accuracy of 0.9972–1.0000 in complex mixed scenarios for scenario detection and 0.9987–0.9999 for attack category detection, while exhibiting strong robustness against interference.
Generative models have emerged as a powerful paradigm in sequential recommendation due to their superior distribution modeling. However, long-tail data distributions inevitably induce popularity bias, as iterative generation trajectories gravitate toward dense clusters of popular items. Current debiasing methods struggle to enforce consistent step-wise constraints, rendering them ineffective for the multi-step process of generative recommendation. To address this challenge, we propose a method called Popularity-Debiased Flow Matching for sequential recommendation (PDFlow). Specifically, grounded in the flow matching framework, PDFlow integrates a residual popularity vector field beyond the main flow to model the popularity debiasing flow at each step. This enables real-time trajectory intervention to adjust the generative path, preventing the model from overemphasizing popular items. Subsequently, we incorporate cross-popularity alignment loss, using co-occurrence patterns within the same sequence to align features of both popular and tail items, thereby mitigating popularity-driven distribution separation. Jointly, trajectory correction prevents popularity overfitting while cross-popularity alignment ensures latent fairness, effectively mitigating popularity bias. Extensive experiments on four real-world datasets demonstrate the effectiveness of PDFlow in improving long-tail coverage and overall recommendation performance. All codes and datasets are available at https://github.com/eqmll/PDFlow.
Graph self-supervised learning (SSL) typically relies on large-scale unlabeled datasets, heavily inflating computational costs. However, empirical evidence suggests that these datasets contain substantial redundancy—our analysis reveals that uniformly subsampling 50% of graphs retains over 96% of downstream performance. To exploit this redundancy, we introduce GraphSculptor for pre-training coreset construction. Unlike methods dependent on additional training-time signals or limited solely to topological statistics, GraphSculptor provides a label-free solution that constructs coresets via two complementary perspectives: intrinsic structure and contextual semantics. Concretely, structural diversity is quantified using intrinsic graph statistics, yielding a structural feature vector for each graph, while semantic diversity is captured by utilizing a pre-trained language model to encode descriptions generated via graph-to-text. GraphSculptor integrates these signals into a unified metric space and performs cluster-aware selection to preserve joint structural--semantic diversity. We further derive a theoretical bound on the loss gap between coreset and full-data pre-training, offering theoretical motivation for our selection formulation. Extensive experiments demonstrate that GraphSculptor effectively "sculpts" the dataset: a 10% coreset achieves 99.6% of full-data performance while reducing pre-training time by nearly 90%, offering a scalable solution for data-efficient graph pre-training.
Designing effective reward functions remains a major challenge in reinforcement learning (RL), particularly in open-ended environments where task goals are abstract and difficult to quantify. In this work, we present VLM-AR3L, a framework that leverages Vision-Language Models (VLMs) to provide both absolute and relative rewards for RL. VLM-AR3L interprets an agent’s visual observations in the context of a natural language task goal, and learns both absolute and relative rewards from VLM-generated preference labels. The absolute reward model predicts scalar evaluations for individual states, while the relative reward model compares consecutive observations to infer progress or regression toward the task goal. Their integration combines the stability of state-based evaluation with the robustness of comparative supervision. We evaluate VLM-AR3L across benchmarks spanning classic control, manipulation, and open-world embodied tasks, with a particular focus on Minecraft given its visual complexity and long-horizon decision-making requirements. Experimental results show that VLM-AR3L consistently outperforms prior VLM-based reward learning methods. Videos and code are available on the project website: https://vlm-ar3l.github.io/.
The Spherical Sliced Wasserstein (SSW) distance has emerged as a fundamental tool for comparing probability measures on the hypersphere, with broad applications spanning geology, medical imaging, computer vision, and representation learning. However, its reliance on uniformly sampling a large number of projection directions from the unit hypersphere can lead to suboptimal performance, as many directions are weakly informative and fail to capture salient differences between distributions. We address this limitation through Fusion Stereographic Spherical Sliced Wasserstein (FS3W), a data-adaptive divergence that incorporates distribution-dependent information into projection selection. FS3W uses a neural slicing fusion mechanism to refine prior directions, steering them toward projections aligned with discriminative geometric structures in the data. This adaptive approach produces more expressive and accurate characterizations of distributional discrepancies on the hypersphere. We evaluate FS3W across five diverse tasks, including gradient flows, Earth density estimation, and self-supervised representation learning. Empirical results consistently demonstrate that FS3W outperforms existing spherical optimal transport–based methods.
Due to proliferation of vehicle trajectory data from advanced sensing technologies, path representation learning has become a pivotal task in intelligent transportation systems. Although existing self-supervised approaches work to some extent, their dependence on deterministic contrastive learning paradigms and handcrafted view augmentation strategies inherently restricts cross-scenario generalization capabilities. To address these limitations, we present DGCPath – an innovative Distribution-aware Generative Contrastive learning framework for Path representation. This architecture establishes a synergistic connection between generative modeling and distributional contrastive learning, enabling the acquisition of robust and transferable feature embeddings. Specifically, our framework incorporates: (1) a diffusion-based view generator that autonomously produces semantically coherent yet diversified trajectory views from Gaussian noise distributions; (2) a variational contrastive mechanism enforcing latent feature alignment at the distribution level, transcending conventional instance-wise consistency; and (3) a novel generative cross-supervision module that reinforces view-level consistency through cross-view reconstruction learning. Comprehensive evaluations on three real-world trajectory datasets demonstrate DGCPath's superior performance over state-of-the-art baselines in two distinct downstream tasks, validating its enhanced generalization capacity and representation effectiveness.
Accurate prediction of prerequisite relations among concepts is important for course planning and intelligent tutoring systems. Existing text-based methods are frequently contaminated by noise such as redundant phrasing, ambiguous sentences, and domain-specific colloquialisms. Moreover, previous methods based on graph structures may have failed to capture the complex interactions between concepts and documents, an information gap exists between the two approaches. Therefore, we propose an LLM Enhancement and Concept-Document Interactive Modeling for Prerequisite Relation Prediction (LECDPR) model. Concretely, LECDPR elicits an LLM via two-stage prompting to distill concept descriptions and contextual evidence from documents, resulting in semantic space embeddings of concepts. Then, the complex interactive relations between concepts and documents are modeled through three propagation mechanisms to produce relational space embeddings, aligns the two embeddings through random masking and contrastive learning, and adaptively fuses the resulting embeddings. Finally, the fused concept representations are used for more accurate prerequisite relation prediction. Extensive experiments on the public UCD, Lecture Bank, and MOOC datasets show that LECDPR outperforms ten baselines on ACC, AUC, and F1. Our code is available at https://github.com/lecdpr/LECDPR.
Clustering is fundamental to scRNA-seq analysis, serving as a cornerstone for identifying cell populations and resolving tissue heterogeneity. However, existing methods focus on mining numerical statistical patterns, suffering from semantic agnosticism by neglecting the intrinsic biological functions encoded by genes. While Large Language Models (LLMs) offer promising semantic capabilities, their direct adaptation to cell clustering is hindered by the structural mismatch between generative pre-training objectives and discriminative downstream tasks. To bridge this gap, we propose scLLM-DSC, a novel LLM-Knowledge Enhanced Cross-Modal Deep Structural Clustering framework. Diverging from data-driven paradigms, scLLM-DSC establishes a semantically-grounded representation by synergizing two views: a Knowledge-Driven Semantic View derived from NCBI gene priors and contextualized Cell2Sentence embeddings, and a Structure-Aware Topological View extracted via a graph-guided encoder. Crucially, we introduce a cross-modal contrastive alignment mechanism to enforce consistency between biological semantics and transcriptomic features within a unified latent space. Extensive benchmarks demonstrate that scLLM-DSC significantly outperforms eleven state-of-the-art baselines in clustering accuracy.
Virtual Knowledge Graphs (VKGs) provide an effective solution for data integration by mapping heterogeneous data sources to a unified ontology. However, existing VKG construction frameworks primarily focus on one-shot construction, which often results in partial data coverage and support for only initial information needs. After the VKG has been deployed, new information needs will inevitably arise over time. Therefore, enriching VKGs to support evolving information needs remains an expert-intensive iterative task. In this work, we formulate the task of Information-Needs-Guided VKG Enrichment (IN-VKGE), and propose an iterative framework that leverages large language models to assess whether information needs can be supported using SPARQL execution feedback and generate ontology and mapping enrichment proposals. Experiments on two real-world VKGs show that our approach outperforms existing paradigms, and produces enrichment proposals that receive high expert ratings for effectively resolving the identified information needs.
Attribute-aware sequential recommendation entails predicting the next item a user will interact with based on a chronologically ordered history of past interactions, enriched with item attributes. Existing methods typically leverage self-attention mechanisms to aggregate the entire sequence into a unified representation used for next-item prediction. While effective, these models often suffer from high computational complexity and memory consumption, limiting their ability to process long user histories. This constraint restricts the model's capacity to fully capture long-term user preferences. In some scenarios, modeling item interactions purely through attention may also not be the most effective approach to extract sequential patterns. In this work, we propose ConvRec, an alternative method with linear computational and memory complexity that employs convolutional layers in a hierarchical, down-scaled fashion to generate compact, yet expressive sequence representations. To further enhance the model's ability to capture diverse sequential patterns, each layer aggregates the neighboring items gradually to reach a comprehensive sequence representation. Extensive experiments on four real-world datasets demonstrate that our approach outperforms state-of-the-art sequential recommendation models, highlighting the potential of convolution-based architectures for efficient and effective sequence modeling in recommendation systems. Our implementation code and datasets are available at https://github.com/ismll-research/ConvRec.