论文检索

Applications · Everything Else

Weiying Xie, Chenhe Hao, Haozhi Shi, Jitao Ma, Daixun Li, Jiazhe Li, Hengyi Wang, Leyuan Fang, Yunsong Li

Aggregation-Free Federated Learning enables joint training by sharing synthetic data, aiming to eliminate data heterogeneity across clients. However, existing methods fail to explicitly separate the principal and residual components of dataset, leading to biased synthetic data. In this paper, we propose a novel Unbiased Synthetic Data optimization method FedUSD for Aggregation-Free Federated Learning, which is achieved by exploring the High-energy Orthogonal Base (HOB) and variance of dataset in feature space. Our FedUSD is inspired by the discovery that principal component concentrates in HOB while residual component independently reflects in variance, regardless of networks. Based on the observation, we develop a method that mathematically optimizes synthetic data by matching both HOB and variance with those of real data. Besides, we experimentally show the superior effectiveness of leveraging HOB and variance to separately extract the principal and residual components over existing methods. We also theoretically prove that FedUSD achieves unbiased synthetic data and thus convergence. Without introducing any constraints, FedUSD thereby yields significant improvements over the state-of-the-arts in terms of global model performance, under equivalent communicational costs. For example, on the SVHN dataset, FedUSD improves 6.74\% to 30.82\% which is higher than others with Dirichlet coefficient $\alpha=0.01$.

Deep Learning · Foundation Models

Wenhang Shi, Yiren Chen, Shuqing Bian, Zhe Zhao, Pengfei Hu, Jinhao Dong, WEI LU, Xiaoyong Du

While prompt engineering is instrumental in maximizing the capabilities of Large Language Models (LLMs) during inference, the role of prompts during training remains critically underexplored. Prevailing fine-tuning paradigms typically treat training prompts as mere surface forms, assuming that semantically equivalent instructions yield identical learning outcomes. However, we reveal that this equivalence is deceptive: while paraphrased prompts often lead to comparable in-task performance, they induce drastically different cross-task impacts regarding catastrophic forgetting and generalization. Crucially, these impacts are positively correlated across tasks, indicating the existence of superior prompts that consistently yield better performance. Furthermore, we discover that these superior prompts can be robustly identified by task loss prior to learning. Leveraging these insights, we introduce State-Adaptive Prompt Optimization (SAPO), a lightweight yet effective training strategy that shifts task formulation from a static input to a dynamic, state-adaptive variable. Comprehensive experiments on diverse benchmarks confirm its effectiveness, which significantly mitigates forgetting while improving generalization, achieving substantial performance gains over state-of-the-art methods. These results provide insights into how training prompts shape learning dynamics and offer a practical recipe for robust fine-tuning.

Applications · Health / Medicine

Zixun Zhang, Yuncheng Jiang, Yuzhe Zhou, Jiayou Zheng, Shuguang Cui, Zhen Li

Tertiary structure-based RNA design aims to generate RNA sequences that can fold into desired 3D structures, but remains a challenging problem due to the scarcity of annotated data, structural noise, and the intrinsic complexity of RNA topology. Existing structure-to-sequence frameworks largely rely on static k-nearest neighbor graphs and rigid message passing schemes, which fail to capture the flexible and heterogeneous nature of RNA geometry. To address these issues, we propose a unified framework, ATL-FGD, that integrates Adaptive Topology Learning (ATL) and FiLM-Guided Distillation (FGD) for robust RNA design. ATL introduces a differentiable edge gating mechanism to jointly learn topology and representation, enabling the model to construct data-driven, layer-adaptive graphs that better reflect structural dynamics and biochemical consistency. On top of this, FGD bridges structural and sequence representations via feature-wise linear modulation, softly transferring the semantic knowledge from RNA foundation models without relying on them during inference. Extensive experiments on tertiary structure-based RNA design benchmarks demonstrate that our approach achieves significant improvements in both sequence recovery and structural fidelity.

Deep Learning · Large Language Models

He Li, Haoang Chi, Qizhou Wang, Yunxin Mao, Zhiheng Zhang, Jie Tan, Tongliang Liu, Wenjing Yang, Bo Han

Multimodal large language models (MLLMs) are trained on massive multimodal data, making data unlearning increasingly important as data owners may request the removal of specific content. In practice, these requests often arrive sequentially over time, giving rise to the challenging problem of *MLLM Lifelong Unlearning*. However, most existing benchmarks are limited in scale and scope, failing to capture the complexities of MLLM lifelong unlearning. To fill this gap, we introduce the MLUBench, a large-scale and comprehensive benchmark featuring 127 entities across 9 classes under lifelong unlearning requests. We perform extensive experiments using MLUBench and reveal that existing unlearning methods suffer from severe, cumulative degradation. More critically, we further identify the unique challenge of this problem: unlike in unimodal models, MLLM lifelong unlearning is constrained by the need to preserve multimodal alignment. Continually unlearning from one modality could degrade the entire model. To alleviate this challenge, we propose LUMoE, an effective and efficient method. Experiments demonstrate that LUMoE significantly mitigates the degradation problem faced by baselines. We present source code and the MLUBench in this anonymous [URL](https://anonymous.4open.science/r/Lifelong_Unlearning_main-72EC/).

Theory · Deep Learning

Sejun Park, Yeachan Park, Geonho Hwang

Research on the expressive power of transformers shows that transformers are equivariant to permutations and can approximate all permutation-equivariant continuous functions on a compact domain. However, these results assume real parameters and exact operations, whereas real-world implementations on computers can only use a finite set of numbers and inexact machine operations with round-off errors. In this work, we investigate the representability of floating-point transformers that use floating-point parameters and floating-point operations. Unlike existing results under exact arithmetic, we first show that floating-point transformers can represent non-permutation-equivariant functions even without positional encoding. Furthermore, we prove that floating-point transformers can represent all permutation-equivariant functions when the sequence length is bounded, but they cannot when the sequence length is large. We also identify the minimal equivariance property in floating-point transformers, and show that all non-trivial additive positional encoding can harm the representability of floating-point transformers.

Deep Learning · Theory

Bethan Evans, Jared Tanner

The minimal norm weight perturbations of DNNs required to achieve a specified change in output are derived and the factors determining its size are discussed. These single-layer exact formulae are contrasted with more generic multi-layer Lipschitz constant based robustness guarantees; both are observed to be of the same order which indicates similar efficacy in their guarantees. These results are applied to precision-modification-activated backdoor attacks, establishing provable compression thresholds below which such attacks cannot succeed, and show empirically that low-rank compression can reliably activate latent backdoors while preserving full-precision accuracy. These expressions reveal how back-propagated margins govern layer-wise sensitivity and provide certifiable guarantees on the smallest parameter updates consistent with a desired output shift.

General Machine Learning · Representation Learning

Zeyu Michael Li, Hung Anh Vu, Damilola Awofisayo, Emily Wenger

Numerous works have noted similarities in how machine learning models represent the world, even across modalities. Although much effort has been devoted to uncovering properties and metrics on which these models align, surprisingly little work has explored causes of this similarity. To advance this line of inquiry, this work explores how two factors—dataset overlap and task overlap—influence downstream model similarity. We evaluate the effects of both factors through experiments across model sizes and modalities, from small classifiers to large language models. We find that generally, both task and dataset overlap cause higher representational similarity. Finally, we consider downstream consequences of representational similarity, demonstrating how greater similarity increases vulnerability to transferable adversarial attacks.

Social Aspects · Safety

Qizhuo Han, Xiangrui Cai, Sihan Xu, Ying Zhang, Zheli Liu

While deep neural network-based long-term time series forecasting (LTSF) has become indispensable for critical infrastructures such as smart grids and IoT platforms, the deployment of these models as black-box APIs introduces severe security vulnerabilities that remain largely underexplored. In this paper, we propose TSFAdv, a query-efficient adversarial framework for LTSF models. The framework systematically analyzes model sensitivity to spectral perturbations in both magnitude and phase of the frequency domain. By embedding frequency-domain priors into Natural Evolution Strategies, we achieve sensitivity-guided gradient estimation that improves perturbation efficacy without violating practical query constraints. To overcome ambiguities inherent to point-wise regression metrics, we adopt a trajectory-level evaluation protocol based on Dynamic Time Warping (DTW) and Slope Misalignment Error (SME), enabling the capture of complex geometric and directional deviations. Extensive experiments across seven state-of-the-art architectures demonstrate that TSFAdv achieves substantial performance gains, with average DTW improvements of 21.91–85.00% and SME improvements of 15.04–61.97% under a restrictive 200-query budget. These findings reveal that existing defense mechanisms are ineffective against frequency-domain manipulation, underscoring an urgent necessity for robust LTSF models; the code and artifacts are available at https:// anonymous.4open.science/r/TSFAdv.

Jiangning Zhang, Junwei Zhu, Teng Hu, Yabiao Wang, Donghao Luo, Weijian Cao, Zhenye Gan, Xiaobin Hu, Zhucun Xue, Xiangtai Li 等

Native 4K (2176$\times$3840) video generation remains a critical challenge due to the quadratic computational explosion of full-attention as spatiotemporal resolution increases, making it difficult for models to strike a balance between efficiency and quality. This paper proposes a novel Transformer retrofit strategy termed T3 (**T**ransform **T**rained **T**ransformer) that, without altering the core architecture of full-attention pretrained models, significantly reduces compute requirements by optimizing their forward logic. Specifically, T3-Video introduces a multi-scale weight-sharing window attention mechanism and, via hierarchical blocking together with an axis-preserving full-attention design, can effect an “attention pattern” transformation of a pretrained model using only modest compute and data. Results on 4K-VBench show that T3-Video substantially outperforms existing approaches: while delivering performance improvements (+4.29$\uparrow$ VQA and +0.08$\uparrow$ VTC), it accelerates native 4K video generation by more than 10$\times$. Demo and source code are available in \#Supp.

Theory · Online Learning and Bandits

Dhruv Sarkar, Nishant Pandey, Sayak Ray Chowdhury

Nash regret has recently emerged as a principled fairness-aware performance metric for stochastic multi-armed bandits, motivated by the Nash Social Welfare objective. Although this notion has been extended to linear bandits, existing results suffer from suboptimality in ambient dimension $d$, stemming from proof techniques that rely on restrictive concentration inequalities. In this work, we resolve this open problem by introducing new analytical tools that yield an order-optimal Nash regret bound in linear bandits. Beyond Nash regret, we initiate the study of $p$-means regret in linear bandits, a unifying framework that interpolates between fairness and utility objectives and strictly generalizes Nash regret. We propose a generic algorithmic framework, FairLinBandit, that works as a meta-algorithm on top of any linear bandit strategy. We instantiate this framework using two bandit algorithms: Phased Elimination and Upper Confidence Bound, and prove that both achieve sublinear $p$-means regret for the entire range of $p$. Extensive experiments on linear bandit instances generated from real-world datasets demonstrate that our methods consistently outperform the existing state-of-the-art baseline.

Theory · Everything Else

Antonino Emanuele Scurria, Dimitri Vanden Abeele, Bortolo Matteo Mognetti, Serge Massar

Equilibrium Propagation (EP) is a physics-inspired learning algorithm that uses stationary states of a dynamical system both for inference and learning. In its original formulation it is limited to conservative systems, i.e. to dynamics which derive from an energy function. Given their importance in applications, it is important to extend EP to nonconservative systems, i.e. systems with non-reciprocal interactions. Previous attempts to generalize EP to such systems failed to compute the exact gradient of the cost function. Here we propose a framework that extends EP to arbitrary nonconservative systems, including feedforward networks. We keep the key property of equilibrium propagation, namely the use of stationary states both for inference and learning. However, we modify the dynamics in the learning phase by a term proportional to the non-reciprocal part of the interaction so as to obtain the exact gradient of the cost function. This algorithm can also be derived using a variational formulation that generates the learning dynamics through an energy function defined over an augmented state space. Numerical experiments using the MNIST database show that this algorithm achieves better performance and learns faster than previous proposals.

Deep Learning · Large Language Models

Tatsuya Aoyama, Ethan Wilcox, Nathan Schneider

Specialized attention heads dubbed induction heads (IHs) have been argued to underlie the remarkable in-context learning capabilities of modern language models; yet, a precise characterization of their emergence, especially in the context of language modeling, remains wanting. In this study, we investigate the relationship between statistical properties of the training data and IH formation in both natural and synthetic training data settings. We show that: (1) A simple equation combining batch size and context size predicts the point at which IHs form; (2) Surface bigram repetition frequency and reliability strongly affect the formation of IHs, and we find an effective Pareto frontier in terms of these two values; (3) local dependency with high bigram repetition frequency and reliability is sufficient for IH formation, but when the frequency and reliability are low, categoriality and the shape of the marginal distribution matter.

Aili Chen, Chi Zhang, Junteng Liu, Jiangjie Chen, Chengyu Du, Yunji Li, Ming Zhong, Qin Wang, Zhengmao Zhu, Jiayuan Song 等

Recent work increasingly synthesizes agentic tasks for post-training tool-using LLMs, yet robust generalization under shifts in tasks and toolsets remains an open challenge. We trace this brittleness to insufficient diversity in synthesized training tasks. Scaling diversity is difficult because training requires tasks to remain executable and verifiable, while generalization demands diverse tool types, toolset combinations, and heterogeneous tool-use patterns. We propose DIVE, an evidence-driven recipe that inverts synthesis order, executing diverse real-world tools first and reverse-deriving tasks strictly entailed by the resulting traces, providing grounding by construction. DIVE scales structural diversity along two controllable axes, tool-pool coverage and per-task toolset variety, synthesizing 48k trajectories over 374 tools across five domains that cover 46,398 unique toolsets and 39,810 unique tool-call graphs. Training Qwen3-8B on DIVE data (48k SFT + 3.2k RL) improves by +22 average points across 9 OOD benchmarks and outperforms the strongest 8B baseline by +68%. Under a fixed budget, controlled scaling shows diversity scaling consistently outperforms quantity scaling, even with 4× less data.

Deep Learning · Large Language Models

Alexander Shabalin, Viacheslav Meshchaninov, Dmitry Vetrov

Diffusion models have achieved state-of-the-art performance in generating images, audio, and video, but their adaptation to text remains challenging due to its discrete nature. Prior approaches either apply Gaussian diffusion in continuous latent spaces, which inherits semantic structure but struggles with token decoding, or operate in categorical simplex space, which respect discreteness but disregard semantic relation between tokens. In this paper, we propose Smoothing Diffusion on Token Embeddings (Smoothie), a novel diffusion method that combines the strengths of both approaches by progressively smoothing token embeddings based on semantic similarity. This technique enables gradual information removal while maintaining a natural decoding process. Experimental results on several sequence-to-sequence and unconditional generation tasks demonstrate that Smoothie outperforms existing diffusion-based models in generation quality. Furthermore, ablation studies show that our proposed diffusion space yields better performance than both the standard embedding space and the categorical simplex.

Applications · Computer Vision

Hao Wang, Xujia Li, Lei Chen

Knowledge-Based Visual Question Answering (KB-VQA) relies on retrieval from large-scale knowledge bases, yet this stage is often treated simplistically. Existing methods typically adopt single-view indexing or naive multi-view fusion, leading to systematic coverage gaps. In this work, we demonstrate that different views exhibit strong complementarity in retrieval. Motivated by this observation, we propose SCAR, a Structure-aware Cross-View Retrieval framework that exploits cross-view structural complementarity at inference time without additional training. SCAR enhances retrieval via structure-aware similarity propagation within each view and explicit cross-view redundancy regulation. Experiments on multiple KB-VQA benchmarks demonstrate that SCAR substantially improves retrieval recall, approaches retrieval coverage upper bounds, and consistently boosts end-to-end KB-VQA performance with negligible inference overhead.

Reinforcement Learning · Deep RL

Lei Gao, Zhuoming Li, Mengxi Jia, Jiakang Yuan, Hongbo Sun, Hao Sun, Xuelong Li

Existing reinforcement learning approaches for Large Language Models typically perform policy optimization at the granularity of individual tokens or entire response sequences. However, such formulations often misalign with the natural step-wise structure of reasoning processes, leading to suboptimal credit assignment and unstable training in multi-modal reasoning tasks. To bridge this gap, we propose Segment-Aligned Policy Optimization (SAPO), a novel reinforcement learning paradigm that treats coherent reasoning steps, rather than tokens or full sequences as fundamental units of policy update. SAPO introduces a step-wise Markov decision process abstraction over reasoning segments, accompanied by segment-level value estimation, advantage computation, and importance sampling mechanisms that are semantically aligned with reasoning boundaries. Experiments on representative reasoning benchmarks demonstrate that SAPO consistently outperforms token-level and sequence-level policy optimization methods, achieving significant accuracy improvements while exhibiting better training stability and value estimation consistency. Our work underscores the importance of aligning reinforcement learning updates with the intrinsic structure of reasoning, paving the way for more efficient and semantically grounded policy optimization in complex reasoning tasks. Codes and models will be released to ensure full reproducibility.

General Machine Learning · Hardware and Software

Haoran Zhang, Zhiyu Liang, ZUO Decheng, Hongzhi Wang

Multi-tenant multi-LoRA serving multiplexes many LoRA adapters on a single GPU under high utilization, where most device memory is reserved for the KV cache, leaving only a small residency budget $K$ for adapters. In this regime, adapter readiness is atomic: if an adapter is not device-resident, the engine must perform a mandatory fetch, stalling shared execution and amplifying tail latency system-wide. With only $K$ residency slots, we identify a LoRA residency cliff: once the active adapter working set exceeds $K$, time-to-first-token (TTFT) tail latency can exhibit a congestion collapse rather than smooth degradation. To tame this cliff, we propose CLIMB, a minimal ingress controller that enforces feasibility-first admission by queueing non-resident adapters outside the engine, prioritizing critical (VIP) traffic, and rotating background adapters via round-robin. On a cliff-inducing workload, CLIMB averts collapse, reducing VIP TTFT p99 from 38.7 s to 13.1 s at matched throughput (10.66 rps) by keeping VIP engine latency near 0.13 s and shifting the residual tail into explicit ingress queueing. Overall, CLIMB shifts fetch-induced stalls from inside the engine to managed ingress queues, mitigating tail amplification without throughput loss in the evaluated settings.

Deep Learning · Graph Neural Networks

Yuting Zhang, Zhitao Xiao, Zhitao Xiao, Lei Geng, Yanwei Pang, Xiao Wang

Self-supervised Continual Graph Learning (CGL) aims to successively learn from a graph sequence with different tasks without label supervision—a paradigm that has attracted widespread attention. Most existing self-supervised CGL methods rely on instance-level consistency objectives that enforce stability of individual node (or node-pair) embeddings. Due to optimizing nodes in isolation, these methods fail to maintain global relational structure, causing inter-node correspondences to progressively distort under continual learning. To this end, we propose a novel Structure-Aware Optimal Transport (SAOT) framework that explicitly captures and preserves relational structure within graph representations across sequential tasks. Specifically, SAOT leverages optimal transport theory to capture global inter-node correspondences, thereby facilitating and enhancing graph representation learning. Simultaneously, SAOT incorporates a cross-task knowledge distillation mechanism to preserve the previous structural knowledge. Extensive experiments on four CGL benchmark datasets demonstrate that SAOT outperforms existing self-supervised baselines. In particular, SAOT achieves significant performance gains, improving average accuracy by up to 6% on CoraFull-CL and over 15% on Products-CL compared with state-of-the-art methods in the Class-IL setting.

Deep Learning · Large Language Models

Chunhan Li, Chenglin Xu, Zongyang Zhang, Jiale Liu, Zhuoxi Rao, Xudong jia, JUNXIU HE, Menglin Yang, Wenjuan Gong, Zhengzhe Liu 等

Evaluation of long-context Large Language Models (LLMs) has advanced rapidly. However, most existing benchmarks are limited to the document level and focus mainly on high-resource languages, leaving many fine-grained challenges insufficiently evaluated. To address this gap, we present MGAL, the first multilingual, granularity- and position-aware long-context benchmark. MGAL is constructed from United Nations (UN) reports spanning 8K to 128K tokens across the six official UN languages. It covers four coherent levels of linguistic granularity (word, sentence, paragraph, and document) and further stratifies entries by their position within the document (begin, middle, and end), indexed at both the document and paragraph levels. This design enables systematic diagnosis of multilingual long-context comprehension across different granularities. Through extensive experiments and analyses, we find that: (1) LLMs perform well at word-level tasks but struggle with coarser-grained ones; and (2) Closed-source models retain a clear performance advantage in lower-resource languages. We further identify two new challenges: (1) Under local semantic crowding, where neighboring sentences share topics and entities, models tend to follow surface cues (e.g., connectives like 'however' or repeated entities) rather than the discourse role of the sentence in surrounding context (e.g., background, outcome); and (2) A gap between fluency and consistency in generated outputs, where models produce text that reads smoothly but drifts from the source facts. In addition, we observe several patterns in line with prior studies, including reliance on nearby evidence and reuse of options under uncertainty.

Deep Learning · Large Language Models

Dunwei Tu, Hongyan Hao, Hansi Yang, Yihao Chen, Yu Yang, Yueqing Sun, Xingchen Liu, Furao Shen, Qi GU, Hui Su 等

Equipping agents with interactive environments and verifiable tasks for self-exploration is essential for cultivating generalist agents capable of adapting to diverse scenarios. However, high-quality agentic data remain critically scarce, and existing synthesis methods suffer from significant limitations regarding environmental diversity and scalability. To address these challenges, we introduce ScaleEnv, a framework that constructs fully interactive environments and verifiable tasks entirely from scratch. Specifically, ScaleEnv ensures environment reliability through procedural testing, and guarantees task completeness and solvability via tool dependency graph expansion and executable action verification. By enabling agents to learn through exploration within ScaleEnv, we demonstrate significant performance improvements on unseen, multi-turn tool-use benchmarks such as $\tau^2$-Bench and VitaBench, highlighting strong generalization capabilities. Furthermore, we investigate the relationship between environment and task scaling, providing empirical evidence that scaling environmental diversity is critical for robust agent learning.