论文检索

General Machine Learning · Clustering

Poojan Shah, Shashwat Agrawal, Ragesh Jaiswal

We study beyond worst case analysis for the $k$-means problem where the goal is to model typical instances of $k$-means arising in practice. Existing theoretical approaches provide guarantees under certain assumptions on the optimal solutions to $k$-means, making them difficult to validate in practice. We propose the manifold hypothesis, where data obtained in ambient dimension $D$ concentrates around a low dimensional manifold of intrinsic dimension $d$, as a reasonable assumption to model real world clustering instances. We identify key geometric properties of datasets which have theoretically predictable scaling laws depending on the quantization exponent $\varepsilon = 2/d$ using techniques from optimum quantization theory. We show how to exploit these regularities to design a fast seeding method called $\operatorname{Qkmeans}$ which provides $O(\rho^{-2} \log k)$ approximate solutions to the $k$-means problem in time $O(nD) + \widetilde{O}(\varepsilon^{1+\rho}\rho^{-1}k^{1+\gamma})$; where the exponent $\gamma = \varepsilon + \rho$ for an input parameter $\rho < 1$. This allows us to obtain new runtime - quality tradeoffs. We perform a large scale empirical study across various domains to validate our theoretical predictions and algorithm performance to bridge theory and practice for beyond worst case data clustering.

Deep Learning · Generative Models and Autoencoders

Lunjie Zhu, Yushi Huang, Xingtong Ge, Yufei Xue, Zhening Liu, Yumeng Zhang, Zehong Lin, Jun Zhang

Latent diffusion models have enabled high-quality video synthesis, yet their inference remains costly and time-consuming. As diffusion transformers become increasingly efficient, the latency bottleneck inevitably shifts to VAE decoders. To reduce their latency while maintaining quality, we propose a universal acceleration framework for VAE decoders that preserves full alignment with the original latent distribution. Specifically, we propose (1) an *independence-aware channel pruning* method to effectively mitigate severe channel redundancy, and (2) a *stage-wise dominant operator optimization* strategy to address the high inference cost of the widely used causal 3D convolutions in VAE decoders. Based on these innovations, we construct a **Flash-VAED** family. Moreover, we design a *three-phase dynamic distillation* framework that efficiently transfers the capabilities of the original VAE decoder to Flash-VAED. Extensive experiments on Wan and LTX-Video VAE decoders demonstrate that our method outperforms baselines in both quality and speed, achieving approximately a **6$\times$ speedup** while maintaining the reconstruction performance up to **96.9%**. Notably, Flash-VAED accelerates the end-to-end generation pipeline by up to **36%** with negligible quality drops on VBench-2.0.

General Machine Learning · Hardware and Software

Shihao Han, Hao Yang, Xinting Hu, Xiaofeng Mei, Yi Jiang, XIAOJUAN QI

Scaling Diffusion Transformers to generate high-resolution, long videos is constrained by the quadratic cost of self-attention, and existing sparse attention methods degrade under high sparsity. We show empirically that generation quality is determined not by the sparsity ratio itself, but by how well the sparse mask aligns with the tile-wise geometry of full attention. Based on this insight, we propose Veda, a distilled sparse attention framework that formulates tile selection as an explicit reconstruction problem from full attention. Veda integrates statistics-aware tile scoring with head-aware tiling to reduce estimation error and structural mismatch, enabling aggressive sparsity. A hardware-efficient tile-skipping kernel converts theoretical sparsity into practical wall-clock speedups. Experiments on large video diffusion models, including Waver and Wan, demonstrate substantial acceleration without quality degradation. To generate 720P 10-second videos on Waver-T2V-12B, Veda achieves a 5.1× end-to-end speedup and a 10.5× self-attention speedup, reducing attention overhead from 92% to 50%. Notably, the gains increase with sequence length, indicating that Veda scales favorably with spatiotemporal resolution across models.

Deep Learning · Self-Supervised Learning

Maofeng Tang, Hairong Qi

Vision Transformers increasingly incorporate extra tokens beyond patch tokens—from class tokens for aggregation to register tokens for artifact mitigation. While effective for their intended purposes, these tokens typically lack semantic structure. We ask a more ambitious question: Can we design regularization constraints that transform extra tokens into disentangled representations, enabling them to decompose images into semantic parts (e.g., heads, bodies, legs) without explicit supervision? We propose XTRA, an intuitive yet powerful framework that augments Vision Transformers with dedicated ``factor tokens'' and enforces disentanglement via a novel Minimum Volume Constraint (MVC). A multi-stage aggregation process further enforces these factor tokens into semantically pure components, preventing token collapse that often occurs when training with MVC alone. On ImageNet-1K, XTRA achieves superior disentanglement (8.4× improvement in SEPIN@1 over DINOv2) while simultaneously improving representation quality: KNN accuracy improves by 5.8\% and linear-probe accuracy by 2.3\%.

Applications · Social Sciences

Yanhui Sun, Wu Liu, Haifeng Ming, Xinru Wang, Hantao Yao, Yongdong Zhang

The intelligent verdict is essential for handling voluminous demands of E-commerce dispute. Unlike the legal dispute, it necessitates identifying pivotal clues from redundant multimodal evidence chains, relying on informal transaction rules for dispute verdicts. The complex ``clues-dispute" causal logic and flexible verdict rules render existing methods inadequate. Motivated by this, we propose a pioneering task, **E-commerce Dispute Verdict** (EDV), and introduce **VerdictBench**, the first Multimodal Disputes Verdicts Benchmark for E-commerce, to facilitate the intelligent verdicts. Building upon this, we propose **CyberJurors**, a framework that integrates an Individual Verdict Chain-of-Thought (IV-CoT) and Jury Consensus Verdict (JCV) to clarify the dispute logic and regulate the fair verdict process. *For the individual juror*, IV-CoT decomposes the EDV task into a structured reasoning, enabling fine-grained clues perception and explicit causal logic between clues and dispute. *For the collective jury*, JCV simulates multi-round discussion and voting among jurors guided by Verdict Precedents, effectively mitigating individual biases. Extensive experiments on VerdictBench demonstrate that CyberJurors significantly improves verdict accuracy, fairness, and interpretability, outperforming existing MLLMs by up to 9.48\% in accuracy.

Social Aspects · Security

Zihan Zhou, Yang Zhou, Jianghai Yu, Lingjuan Lyu, Longwei Wang, KC Santosh, Ruoming Jin, Dejing Dou

Recent work shows that even safety aligned large language models (LLM) can be pushed into unsafe behavior by carefully crafted jailbreak prompts. Existing jailbreaking attack methods often rely on disfluent or incoherent prompts, which limit their success and make them easy to detect. We introduce SJA, a structured jailbreak attack built around two ideas. First, inspired by the logic of Spilsbury puzzle, SJA decomposes a harmful query into a sequence of harmless sub-questions and reconstructs the original answer by combining the sub-question responses. Second, by leveraging the theory of Hamiltonian dynamics on hyperbolic space, we propose a hyperbolic Hamiltonian dynamics-based sub-question generation framework that effectively captures the structural and temporal dependencies. We provide a theoretical analysis of how each sub-question evolves along the trajectory and show that the hyperbolic Hamiltonian system effectively captures the underlying semantic structure. Finally, we propose a hyperbolic narrative fusion mechanism built on fractional embedding and Möbius fusion. This mechanism integrates coherent narratives into sub-questions while preserving geometric consistency and improving stealth performance. We theoretically validate that the combination of the generated harmless sub-questions, guided by the stealthy narrative, can effectively preserve the contextual semantics of the original harmful question.

Deep Learning · Large Language Models

Yangbo Wei, Zhen Huang, Ronghao Xu, Hong Wang, WEI XING

The rapid development of Large Language Models has driven Multi-Agent Systems (MAS) growth, but constructing efficient MAS still requires labor-intensive manual design. Current automation methods often generate templated agents, rely on monolithic optimization, and ignore task complexity gradients. This paper presents Evolutionary MAS (EvoMAS), a biologically inspired framework that addresses these limitations through three interconnected dimensions: (1) dynamic and diverse evolutionary strategies with six biologically inspired operators (3 exploration, 3 exploitation) and adaptive strategy selection; (2) role-level evolution that dynamically optimizes agent specialization and collaboration patterns; and (3) curriculum-guided evolution that partitions tasks by difficulty and evolves sequentially from simple to complex under cross-stage stability constraints. To bridge the inefficiency of pure evolutionary search and the rigidity of manual design, we introduce the Cyber Creator, a meta-control system that combines dynamic rule formulation with reflective updates. Experiments show EvoMAS consistently outperforms existing methods across multiple domains while remaining cost-efficient, with agent roles evolving from homogeneous actors to specialized reasoning ensembles.

Deep Learning · Generative Models and Autoencoders

Ming Yang, Xin Zheng, Yi Li, YIZHEN ZHENG, Huan Yee Koh, Yanqing Guo, Xiaofeng Cao, Shirui Pan

Multi-objective protein design is essential for meeting the complex demands of synthetic biology. To adapt to shifting multi-functional targets without the prohibitive cost of retraining, test-time scaling has emerged as a flexible, training-free alternative. However, current test-time diffusion methods face critical challenges: i) ineffective learning from interaction history leading to repetitive design errors, ii) over-reliance on successful cases as the reward signal, and iii) difficulties in balancing multi-objective functional trade-offs . To address these limitations, we propose MoMST, a framework for Multi-objective protein design via Memory-aware Self-contrastive learning with Test-time scaling in diffusion models. At test time, we develop a memory bank to extract generalizable reasoning experience from historical iterations. Building on this powerful experience learner, we derive rich residue-level relative preference signals from both successful and failed cases via self-contrastive learning for guiding protein generation. To ensure balance among competing multi-objective functions, we present an inference-time Pareto alignment strategy to resolve objective conflicts. Evaluations on both single-objective and complex multi-objective tasks demonstrate that MoMST exhibits remarkable performance.

General Machine Learning · Transfer, Multitask and Meta-learning

Fabian Morelli, Stephan Eckstein

Ensembles of neural networks typically outperform individual networks but incur large computational costs, whereas weight aggregation produces less costly, yet also less accurate, aggregate models. We introduce partial fusion of networks, which interpolates between ensembles and weight aggregation and thus allows for a flexible tradeoff between computational cost and performance. A direct way to achieve this is to extend existing weight aggregation methods based on neuron-level similarity between different networks, where partial fusion then only aggregates weights of neurons which are most similar. We showcase one particular method to jointly identify which neurons are most similar and match them via partial optimal transport. Further, we consider the more general perspective of weight aggregation and partial fusion as generalized pruning of ensemble models, where neurons cannot just be deleted, but also linearly combined. Finally, we show that generalized pruning applied to a single network yields similar benefits as partial fusion by allowing for a tradeoff between isolating, deleting, and linearly combining neurons based on similarity.

Deep Learning · Graph Neural Networks

Ziyu Zheng, Yaming Yang, Ziyu Guan, Wei Zhao, Xinyan Huang

Multi-domain graph pre-training is a crucial step in constructing foundational graph models with cross-domain generalization capabilities. However, existing methods predominantly rely on jointly training all source domain graphs, resulting in high computational costs. Furthermore, it remains unclear whether all source domain graph data contribute equally to effective transfer. This paper empirically reveals significant data redundancy in multi-domain graph pre-training. Based on this finding, we propose the Multi-domain Graph Pre-training Framework, MDGMIX, which combines boundary-aware subgraph mixing with hierarchical discrimination. By selecting boundary nodes to construct challenging mixed-domain subgraphs, MDGMIX employs coarse-grained domain discrimination and fine-grained domain decomposition losses to decouple shared patterns from domain-specific patterns. During adaptation, MDGMIX employs a lightweight prompt weighting mechanism to transfer source domain knowledge. Extensive experiments demonstrate that MDGMIX consistently outperforms strong baselines in few-shot classification tasks while exhibiting superior time and memory efficiency.

Deep Learning · Large Language Models

Ziqin Wang, Hengyuan Zhao, Qixin Sun, Yilin Li, Kaiyou Song, Xiaolin Hu, Qingpei Guo, Linjiang Huang, Si Liu

Mixture of Experts architectures have recently advanced the scalability and adaptability of Large Language Models for continual multimodal learning. However, extending these models to accommodate sequential tasks remains challenging. As new tasks arrive, naive model expansion leads to rapid parameter growth, while modifying shared routing components often causes catastrophic forgetting, undermining previously learned knowledge. To address these issues, we propose CoPE, a continual learning framework for LLMs that requires no replay data of previous tasks and ensures both parameter efficiency and robust knowledge retention. Our approach introduces the Probe-Guided Knowledge Extension mechanism, which uses probe experts to dynamically determine when and where new experts should be added, enabling adaptive and minimal parameter expansion tailored to task complexity. To support inference without task labels, we further incorporate a Probabilistic Task Locator that dynamically matches inputs to the correct task-specific components. To handle the practical issue that task labels are unknown during inference, we leverage a VAE-based reconstruction strategy to identify the most suitable router by matching input distributions, allowing automatic and accurate expert allocation. This design mitigates routing conflicts and catastrophic forgetting, enabling robust continual learning without explicit task labels. Extensive experiments on the CoIN benchmark, covering eight diverse VQA tasks, demonstrate that CoPE delivers strong continual learning performance with a compact model size, significantly reducing forgetting and parameter overhead compared to prior methods. These results showcase the effectiveness and scalability of our approach for parameter-efficient continual learning in large language models. Our code will be open-sourced soon.

Weichao Zhang, Shuai Zheng, Yeyu Yan, Zhizhe Liu, Zhenfeng Zhu, Yao Zhao

Graph class-incremental learning (GCIL) has emerged to address the challenge of learning from dynamically evolving graphs, which continuously learns new classes over a sequence of tasks while retaining performance on previously seen classes. However, existing GCIL methods assume a closed-set test distribution drawn only from seen tasks. This fundamentally contradicts real-world open-ended scenarios where future unknown classes inevitably emerge. Empirically, we observe that existing GCIL methods falter in such open-set settings due to severe representation drift and generalized overconfidence. To bridge this gap, we investigate the Open-Set GCIL problem and propose \textbf{SAFER} (\underline{S}ubspace-\underline{A}ware \underline{FE}ature \underline{R}eshaping), a novel framework that endows GCIL with intrinsic open-set capabilities under a replay-free constraint. Specifically, \textbf{SAFER} performs subspace-aware feature reshaping with drift-free fingerprints, unifying task routing and open-set rejection into a single energy-based metric. Furthermore, we introduce a geometric space-consistency regularization that explicitly improves intra-class compactness and suppresses cross-task representation drift. Extensive experiments on four benchmarks demonstrate that SAFER outperforms state-of-the-art baselines by margins of up to 5.2\% in accuracy and 31.3\% in open-set AUROC, all while maintaining near-zero forgetting under strict no-replay constraints.

Applications · Language, Speech and Dialog

Yangbo Wei, Zhen Huang, Shaoqiang Lu, Junhong Qian, Chen Wu, Lei He

In open-ended domains, natural language instructions are often *underspecified*, mapping to multiple valid yet functionally distinct latent intents. While Large Language Models (LLMs) excel at generation, their ability to resolve such *task ambiguity* through interaction is currently hampered by *semantic blindness*—a tendency to squander interaction budgets on distinguishing trivial syntactic variants rather than fundamental intent differences. To address this, we propose *Topological Active Inference (TAI)*, a geometric framework that recasts disambiguation as a process of *intent-manifold contraction*. TAI first leverages *Persistent Homology* to recover the topological skeleton of the solution space, theoretically guaranteeing the separation of semantic signal from syntactic noise. Subsequently, it synthesizes clarifying questions as *separating hyperplanes* designed to efficiently bisect the probability mass of the intent manifold. We introduce *Topological Expected Information Gain (TEIG)* for question selection and prove that maximizing TEIG reduces query complexity from linear $\mathcal{O}(N)$ to logarithmic $\mathcal{O}(\log K)$, where $K$ is the number of latent intents. Extensive experiments demonstrate that TAI recovers user intent with significantly fewer turns, achieving state-of-the-art disambiguation efficiency.

General Machine Learning · Clustering

Yiming Wang, Qun Li, Dongxia Chang, Jie Wen, Hua Dai, Fu Xiao

Deep Multi-View Clustering (MVC) aims to extract a unified semantic consensus from diverse data sources without supervision. However, current approaches relying on flat Euclidean embeddings often fail to model data uncertainty, resulting in rigid alignment where high-quality views are forced to drift toward corrupted ones. To address these challenges, we propose the Hyperbolic Asymmetric Multi-view Clustering (HAMC) framework. By embedding features into the Poincaré ball model, HAMC leverages the exponential volume growth of hyperbolic geometry to optimize cluster separability. It pushes high-confidence representations toward the boundary while retaining noisy ones near the origin. To mitigate noise, we introduce an asymmetric view alignment mechanism, enabling reliable views to unidirectionally guide unreliable ones. Furthermore, a consensus-aware cluster learning strategy is designed to construct robust global pseudo-labels via a confidence-based screening scheme, refining the cluster structure. Extensive experiments against 13 baselines demonstrate that HAMC achieves state-of-the-art performance.

Theory · Reinforcement Learning and Planning

Abdullah Akgül, Gulcin Baykal, Manuel Haussmann, Mustafa Mert Çelikok, Melih Kandemir

Optimal control of complex environments with robotic systems faces two complementary and intertwined challenges: efficient organization of sensory state information and far-sighted action planning. Because the reinforcement learning framework addresses only the latter, it tends to deliver sample-inefficient solutions. Active inference is the state-of-the-art process theory that explains how biological brains handle this dual problem. However, its applications to artificial intelligence have thus far been limited to extensions of existing model-based approaches. We present a formal abstraction of reinforcement learning algorithms that spans model-based, distributional, and model-free approaches. This abstraction seamlessly integrates active inference into the distributional reinforcement learning framework, making its performance advantages accessible without transition dynamics modeling.

General Machine Learning · Causality

Yiwen (Evie) Qiu, Filip Kovačević, Shimeng Huang, Peter Spirtes, Francesco Locatello

Selection bias is pervasive in observational studies. For example, large scale biobanks data can exhibit ``healthy volunteer bias'' when respondents are healthier and of higher socio-economic status than the population they are meant to represent. Recovering causal effects from such sub-population is an important problem in causal inference, as estimating average treatment effects (ATE) from selected populations can result in a severely biased estimate of the ATE from the whole population. In this paper, we investigate the identifiability of the ATE under selection bias. We provide *necessary and sufficient conditions* for ATE identifiability, leveraging weak assumptions on probability classes to characterize propensity score and selection probability. Compared to previous works, our results extend existing graphical identifiability criteria and offer a more comprehensive understanding of causal effect identification *with strictly weaker conditions* in the presence of selection bias.

Deep Learning · Theory

Yuri Kinoshita, Naoki Nishikawa, Taro Toyoizumi

Dataset distillation, a training-aware data compression technique, has recently attracted increasing attention as an effective tool for mitigating costs of optimization and data storage. However, progress remains largely empirical. Mechanisms underlying the extraction of task-relevant information from the training process and the efficient encoding of such information into synthetic data points remain elusive. In this paper, we theoretically analyze practical algorithms of dataset distillation applied to the gradient-based training of two-layer neural networks with width $L$. By focusing on a non-linear task structure called multi-index model, we prove that the low-dimensional structure of the problem is efficiently encoded into the resulting distilled data. This dataset reproduces a model with high generalization ability for a required memory complexity of $\Theta(r^2d+L)$, where $d$ and $r$ are the input and intrinsic dimensions of the task. To the best of our knowledge, this is one of the first theoretical works that include a specific task structure, leverage its intrinsic dimensionality to quantify the compression rate and study dataset distillation implemented solely via gradient-based algorithms.

Optimization · Large Scale, Parallel and Distributed

zhixin wang, Jiaming Xu, Tianyi Zhou, Mingjun Zhang, Liming Liu, JiaruiHu, Dian Yang, TongYu Wang, Ping Zhang, Jinlong Hou 等

Effectively scaling Reinforcement Learning (RL) is crucial for enhancing the reasoning and alignment of Large Language Models. The massive data and complex execution flows inherent in these tasks require a distributed architecture capable of efficient scaling. However, to simplify programming and dependency management, mainstream frameworks often rely on a centralized architecture where a single node dispatches both control and data. This inherent coupling creates significant communication bottlenecks, severely limiting system scalability and efficiency. We present DistFlow, a novel, fully distributed RL framework that adopts a multi-controller paradigm. By decoupling data transmission from control dispatch, DistFlow establishes a parallelism-aware, decentralized Data Coordinator that leverages local caching, load balancing, and asynchronous double buffer to minimize communication overhead and mitigate straggler effects. For control logic, it introduces a task scheduler built upon Directed Acyclic Graph (DAG) that facilitates fine-grained, independent execution. Experimental results demonstrate that DistFlow achieves near-linear scalability up to 512 GPUs and delivers up to a 2.63x throughput improvement over state-of-the-art (SOTA) frameworks.

Theory · Reinforcement Learning and Planning

Yifan Jiang, Jiasheng Pan, Mengtian Li, Li Jin

We study decentralized multi-agent reinforcement learning (MARL) for networked service systems with affinity in the presence of Byzantine nodes. The way that a server processes a job depends on an affinity state that captures the correlation between the job and the server. Each node learns a local control policy via an actor-critic algorithm with linear function approximation over inherently unbounded space of traffic states, while exchanging parameter information with neighbors through a communication graph. A set of Byzantine agents can exploit the unbounded state space and the resulting stochastic variance to compromise the consensus mechanism, destabilizing both learning and queuing processes. To address this vulnerability, we propose a resilient consensus-based MARL algorithm with momentum-based smoothing, which mitigates adversarial parameter manipulation and guarantees traffic stability under mild assumptions. We prove that the cooperative agents’ policies converge almost surely to a bounded neighborhood of a stationary solution of the global objective. We demonstrate the effectiveness and generality of the proposed framework in several representative service systems, including semantic routing for large language model serving, distributed polling in cloud computing, and smart manufacturing logistics.

Applications · Chemistry, Physics, and Earth Sciences

Minghan Li, fengji Li, Yilin Tao, Yue Deng

Computational protein design typically employs a sequential workflow of structure generation followed by sequence (re)design. While structure generators can be explicitly conditioned on functional objectives, inverse folding models are constrained by their function-agnostic nature and sequence-structure degeneracy. More critically, the associated training objectives do not account for the *Best-of-N* (BoN) inference protocol, resulting in a fundamental training-inference misalignment. Here, we propose FIDIA, a reinforcement learning framework that enables **F**unction-**I**nformed sequence **D**esign via **I**nference-**A**ligned policy optimization. Specifically, FIDIA integrates functional constraints into composite rewards and explicitly optimize the induced policy under BoN toward high-fitness sequence regions. We achieve this via a grounded gradient estimator that directly maximizes the expected maximum reward. FIDIA consistently outperforms both standard and RL-optimized baselines in success rate and precision on a general motif scaffolding benchmark. Further experiments on realworld cases including vaccine and affinity-enhancing enzyme design validate FIDIA’s efficacy in complex therapeutic and biocatalytic contexts.