论文检索

Optimization · Stochastic

Amit Attia, Tomer Koren

The learning rate in stochastic gradient methods is a critical hyperparameter that is notoriously costly to tune via standard grid search, especially for training modern large-scale models with billions of parameters. We identify a theoretical advantage of learning rate annealing schemes that decay the learning rate to zero at a polynomial rate, such as the widely-used cosine schedule, by demonstrating their increased robustness to initial parameter misspecification due to a coarse grid search. We present an analysis in a stochastic convex optimization setup demonstrating that the convergence rate of stochastic gradient descent with annealed schedules depends *sublinearly* on the multiplicative misspecification factor $\rho$ (i.e., the grid resolution), achieving a rate of $\smash{O(\rho^{1/(2p+1)}/\sqrt{T})}$ where $p$ is the degree of polynomial decay and $T$ is the number of steps. This is in contrast to the $\smash{O(\rho/\sqrt{T})}$ rate obtained under the inverse-square-root and fixed stepsize schedules, which depend linearly on $\rho$. Experiments confirm the increased robustness compared to tuning with a fixed stepsize, that has significant implications for the computational overhead of hyperparameter search in practical training scenarios.

General Machine Learning · Causality

Roel Hulsman, Carles Balsells-Rodas, Sara Magliacane

Temporal systems often exhibit non-stationary behaviour, such as seasonal climate variation or glucose fluctuations in patients with type-1 diabetes. One way to model non-stationarity is through discrete latent regimes, i.e., stationary segments of time. Such systems induce a Markov Switching Model (MSM), a class of Hidden Markov Models with autoregressive dependencies among latent regimes and observed variables. Identifying latent regimes is challenging in the presence of frequent regime switches and nonlinear and non-Gaussian dynamics, particularly when there are instantaneous effects between the variables, e.g., due to slow rates of measurements. In this work, we establish the identifiability of both latent regimes and regime-dependent causal structures under temporal regime dependencies, nonlinear lagged and instantaneous effects, and independent noise from the exponential family. Our identifiability theory subsumes non-temporal mixtures of causal models. Furthermore, we introduce FlowMSM, a regime detection framework that can be paired with any stationary causal discovery method to recover regime-dependent causal structures. Experiments on synthetic benchmarks and a financial economics dataset demonstrate the effectiveness of our approach to detect latent regimes and discover causal structures from non-stationary time series.

Deep Learning · Large Language Models

Xinyi Hu, Yuhao Shen, Zhang Baolin, Hengxin Zhang, Jun Dai, Shuang Ge, Yue Li, Mingcheng Wan

Speculative Decodin promises to accelerate Large Language Model inference, yet its efficacy often degrades in production-grade scenarios. Existing evaluations typically overlook the compute-bound nature of high-concurrency regimes, where verification compute becomes the dominant bottleneck. Consequently, prior methods face a dilemma: static trees incur massive verification waste, while dynamic trees suffer from cumulative misjudgments and kernel incompatibility. To bridge this gap, we introduce ECHO, a high concurrency-oriented framework integrated into SGLang that reformulates speculative execution as a budgeted scheduling problem. Crucially, ECHO employs sparse confidence gating to manage the batch as a unified super-tree, elastically pivoting budget between depth and width to co-optimize the trade-off between reducing global verification steps and maximizing per-step efficiency. Extensive evaluations across diverse model scales—particularly the industrial-grade Qwen3-235B—demonstrate that ECHO consistently outperforms state-of-the-art baselines in both low-load and high-load scenarios, achieving up to 5.35$\times$ walltime speedup and delivering over 20\% relative speedup gain against the strongest baselines.

Deep Learning · Generative Models and Autoencoders

Xuanyi Liu, Deyi Ji, Liqun Liu, Lanyun Zhu, Xuhang Chen, Qianxiong Xu, Peng Shu, Huan Yu, Jie Jiang, Feng Gao 等

Sparse camera-conditioned image-to-video generation presents a pivotal challenge: synthesizing geometrically consistent 3D motion from minimal pose cues. Existing methods, which largely rely on dense supervision or naive interpolation, suffer from severe pose drift and motion discontinuities due to the lack of robust 3D priors. In this paper, we introduce \textbf{CamGeo}, a novel framework that distills rich 3D geometric knowledge from a pre-trained video-to-3D model (VGGT) directly into the diffusion backbone. To achieve this without incurring inference latency, we propose a training-only distillation strategy. Specifically, CamGeo incorporates: (1) keyframe trajectory distillation that enforces cycle-consistency with sparse input poses, (2) cross-frame consistency distillation with both camera trajectory and depth constraints to generate consistent structure across unsupervised frames, and (3) a three-stage coarse-to-fine curriculum learning, progressively scales geometric complexity, from global structure coherence to fine-grained refinement, achieving stable optimization. Extensive experiments demonstrate that CamGeo achieves consistent improvements under various sparsity ratios.

Xinyi Hu, Yuhao Shen, Zhang Baolin, Hengxin Zhang, Jun Dai, Shuang Ge, Chen Lei, Yue Li, Mingcheng Wan

Speculative Decodin promises to accelerate Large Language Model inference, yet its efficacy often degrades in production-grade scenarios. Existing evaluations typically overlook the compute-bound nature of high-concurrency regimes, where verification compute becomes the dominant bottleneck. Consequently, prior methods face a dilemma: static trees incur massive verification waste, while dynamic trees suffer from cumulative misjudgments and kernel incompatibility. To bridge this gap, we introduce ECHO, a high concurrency-oriented framework integrated into SGLang that reformulates speculative execution as a budgeted scheduling problem. Crucially, ECHO employs sparse confidence gating to manage the batch as a unified super-tree, elastically pivoting budget between depth and width to co-optimize the trade-off between reducing global verification steps and maximizing per-step efficiency. Extensive evaluations across diverse model scales—particularly the industrial-grade Qwen3-235B—demonstrate that ECHO consistently outperforms state-of-the-art baselines in both low-load and high-load scenarios, achieving up to 5.35$\times$ walltime speedup and delivering over 20\% relative speedup gain against the strongest baselines.

Deep Learning · Robustness

Gabriel Afriat, Hussein Hazimeh, Dimitris Paparas, Rahul Mazumder

Pruning is a common approach to reduce the memory footprint and inference cost of large vision and language models. As these architectures continue to scale, one-shot pruning methods - i.e. approaches that prune the network without any retraining - have become increasingly attractive. Many popular one-shot pruning methods (e.g., WoodFisher, CAP, SparseGPT, and ALPS) typically optimize a quadratic objective under sparsity constraints. However, in practice, this objective is affected by multiple sources of uncertainty, including noise in the calibration data and variability introduced by algorithmic updates. To address these issues, we introduce RobOP, a robust optimization framework that explicitly accounts for such uncertainties. RobOP is modular and flexible, and can be applied with any existing pruning method through simple modifications motivated by our theoretical framework. We demonstrate that by taking into account uncertainty, RobOP offers improvements over prior pruning approaches. Our framework applies tractably across a range of stylized uncertainty sets, enabling robust one-shot pruning at scale.

Qing Zhou, Hongyuan Zhang, Tao Yang, Junyu Gao, Qi Wang

Token merging accelerates Transformer inference by clustering similar tokens to reduce sequence length (retention ratio $r$), but distorts attention outputs, inducing covariate shift in residual streams and performance collapse under high compression. Existing heuristics, such as proportional attention, mitigate mild compression effectively but degrade sharply at aggressive ratios due to unaddressed energy drift and biased attention distributions. We reframe token merging as a statistical reconstruction problem in high dimensions and introduce an asymptotic radial-angular decomposition of the reconstruction error, an analytical framework decoupling magnitude and distributional distortions. Minimizing this decomposed risk under minimal assumptions of finite second moments and variance stationarity yields closed-form optimal corrections governed by a single scaling factor $\sqrt{r}$: scaling merged values and shrinking merged logits toward the cluster-size prior. This calibrates both energy balance and distributional fidelity. Extensive experiments on vision Transformers demonstrate superior accuracy and robustness across compression levels.

Applications · Health / Medicine

Ruihan Guo, Chaoran Cheng, Zhanghan Ni, Neil He, Bangji Yang, Ge Liu

Predicting protein mutation effects is fundamental to protein engineering and disease variant interpretation, yet experimental mutation data remain accurate but extremely sparse. Large-scale computational augmentation offers scalability, but introduces heterogeneous and poorly calibrated supervision signals arising from distinct modeling paradigms. We construct a PDB-wide, structure-aligned mutation augmentation dataset that exhaustively enumerates single-site substitutions and aligns mutation signals from physics-based energy models, protein language models, and structure-conditioned inverse folding models. Large-scale analysis under a unified mutation preference representation reveals substantial differences in preference structure, confidence, and cross-model agreement, indicating that disagreement is pervasive and reflects conflicting inductive biases rather than random noise. Motivated by these observations, we propose an unsupervised multi-source mutation preference distillation framework that learns from relative mutation preferences while explicitly modeling cross-source disagreement. Without using any experimental mutation labels during training, our approach achieves consistent improvements on the ProteinGym benchmark over zero-shot baselines and naive multi-source fusion strategies. We release the dataset and evaluation pipeline to support reproducible studies of protein mutation effects.

Deep Learning · Other Representation Learning

Ying Wang, Oumayma Bounou, Gaoyue Zhou, Randall Balestriero, Tim G. J. Rudner, Yann LeCun, Mengye Ren

Learning good representations is essential for latent planning with world models. While pretrained visual encoders provide strong visual features, they are not tailored to planning and contain substantial information which is irrelevant to planning. Inspired by the perceptual straightening hypothesis in human visual processing, we introduce temporal straightening for representation learning in latent planning. We add a lightweight projector on top of a pretrained visual encoder to map to a lower-dimensional space, trained with a curvature regularizer that encourages locally straightened latent trajectories. We show that reducing curvature improves the conditioning of the planning objective, making gradient-based planning more stable and yielding significantly higher success rates across four goal-reaching tasks.

Deep Learning · Large Language Models

Xianjie Liu, Yiman Hu, Yixiong Zou, Liang Wu, Jian Xu, Bo Zheng

Multimodal Large Language Models (MLLMs) have made substantial progress on visual understanding tasks, yet they still perform poorly on high-resolution images. Prior work often attributes this limitation to perceptual constraints, arguing that MLLMs fail to recognize small objects and therefore rely on "zoom-in" strategies to recover fine details. In contrast, our analysis shows that the dominant failure mode is background interference rather than object size. We study the "zoom-in" operation through a **hierarchical decoupling analysis** and propose the **Hierarchical Decoupling Framework (HiDe)**, a training-free method that turns implicit attention into explicit region selection. HiDe first performs Token-wise Attention Decoupling (TAD) to disentangle question semantics and identify the most informative tokens, then uses their attention patterns to pinpoint the corresponding visual regions. It subsequently applies Layout-Preserving Decoupling (LPD) to extract these regions from cluttered backgrounds and construct a compact representation that retains key spatial structure while filtering out irrelevant context. HiDe achieves state-of-the-art results on high-resolution benchmarks like V\*Bench, HRBench4K, and HRBench8K. It boosts Qwen2.5-VL 7B and InternVL3 8B to state of the art performance, reaching 92.1\% and 91.6\% on V\*Bench, and even surpasses reinforcement learning based methods. After optimization, HiDe reduces memory usage by 75\% compared with the previous training-free approach.

General Machine Learning · Transfer, Multitask and Meta-learning

Aecheon Jung, Seunghwan Lee, Dongyoon Han, Sungeun Hong

Model merging combines independently trained models into a single multi-task model. However, most existing approaches focus primarily on avoiding task interference. We argue that its greater potential lies in enabling task synergy, where tasks actively improve one another. We identify cross-task performance, defined by compatibility between encoders and predictors across tasks, as a key indicator of merge quality. We demonstrate that adapting only a single task-specific layer is sufficient to induce such synergy. This study proposes SyMerge, a lightweight framework that jointly optimizes merging coefficients and a single task-specific layer. We adopt an expert-guided self-labeling objective, providing stable supervision beyond entropy minimization. Intriguingly, we further show that SyMerge successfully merges models trained from different initializations, a regime where standard methods break down. Our minimalist yet principled method achieves state-of-the-art results across vision, dense prediction, and NLP benchmarks.

Applications · Computer Vision

Shangze Li, Yifan Xu, Jingmiao Liang, Yongfei Zhang, Yuzhuo Ma, Yingbo Qu

While Vision Transformers (ViTs) offer strong global modeling, their quadratic computational cost limits utility in latency-sensitive applications like person re-identification (ReID). Existing compression strategies, such as token pruning or generic merging, typically rely on coarse-grained criteria tailored for image classification. In fine-grained retrieval, these approaches often discard or smooth out subtle but discriminative local details. To resolve this, we propose SRE-Merge, a training-free framework designed for discriminative token compression. SRE-Merge injects spatial priors into the merging process through three mechanisms: (i) Spatial-Entropy Saliency Assessment (SES-Assess), which quantifies token importance as Spatial-Entropic Mass (SE-Mass) by coupling spatial structure with local attention entropy; (ii) Hybrid Context-Affinity Matching (HCA-Match), which guides precise pair selection by combining feature similarity with mass-derived context; and (iii) Energy-Preserving Weighted Fusion (EPW-Fuse), which incorporates SE-Mass weighting to counteract feature variance reduction. Extensive experiments on standard benchmarks show that SRE-Merge reduces GFLOPs of the base ViT model by about 24\% while maintaining state-of-the-art accuracy, establishing a superior accuracy-efficiency trade-off.

Reinforcement Learning · Everything Else

Hong Je-Gal, Hyun-Suk Lee

Scheduling problems arise from repeatedly selecting one item from a set of candidates based on their states. These problems often reduce to assigning priority scores and choosing the highest-ranked item. In this work, we propose a factorized scheduling principle (FSP) framework to learn interpretable and transferable scheduling rules. The FSP framework represents system states as condition distributions and decomposes a global scheduling principle into additive univariate and pairwise components with identifiability constraints. The scheduling principle enables the framework to maintain a simple priority-based structure during deployment. This principle is learned by using a policy-based objective combined with a temporal-difference signal defined on the condition distribution. Experiments on synthetic and realistic scheduling tasks demonstrate the FSP framework's strong performance, interpretability, and zero-shot generalization across different system scales.

Deep Learning · Large Language Models

Baochen Fu, Yuntao Du, Cheng Chang, Baihao Jin, Wenzhi Deng, Muhao Xu, Hongmei Yan, Weiye Song, Yi Wan

As real-world knowledge continues to evolve, the parametric knowledge acquired by multimodal models during pretraining becomes increasingly difficult to remain consistent with real-world knowledge. Existing research on multimodal knowledge updating focuses only on learning previously unknown knowledge, while overlooking the need to update knowledge that the model has already mastered but that later changes; moreover, evaluation is limited to the same modality, lacking a systematic analysis of cross-modal consistency. To address these issues, this paper proposes MMKU-Bench, a comprehensive evaluation benchmark for multimodal knowledge updating, which contains over 25k knowledge instances and more than 49k images, covering two scenarios, updated knowledge and unknown knowledge, thereby enabling comparative analysis of learning across different knowledge types. On this benchmark, we evaluate a variety of representative approaches, including supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and knowledge editing (KE). Experimental results show that SFT and RLHF are prone to catastrophic forgetting, while KE better preserve general capabilities but exhibit clear limitations in continual updating. Overall, MMKU-Bench provides a reliable and comprehensive evaluation benchmark for multimodal knowledge updating, advancing progress in this field.

Deep Learning · Large Language Models

Zijun Wang, Haoqin Tu, Weidong Zhou, Yiyang Zhou, Xiaohuan ZHOU, Bingni Zhang, Weiguo Feng, Taifeng Wang, Cihang Xie, Fengze Liu

Everyday tasks come with a target, and pretraining models around this target is what turns them into experts. In this paper, we study target-oriented language model (LM) pretraining by introducing ***N**euron-**A**ctivated **G**raph Ranking* (NAG-based Ranking), a training-free and interpretable framework for target pretraining data selection. Rather than using black-box representations, our approach directly characterizes each target input by a sparse set of high-impact neurons in any off-the-shelf LLMs. Concretely, we quantify neuron impact and select the most influential neurons across layers into a compact ***N**euron-**A**ctivated **G**raph* (NAG), and rank candidate data by NAG similarity to target examples. We conduct experiments across six benchmarks, where our NAG-based Ranking improves target-oriented pretraining by 4.9\% on average over random sampling, and also outperforms state-of-the-art baselines by 5.3\% accuracy on HellaSwag. It also remains effective under a more applicable multi-target setting, where our best setup surpasses two baselines by 1.1\% and 4.1\%, respectively. Furthermore, we provide a comprehensive analysis on *why* and *how* our NAG works, e.g., deactivating NAG-selected neurons (only 0.12\% of all) causes a 23.5\% performance collapse, and restricting NAG to the final layer incurs a 4.1\% average drop, indicating that NAG captures a sparse ``functional backbone'' for learning target features.

Applications · Computer Vision

Wenbin Wang, Yuge Huang, Jianqing Xu, Yue Yu, Jiangtao Yan, Shouhong Ding, Pan Zhou, Yong Luo

Rapid advances in AI-generated image (AIGI) technology enable highly realistic synthesis, threatening public information integrity and security. Recent studies have demonstrated that incorporating texture-level artifact features alongside semantic features into multimodal large language models (MLLMs) can enhance their AIGI detection capability. However, our preliminary analyses reveal that artifact features exhibit high intra-feature similarity, leading to an almost uniform attention map after the softmax operation. This phenomenon causes attention dilution, thereby hindering effective fusion between semantic and artifact features. To overcome this limitation, we propose a lightweight fusion adapter, TranX-Adapter, which integrates a Task-aware Optimal-Transport Fusion that leverages the Jensen-Shannon divergence between artifact and semantic prediction probabilities as a cost matrix to transfer artifact information into semantic features, and an X-Fusion that employs cross-attention to transfer semantic information into artifact features. Experiments on standard AIGI detection benchmarks upon several advanced MLLMs, show that our TranX-Adapter brings consistent and significant improvements (up to +6% accuracy). Code can be found in supplementary material.

Theory · Domain Adaptation and Transfer Learning

Tobias Wegel, Federico Di Gennaro, Geelon So, Fanny Yang

When a learner is faced with a new task, but is given very few samples, it must leverage any available side-information. In practice, this often comes in the form of benchmarks, where there is abundant data to evaluate model performance on related tasks. Though task relatedness is difficult to formalize theoretically, it can be empirically observed through *weak monotonicity*: if a model dominates another on all benchmarks, it also tends to outperform the other on the new task. We explore the statistical complexity of learning under weak monotonicity, leveraging it within two learning paradigms: transfer learning and model selection aggregation. We show that not only can we prune the model class based on monotonicity, but that we can further adapt to the geometry of the available trade-offs by *hedging on the frontier*.

Yanjiang Guo, Tony Lee, Lucy Xiaoyang Shi, Jianyu Chen, Percy Liang, Chelsea Finn

The goal of this paper is to improve the performance and reliability of vision-language-action (VLA) models through iterative online interaction. Since collecting policy rollouts in the real world is expensive, we investigate whether a learned simulator—specifically, an action-conditioned video generation model—can be used to generate additional rollout data. Unfortunately, existing world models lack the physical fidelity necessary for policy improvement: they are predominantly trained on demonstration datasets that lack coverage of many different physical interactions (particularly failure cases) and struggle to accurately model small yet critical physical details in contact-rich object manipulation. We propose a simple iterative improvement algorithm that uses real-world roll-out data to improve the fidelity of the world model, which can then, in turn, be used to generate supplemental synthetic data for improving the VLA model. In our experiments on a real robot, we use this approach to improve the performance of a state-of-the-art VLA model on multiple downstream tasks. We achieve a 39.2\% absolute success rate improvement over the base policy and 11.6\% improvement from training with the generated synthetic rollouts. Videos can be found at this anonymous website: \url{https://sites.google.com/view/vla-w}.

Applications · Health / Medicine

Hengyuan Cao, Shizhuo Cheng, Mingxuan Liu, Weicheng Huang, Yunhong Lu, chenxi cai, Yan Zhang, Min Zhang

The rapid evolution of generative models has unlocked new potentials in protein binder design, a pivotal task in structural biology, by facilitating end-to-end generation via joint sequence-structure modeling or hallucination. However, existing approaches are predominantly implemented under a single-target, single-state assumption, limiting their ability to model multi-target or multi-state interactions required for advanced function-oriented protein design. Here, we introduce Chamaileon, which unifies multi-target and multi-state binder design by formulating the problem as cross-context binding landscape modeling. The framework is underpinned by a training paradigm termed \textit{In-Context Complex Co-Design (I3CD)} for context-aware sequence-structure co-modeling. During inference, we employ \textit{Mixture-of-Paths Sampling (MoPS)}, a scalable strategy that optimizes a single sequence across contexts while alleviating the scarcity of high-quality multi-conformational paired data. Extensive evaluation on our newly constructed benchmark, \textit{CROSS}, demonstrates that Chamaileon effectively generates sequences adaptable to diverse conformational landscapes and multi-target requirements.

Deep Learning · Robustness

Ziteng Hong, Bingzhi Chen, Yishu Liu, Sudong Cai, Guangming Lu

Real-world multimodal systems inevitably face partial observability due to sensor dropout and degradation. Standard robustness methods can improve average performance, but they often remain unreliable in rare, adverse long-tail conditions. Under a locked same-support contract, we uncover a same-support tail failure where two runs with matched average performance still diverge on worst-case and shift-reweighted metrics computed on the same observable support. We attribute this instability to conditional interaction where environment coverage does not guarantee parameter exposure. Information is routed through different parameter groups, leaving some groups rarely updated even when environment coverage is complete, and tail-focused aggregation amplifies errors from these high-leverage but underexposed groups. This mismatch is auditable from gating logs, and we summarize it with TailPressure, an exposure-normalized statistic of tail-leveraged interaction. Guided by this diagnosis, we propose Heterogeneity-aware Closed-loop Exposure Stabilizer (H-CES), a controller that regulates per-group pressure via deterministic increment-branch gating and group-wise decoupled weight decay, without changing the task loss or adding inference branches. Experiments across diverse multimodal settings and backbones show that H-CES improves tail reliability under the same-support contract while preserving clean performance.