论文检索

Deep Learning · Theory

Jingwen Liu, Ezra Edelman, Surbhi Goel, Bingbin Liu

As data scaling approaches its natural limits, recent work has explored the potential of data reuse, whose impact on optimization remains poorly understood. This work investigates how data repetition affects learning efficiency in terms of compute. We investigate an inverse-scaling effect that using less data can lead to faster convergence, which is observed across algorithmic tasks, architectures and optimizers and cannot be explained using prior theory. Instead, we argue that the speedup comes from appropriate layer-wise norm growth, which can be achieved faster when the dataset size is smaller. We provide theoretical justification by analyzing the benefits of sampling biases induced by small datasets, and we present extensive empirical evidence supporting this hypothesis. Together, our results highlight the potential of unlocking more efficiency gain by jointly considering different scaling axes.

Probabilistic Methods · Bayesian Models and Methods

Weifeng Zhang, Wenyuan Zhao, Amir Hossein Rahmati, Yucheng Wang, Zhiyuan Wang, Chao Tian, Xiaoning Qian

Principled uncertainty quantification (UQ) is increasingly recognized as essential for trustworthy artificial general intelligence (AGI). Bayesian Low-Rank Adaptation (LoRA) provides a principled mechanism for uncertainty-aware fine-tuning of large language models (LLMs). However, existing techniques either face scalability constraints, e.g. Laplace-LoRA, or rely on approximate inference schemes that lead to poorly calibrated posterior uncertainty, often manifesting as overconfident predictions under distribution shift. To address this challenge, we propose GPan-LoRA, the first scalable Gaussian Process (GP)-based framework for Bayesian LoRA, which integrates neural network-based sparse GP approximations with amortized variational inference. By preserving the Bayesian function prior and posterior semantics intrinsic to GPs, GPan-LoRA achieves a faithful balance between computational scalability and principled UQ. Empirically, GPan-LoRA produces well-calibrated uncertainty that remains reliable under distribution shift, mitigating overconfident failures while preserving competitive task performance.

Deep Learning · Large Language Models

Wenbo An, Lianwei Wu, Zehao Wang

Watermarking has emerged as a critical solution for the detection and provenance tracing of content generated by large language models. However, existing methods still suffer from significant limitations, including difficulties in achieving personalized attribution, substantial degradation of generation quality, and weak robustness against attacks. To address these challenges, we propose IPMark, the first IP-inspired hierarchical personalized watermarking framework. Specifically, to enable personalization and efficient detection, IPMark employs a hierarchical addressing framework to structurally organize model and user identities. Subsequently, addressing the inherent semantic distortion caused by token-level watermarking, we design a semantic-syntactic dual-stream embedding strategy. Centered on sentence-level candidate selection and reinforced by dual signals from syntactic and semantic features, this approach optimizes the injection process, thereby significantly enhancing generation quality while ensuring strong robustness. Experimental results demonstrate that IPMark achieves the lowest perplexity among baselines, ensuring superior generation quality while maintaining strong robustness and significantly reducing detection latency through hierarchical retrieval.

Social Aspects · Safety

Xikang Yang, Biyu Zhou, Xuehai Tang, Jizhong Han, Songlin Hu

Large language models (LLMs) are increasingly deployed in high-stakes settings, yet they remain vulnerable to sequential jailbreaks that exploit multi-turn interaction to circumvent safety mechanisms. Current safety evaluations are largely outcome-based, offering little insight into the latent decision processes that lead to unsafe compliance. We propose an interpretable cognitive modeling framework that couples a controlled elicitation paradigm, the Contextual Iowa Gambling Task (C-IGT), with a Generalized Rescorla--Wagner (GRW) architecture to decompose behavior into measurable mechanisms. Across a diverse set of mainstream LLMs, we find that sequential vulnerability is not explained by scale alone but emerges from interactions among cognitive factors, including optimism-biased learning, perceptual reward amplification, and choice inertia. Moreover, counterfactual feedback and psychologically framed rewards (e.g., regret, authority, threat) substantially accelerate the transition from refusal to compliance. These results yield principled cognitive profiles of LLM ``irrationality'' and provide insights for interdisciplinary research on LLM agents at the intersection of machine learning and human behavioral science.

Deep Learning · Large Language Models

Luke Huang, Zhuoyang Zhang, Qinghao Hu, Shang Yang, Song Han

Reinforcement learning (RL) is widely used to improve large language models (LLMs) on reasoning tasks, and asynchronous RL training is attractive because it increases end-to-end throughput. However, for widely adopted critic-free policy-gradient methods such as REINFORCE and GRPO, high asynchrony makes the policy-gradient estimator **markedly higher variance**: stale off-policy rollouts induce heavy-tailed importance ratios, causing a small fraction of samples to dominate each update. This amplification makes gradients noisy and learning unstable relative to matched on-policy training. Across math and reasoning benchmarks, we find collapse is preceded by sharp drops in effective sample size (ESS) and unstable gradient norms. Motivated by this diagnosis, we propose **V**ariance **C**ontrolled **P**olicy **O**ptimization (VCPO), a drop-in stabilization method for REINFORCE/GRPO-style algorithms that (i) **rescales learning rate according to effective sample size** to dampen unreliable updates, and (ii) applies a **closed-form minimum-variance baseline** for the off-policy setting, avoiding an auxiliary value model and adding minimal overhead. Empirically, VCPO substantially improves robustness in highly asynchronous regimes across models sizes and tasks, reducing long-context, multi-turn training compute by 1.96×. Overall, our results demonstrate explicitly controlling policy-gradient variance is key to making asynchronous RL reliable at scale.

Deep Learning · Generative Models and Autoencoders

Harold Haodong Chen, Xinxiang Yin, Wenjie Shu, Hongfei (Faye) Zhang, Zixin Zhang, Chenfei Liao, Litao Guo, Qifeng Chen, YINGCONG CHEN

Text-to-image (T2I) generation has achieved remarkable progress, yet existing methods often lack the ability to dynamically reason and refine during generation--a hallmark of human creativity. Current reasoning-augmented paradigms mostly rely on explicit thought processes, where intermediate reasoning is decoded into discrete text at fixed steps with frequent image decoding and re-encoding, leading to inefficiencies, information loss, and cognitive mismatches. To bridge this gap, we introduce **LatentMorph**, a novel framework that seamlessly integrates implicit latent reasoning into the T2I generation process. At its core, LatentMorph introduces four lightweight components: (***i***) a **condenser** for summarizing intermediate generation states into compact visual memory, (***ii***) a **translator** for converting latent thoughts into actionable guidance, (***iii***) a **shaper** for dynamically steering next image token predictions, and (***iv***) an RL-trained **invoker** for adaptively determining when to invoke reasoning. By performing reasoning entirely in continuous latent spaces, LatentMorph avoids the bottlenecks of explicit reasoning and enables more adaptive self-refinement. Extensive experiments demonstrate that LatentMorph **(I)** enhances the base model Janus-Pro by 16% on GenEval and 25% on T2I-CompBench; **(II)** outperforms explicit paradigms (*e.g.*, TwiG) by 15% and 11% on abstract reasoning tasks like WISE and IPV-Txt, **(III)** while reducing inference time by 44% and token consumption by 51%; and **(IV)** exhibits 71% cognitive alignment with human intuition on reasoning invocation.

Deep Learning · Everything Else

Antonios Tragoudaras, Daniil Cherniavskii, Chenyu Zhang, Antonis Vozikis, Thijmen Nijdam, Derck Prinzhorn, Márk Bodrácska, Nicu Sebe, Andrii Zadaianchuk, Efstratios Gavves

Recent advances in image and video generation raise hopes that these models possess world modeling capabilities—the ability to generate realistic, physically plausible videos. This could revolutionize applications in robotics, autonomous driving, and scientific simulation. However, before treating these models as world models, we must ask: Do they adhere to physical laws? Current evaluation methods rely on subjective judgments or trajectory matching, limiting their usage for physical reasoning estimation, where many generations could be physically plausible. Thus, we introduce **Morpheus**, one of the first physics-informed evaluation frameworks for measuring the ability of video generation models to comprehend Newtonian dynamics. **Morpheus** features 130 real-world videos capturing physical phenomena, guided by conservation laws. Using those as conditioning for video generation, we assess physical plausibility leveraging interpretable metrics evaluated with respect to infallible conservation laws known per physical setting, leveraging advances in physics-informed neural networks and vision-language foundation models. Importantly, **Morpheus** targets controlled Newtonian rigid-body settings to enable quantitative checks. Our findings reveal that even with advanced prompting and video conditioning, contemporary models struggle to encode physical principles despite generating aesthetically pleasing videos.

Deep Learning · Large Language Models

Yuntong Hu, Matthew Trager, Yuting Zhang, Yi Zhang, Shuo Yang, Wei Xia, Stefano Soatto

Large language model (LLM)–based multi-agent systems (MAS) show strong promise for complex reasoning, planning, and tool-augmented tasks, but designing effective MAS architectures remains labor-intensive, brittle, and hard to generalize. Existing automatic MAS generation methods either rely on code generation, which often leads to executability and robustness failures, or impose rigid architectural templates that limit expressiveness and adaptability. We propose Evolutionary Generation of Multi-Agent Systems (EvoMAS), which formulates MAS generation as structured configuration generation. EvoMAS performs evolutionary generation in configuration space. Specifically, EvoMAS selects initial configurations from a pool, applies feedbackconditioned mutation and crossover guided by execution traces, and iteratively refines both the candidate pool and an experience memory. We evaluate EvoMAS on diverse benchmarks, including BBEH, SWE-Bench, and WorkBench, covering reasoning, software engineering, and tool-use tasks. EvoMAS consistently improves task performance over both human-designed MAS and prior automatic MAS generation methods, while producing generated systems with higher executability and runtime robustness. EvoMAS outperforms the agent evolution method EvoAgent by +10.5 points on BBEH reasoning and +7.1 points on WorkBench. With Claude-4.5-Sonnet, EvoMAS also reaches 79.1% on SWE-BenchVerified, matching the top of the leaderboard.

Deep Learning · Large Language Models

Venmugil Elango, Nidhi Bhatia, Roger Waleffe, Rasoul Shafipour, Tomer Asida, Abhinav Khattar, Nave Assaf, Maximilian Golub, Joseph Guman, Tiyasa Mitra 等

Mixture-of-Experts (MoEs) have become a central component of many state-of-the-art open-source and proprietary large language models. Despite their widespread adoption, it remains unclear how close existing MoE architectures are to optimal with respect to inference cost, as measured by accuracy per floating-point operation and per parameter. In this work, we revisit MoE design from a hardware-software co-design perspective, grounded in empirical and theoretical considerations. We characterize key performance bottlenecks across diverse deployment regimes, spanning offline high-throughput execution and online, latency-critical inference. Guided by these insights, we introduce LatentMoE, a new model architecture resulting from systematic design exploration and optimized for maximized accuracy per unit of compute. Empirical design space exploration at scales of up to 95B parameters and over a 1T-token training horizon, together with supporting theoretical analysis, show that LatentMoE consistently outperforms standard MoE architectures in terms of accuracy per FLOP and per parameter.

Deep Learning · Attention Mechanisms

Jie Hu, Zixiang Gao, Yutong He, Kun Yuan

Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve–based token reordering to achieve fine-grained sparsity while retaining hardware-friendly operations, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1$\times$ end-to-end speedup while maintaining high generation quality.

General Machine Learning · Supervised Learning

Zihan Yu, Guanren Wang, Jingtao Ding, Huandong Wang, Yong Li

Symbolic regression (SR) traditionally balances accuracy and complexity, implicitly assuming that simpler formulas are structurally more rational. We argue that this assumption is insufficient: existing algorithms often exploit this metric to discover accurate and compact but structurally irrational formulas that are numerically ill-conditioned and physically inexplicable. Inspired by the structural stability of real physical laws, we propose the Effective Information Criterion (EIC) to quantify formula rationality. EIC models formulas as information channels and measures the amplification of inherent rounding noise during recursive calculation, effectively distinguishing physically plausible structures from pathological ones without relying on ground truth. Our analysis reveals a stark structural stability gap between human-derived equations and SR-discovered results. By integrating EIC into SR workflows, we provide explicit structural guidance: for heuristic search, EIC steers algorithms toward stable regions to yield superior Pareto frontiers; for generative models, EIC-based filtering improves pre-training sample efficiency by 2–4 times and boosts generalization $R^2$ by 22.4\%. Finally, an extensive study with 108 human experts shows that EIC aligns with human preferences in 70\% of cases, validating structural stability as a critical prerequisite for human-perceived interpretability. We release our code at https://anonymous.4open.science/r/EIC-91B2.

Deep Learning · Large Language Models

Jingchu Gai, Guanning Zeng, Huaqing ZHANG, Aditi Raghunathan

It is widely recognized that reinforcement learning (RL) fine-tuning of large language models often leads to \textit{diversity collapse}, where outputs lack variety. Prior work has proposed a range of heuristics to counteract this effect, but these methods are ad hoc: they frequently trade off correctness for diversity, their effectiveness varies across tasks, and in some cases they even contradict one another. In this work, we place these observations on a rigorous foundation. We first provide a formal proof of why RL fine-tuning exhibits diversity collapse via a selection and reinforcement bias. Next, we make a key observation that any reward modification to address diversity collapse only needs to be applied on the correct trajectories. Building directly on this analysis, we introduce a principled method---\textit{differential smoothing}---that provably improves both correctness and diversity, outperforming vanilla RL as well as widely used entropy-based heuristics. Our theory precisely characterizes when existing heuristics help and why they fail, while showing that differential smoothing is universally superior. Extensive experiments with models from 1B to 7B parameters, across domains including CountDown and real-world mathematical reasoning, demonstrate consistent gains. Differential smoothing improves both Pass@1 and Pass@k, with up to 6.7\% improvements on AIME24 dataset.

Applications · Time Series

Wanjin Feng, Yuan Yuan, Jingtao Ding, Yong Li

In the era of increasingly complex AI models for time series forecasting, progress is often measured by marginal improvements on benchmark leaderboards. However, standard evaluations rely on aggregate metrics (e.g., MSE) that conflate model capability with the intrinsic difficulty of the evaluated instances. To address this, we propose a diagnostic framework anchored in **Spectral Coherence Predictability (SCP)**, which provides an efficient $\mathcal{O}(N\log N)$ per-instance difficulty reference and yields a corresponding linear MSE lower bound. Complementing this, we introduce the **Linear Utilization Ratio (LUR)** to quantify how effectively models exploit linearly predictable structures across frequencies. Experiments on synthetic and real-world benchmarks show that SCP aligns strongly with realized forecasting errors across diverse state-of-the-art forecasters. Using this lens, we uncover ``predictability drift,'' revealing that task difficulty is not static but fluctuates significantly over time and variables. Furthermore, stratified evaluation exposes complementary architectural strengths across distinct frequency bands and difficulty regimes. Overall, we advocate moving beyond leaderboard-style ranking toward a more insightful, predictability-aware evaluation that fosters fairer model comparisons and a deeper understanding of model behavior. Code and data are available at https://anonymous.4open.science/r/TS_Predictability-C8B7.

Applications · Chemistry, Physics, and Earth Sciences

Ruikun Li, Huandong Wang, Jingtao Ding, Yuan Yuan, Qingmin Liao, Yong Li

Data-driven dynamics prediction often fails under environmental shifts, while traditional fine-tuning remains computationally prohibitive for hardware-constrained or data-scarce applications. We propose DynaDiff, a generative meta-learning framework that transitions the paradigm from gradient-based tuning or modulation to direct weight-space generation. Specifically, we first abstract expert weights as novel weight graphs, utilizing multi-head attention to explicitly capture topological coupling within weights. Subsequently, we design a functional loss to ensure that the generated models achieve consistency with expert models in physical behavior. Finally, we develop a dynamics-informed prompter that extracts cross-domain physical and spectral features from observation sequences to condition the diffusion model. Experiments demonstrate that DynaDiff boosts average prediction accuracy by 10.78\% over competitive baselines. Furthermore, by pre-constructing a model zoo of expert predictors, we amortize the fine-tuning overhead into a one-time offline cost, significantly boosting deployment efficiency in new environments.

Deep Learning · Foundation Models

Chang Liu, Bohao Zhao, Jingtao Ding, Yong Li

Foundation models have shown great promise in achieving zero-shot or few-shot forecasting for ODE-based chaotic systems via large-scale pretraining. However, existing architectures often fail to capture the multi-scale temporal structures and distinct spectral characteristics of chaotic dynamics. To address this, we introduce ChaosNexus, a foundation model for chaotic system forecasting underpinned by the proposed ScaleFormer architecture. By processing temporal contexts across hierarchically varying patch sizes, ChaosNexus effectively captures long-range dependencies and preserves high-frequency fluctuations. To address heterogeneity across distinct systems, we integrate Mixture-of-Experts (MoE) layers into each ScaleFormer block and explicitly condition the final forecasts on a learned frequency fingerprint, providing the model with a global spectral view of the system. Extensive evaluations on over 9,000 synthetic systems demonstrate that ChaosNexus achieves superior fidelity in long-term attractor statistics while maintaining competitive point-wise accuracy. Furthermore, in real-world applications, it achieves a remarkable zero-shot mean error below 1°C for 5-day station-based weather forecasting. Codes are available at https://anonymous.4open.science/r/ChaosNexus-C809.

Deep Learning · Robustness

Eitan Wagner, Omri Abend

Language modeling has shifted in recent years from a distribution over strings to prediction models with textual inputs and outputs for general-purpose tasks. This position paper highlights the often overlooked implications of this shift for the use of large language models (LLMs) as probability estimators, especially for world probabilities. In light of the theoretical distinction between distribution estimation and response prediction, we examine LLM training phases and common use cases for LLM output probabilities. We show that the different settings lead to distinct, potentially conflicting, desired output distributions. This lack of clarity leads to pitfalls when using output probabilities as event probabilities. Our position is that second-order prediction—incorporating probabilities as part of the output—is the only theoretically sound method. We conclude with suggestions for potential directions to improve the probabilistic soundness of this method.

Yuan Gao, Fan Xu, Hao Wu, Yuxu Lu, Penghao Zhao, Fan Zhang, Hao Jia, Yuxuan Liang, Ruijian Gou, Qingsong Wen 等

Coupled spatiotemporal forecasting is important for predicting the future evolution of multiple interacting dynamical systems, such as in climate models. However, existing methods are severely constrained by the persistent bottleneck of compounding errors. In coupled systems, errors from each subsystem simulator propagate and amplify one another, a phenomenon we term Reciprocal Error Amplification leading to a rapid collapse of long-range predictions. To address this challenge, we propose a universal framework called PnP-Corrector (Plug-and-Play Corrector). The core idea of our framework is to decouple the physical simulation from the error correction process: it freezes pre-trained physics simulation engines and exclusively trains a lightweight correction agent to proactively counteract the systematic biases emerging from the coupled system. Furthermore, we design an efficient predictive model architecture, DSLCast, to serve as the backbone of this framework. Extensive experiments demonstrate that our method significantly enhances the long-term stability and accuracy of coupled forecasting systems. For instance, in the challenging task of a 300-day global ocean-atmosphere coupled forecast, our PnP-Corrector framework reduces the prediction error of the baseline model by 28% and surpasses state-of-the-art models on several key metrics. Codes link: \url{https://anonymous.4open.science/r/Anonymous_PnP-E365}.

Deep Learning · Large Language Models

Qianyue Wang, Jinwu Hu, Zhiquan Wen, Bailin Chen, Huanxiang Lin, Yufeng Wang, Yaofo Chen, Yu Rong, Yuanqing Li, Mingkui Tan

Large Reasoning Models (LRMs) excel at complex reasoning but suffer from inefficient reasoning, like overthinking and overshoot. These issues stem from excessive or misdirected reasoning triggered by the model's "doubt", manifested as self-validation and exploratory extension, increasing computational cost and degrading performance. Existing efficient reasoning methods seek to regulate reasoning via internal signals or static schedules, lacking specialization in the "doubt" characteristics of LRMs. To address this, we propose a Conjunction-Guided Intervention (CGI) reasoning framework that intervenes when the model shows signs of doubt. Our key insight is that overthinking and overshoot in LRMs arise from conjunction-triggered extensions where LRMs exhibit "doubt" through transitional conjunctions, extending redundant self-validation and exploration without timely state-based correction. Building on this insight, CGI pauses reasoning at conjunction markers of doubt for external state-based feedback, adaptively extending or terminating reasoning to reduce redundancy while preserving accuracy. The feedback is generated via criteria evaluation (rationality and completeness) and comes from either human or LLM proxies. We train the target model by Group Relative Policy Optimization (GRPO) to adapt to the interactive mode. Experiments show that our framework achieves a superior balance between accuracy and reasoning length.

Applications · Language, Speech and Dialog

Eitan Wagner, Elisha Rosensweig, Omri Abend

Recent advances in large language models and their widespread adoption have prompted claims that natural language could entirely replace formal languages, such as programming languages, for software design. In this position paper, we argue that this perspective overlooks fundamental linguistic properties of natural language, specifically that it is optimized for underspecification in open-ended contexts. We introduce a formal framework centered on *task specificity*, defining it as the information-theoretic reduction of uncertainty—in an output space, such as all possible images—given a user's specific requirements. We prove a *specificity crossover theorem*, showing the existence of a threshold beyond which the cost to express formal requirements into natural language exceeds the cost of direct formal specification. By analyzing case studies across modalities, such as image generation, code synthesis, and audio production, we demonstrate that natural language excels at low specificity tasks, while formal languages are advantageous on tasks with stricter requirements. We conclude that natural and formal languages are complementary tools and advocate the development of hybrid systems that allow users to move across the specificity spectrum.

Hongwei Ren, Fei Ma, Xiaopeng LIN, Yuetong Fang, Hongxiang Huang, Yue Zhou, Yulong Huang, Haotian FU, Ziyi Yang, Youxin Jiang 等

Event cameras are biologically inspired sensors garnering significant attention from both industry and academia. Mainstream methods favor frame and voxel representations, which reach a satisfactory performance while introducing time-consuming transformations, bulky models, and sacrificing fine-grained temporal information. Alternatively, Point Cloud representation demonstrates promise in addressing the mentioned weaknesses, but it has limited scalability in abstracting features of higher spatial resolution and longer temporal sequence events. In this paper, we propose a \textbf{S}calable \textbf{N}etwork named SECNet to leverage \textbf{E}vent \textbf{C}loud representation. SECNet integrates polarity at the structural level by innovating the Event-based Group and Sampling module rather than only at the input level. To accommodate the surge in the number of events, SECNet embraces feature extraction in the frequency domain via the Fourier transform. This approach not only substantially extinguishes the explosion of Multiply Accumulate Operations but also effectively abstracts spatio-temporal features. We conducted extensive experiments on \textbf{ten} event-based datasets, and substantiate the scalability, effectiveness, and efficiency of SECNet.