论文检索

Applications · Chemistry, Physics, and Earth Sciences

Alicja Maksymiuk, Alexandre Duplessis, Ismail Ceylan, Alexander Tong, Fernanda Duarte, Michael Bronstein

Macrocycles are ring-shaped molecules that offer a promising alternative to small-molecule drugs due to their enhanced selectivity and binding affinity against difficult targets. Despite their chemical value, they remain underexplored in generative modeling, likely owing to their scarcity in public datasets and the challenges of enforcing topological constraints in standard deep generative models. We introduce MacroGuide: Topological Guidance for Macrocycle Generation, a diffusion guidance mechanism that uses Persistent Homology to steer the sampling of pretrained molecular generative models toward the generation of macrocycles, in both unconditional and conditional (protein pocket) settings. At each denoising step, MacroGuide constructs a Vietoris-Rips complex from atomic positions and promotes ring formation by optimizing persistent homology features. Empirically, applying MacroGuide to pretrained diffusion models increases macrocycle generation rates from 1% to 99%, while matching or exceeding state-of-the-art performance on key quality metrics such as chemical validity, diversity, and PoseBusters checks.

Theory · Everything Else

Qiming Cui, Michael Dinitz

We revisit the central online problem of *ski rental* in the ``algorithms with predictions'' framework from the point of view of *distributional* predictions. Ski rental was one of the first problems to be studied with predictions, where a natural prediction is simply the number of ski days. But it is both more natural and potentially more powerful to think of a prediction as a *distribution* $\hat p$ over the ski days. If the true number of ski days is drawn from some true (but unknown) distribution $p$, then we show as our main result that there is an algorithm with expected cost at most $OPT + O\left(\min \left(\max(\eta,1) \cdot \sqrt{b},\ b \log b \right) \right)$, where $OPT$ is the expected cost of the optimal policy for the true distribution $p$, $b$ is the cost of buying, and $\eta$ is the Earth Mover's (Wasserstein-1) distance between $p$ and $\hat p$. Note that when $\eta < o(\sqrt{b})$ this gives additive loss less than $b$ (the trivial bound), and when $\eta$ is arbitrarily large (corresponding to an extremely inaccurate prediction) we still do not pay more than $O(b \log b)$ additive loss. An implication of these bounds is that our algorithm has *consistency* $O(\sqrt{b})$ (additive loss when the prediction error is $0$) and *robustness* $O(b \log b)$ (additive loss when the prediction error is arbitrarily large). Moreover, we do not need to assume that we know (or have any bound on) the prediction error $\eta$, in contrast with previous work in robust optimization which assumes that we know this error. We complement this upper bound with a variety of lower bounds showing that it is essentially tight: not only can the consistency/robustness tradeoff not be improved, but our particular loss function cannot be meaningfully improved.

Deep Learning · Large Language Models

Xin Wang, Hui Shen, Boyuan Zheng, Xueshen Liu, Minkyoung Cho, Zhongwei Wan, Zesen Zhao, Zhuoqing Morley Mao, Mi Zhang, Shen Yan

The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost. To improve representation capacity under long contexts, recent approaches organize memory in a multi-state manner. However, existing multi-state linear attention methods rely on fixed state merging policies that cannot adapt to dynamically varying token importance, irreversibly obscuring critical tokens and causing severe error accumulation over long sequences. To address this limitation, we propose DLA, a dynamic memory modeling framework for multi-state linear attention. DLA introduces (i) Information-Aware Dynamic State Merging, which adaptively determines state boundaries based on token-level information variation, preserving high-resolution representations around semantic transitions while aggressively summarizing stable regions, and (ii) Capacity-Bounded Memory Modeling, which maintains a fixed-size, chronologically ordered state cache by selectively merging adjacent low-information states to control memory growth with minimal information loss. We pre-train DLA on two different linear attention models and evaluate on 10 datasets from three different aspects. Experimental results demonstrate the superiority of DLA over state-of-the-art.

Deep Learning · Large Language Models

Yiran Wu, Mauricio Velazco, Andrew Zhao, Manuel Luján, Srisuma Movva, Yogesh Roy, Quang Nguyen, Roberto Rodriguez, Qingyun Wu, Michael Albada 等

We present \textbf{ExCyTIn-Bench}, the first benchmark to \textbf{E}valuate an LLM agent \textbf{X} on the task of \textbf{Cy}ber \textbf{T}hreat \textbf{In}vestigation through security questions derived from investigation graphs. Real‑world security analysts must sift through a large number of heterogeneous security logs, follow multi‑hop chains of evidence to investigate threats. With the developments of LLMs, building LLM-based agents for automatic threat investigation is a promising direction. We construct a benchmark from a controlled Azure tenant including a SQL environment covering 57 log tables from Microsoft Sentinel and related services, and 7542 generated questions. We leverage security logs extracted with expert-crafted detection logic to build threat investigation graphs, and then generate questions with LLMs using paired nodes on the graph, taking the start node as background context and the end node as answer. Anchoring each question to these explicit nodes and edges not only provides automatic, explainable ground truth answers but also makes the pipeline reusable and readily extensible to new logs. Our comprehensive experiments on the test set with different models confirm the difficulty of the task: the best model so far can achieve a reward of 0.606, leaving much headroom for future research.

Social Aspects · Accountability, Transparency, and Interpretability

David Rundel, Fabian Fumagalli, Maximilian Muschalik, Bernd Bischl, Matthias Feurer

Shapley values are a principled attribution measure widely used in interpretable machine learning, but their exact computation scales exponentially with the number of players, motivating a wide range of approximation methods based on value-function evaluations of sampled coalitions. This raises the question of whether approximation accuracy can be improved by *adaptively* selecting coalitions for evaluation based on previous outcomes. This is particularly relevant in settings where the value function is costly, and the number of evaluations is severely limited, such as retraining-based feature importance, data valuation, and hyperparameter importance. For this purpose, we propose $\texttt{ShaplEIG}$, a Bayesian experimental design approach that approximates the expensive value function via a Gaussian process surrogate and adaptively selects coalitions based on their expected information gain about the Shapley values. Since Shapley values are a linear function of the value function, we show that the expected information gain is available in *closed form* and *efficiently* computable. In extensive experiments across diverse costly applications, our method consistently improves estimation accuracy over state-of-the-art baselines.

Deep Learning · Large Language Models

Qingyao Ai, Yichen Tang, Changyue Wang, Jianming Long, Weihang Su, Yiqun LIU

Scaling up data, parameters, and test-time computation has been the mainstream methods to improve LLM systems (LLMsys), but their upper bounds are almost reached due to the gradual depletion of high-quality data and marginal gains obtained from larger computational resource consumption. Inspired by the abilities of human and traditional AI systems in learning from practice, constructing memory and continual learning frameworks for LLMsys has become an important and popular research direction in recent literature. Yet, existing benchmarks for LLM memory often focus on evaluating the system on homogeneous reading comprehension tasks with long-form inputs rather than testing their abilities to learn from accumulated user feedback in service time. Therefore, we propose a user feedback simulation framework and a comprehensive benchmark covering multiple domains, languages, and types of tasks to evaluate the continual learning abilities of LLMsys. Experiments show that the effectiveness and efficiency of state-of-the-art baselines are far from satisfying, and we hope this benchmark could pave the way for future studies on LLM memory and optimization algorithms.

Applications · Neuroscience, Cognitive Science

Xinyu Shi, Tong Bu, Zhaofei Yu

Spiking Neural Networks (SNNs) offer a notable energy-saving advantage compared to Artificial Neural Networks (ANNs) when deployed on neuromorphic hardware. While recent SNNs achieve superior performance using larger and deeper backbones, this comes at a cost of diminishing their energy-saving benefits. In this paper, we propose UniSparse, a unified sparsification framework for enhancing the energy efficiency of SNNs. We demonstrate that the affine parameters in batch normalization also serve as the learnable threshold of its subsequent spiking neurons. Based on this, we propose a novel spike sparsification method that reduces firing rate by constraining the affine parameters. As a complement to spike sparsification, we propose a weight pruning method based on the same energy constraint, which can be naturally integrated with spike sparsification. Experimental results demonstrate that UniSparse achieves a state-of-the-art trade-off between accuracy and energy efficiency across models and datasets. The sparsified ResNet-18 model requires only 7.04M SOPs for inference to achieve 92.38\% accuracy on the CIFAR-10 dataset. Our work highlights the great potential of deep SNNs in improving energy efficiency.

Applications · Neuroscience, Cognitive Science

Xinyu Shi, Zhaofei Yu

The training of deep Spiking Neural Networks (SNNs) has traditionally relied on Batch Normalization (BN), which stabilizes input currents and gradients during training. However, BN is not a universal solution. It is unsuitable for variable-length tasks and scenarios with reduced batch size, constraining the development of deep SNNs, where removing BN typically causes the training to fail to converge. This dependence stems not from a fundamental necessity of BN but from the current lack of reasonable initialization methods for SNNs. This paper addresses this core limitation by proposing SpikeInit, a novel initialization framework for SNNs. By modeling the response curve and gradient of spiking layers, SpikeInit initializes the weights and shape parameters of surrogate gradients to maintain stable firing rates during forward propagation and stable gradient magnitudes during backpropagation. Extensive experiments demonstrate that deep SNNs with SpikeInit can be trained stably without normalization and achieve superior performance compared to their normalized counterparts under identical settings. Furthermore, we demonstrate the scalability of SpikeInit by successfully training an ultra-deep, 1000-layer SNN without normalization. Our work provides a foundational step toward large-scale normalization-free SNN, liberating SNN design from the constraints of normalization.

Arnas Uselis, Andrea Dittadi, Seong Joon Oh

Compositional generalization, the ability to recognize familiar parts in novel contexts, is a defining property of intelligent systems. Modern models are trained on massive datasets, yet these are vanishingly small compared to the full combinatorial space of possible data, raising the question of whether models can reliably generalize to unseen combinations. To formalize what this requires, we propose a set of practically motivated desiderata that any compositionally generalizing system must satisfy, and analyze their implications under standard training with linear classification heads. We show that these desiderata necessitate \emph{linear factorization}, where representations decompose additively into per-concept components, and further imply near-orthogonality across factors. We establish dimension bounds that link the number of concepts to the geometry of representations. Empirically, we survey CLIP and SigLIP families, finding strong evidence for linear factorization, approximate orthogonality, and a tight correlation between the quality of factorization and compositional generalization. Together, our results identify the structural conditions that embeddings must satisfy for compositional generalization, and provide both theoretical clarity and empirical diagnostics for developing foundation models that generalize compositionally.

Applications · Chemistry, Physics, and Earth Sciences

Mica Consens, Kevin Yang, James Hall, Ashley Conard, BO WANG, Lorin Crawford, Alan Moses, Alex Lu

Genome language models (gLM) have the potential to further understanding of regulatory genomics without requiring labeled data. Most gLMs are pretrained using sequence reconstruction tasks inspired by natural language processing, but recent studies have shown that these gLMs often fail to capture biological signal. To overcome this, we introduce pretraining tasks that predict the rate of evolution. These tasks are designed so that they can be composed with sequence reconstruction, enabling a controlled comparison of predicting sequence only, evolutionary rate only, or both. To address gaps in existing evaluations, we developed a suite of biologically grounded benchmarks. Across these tasks, and for established variant effect prediction benchmarks, models pretrained on both sequence and evolutionary rate outperform those trained on sequence alone, and training on evolutionary rate can make the even the relatively small models in our work competitive with much larger existing gLMs for some tasks. These results establish evolution as a key training target for genome-scale models.

Deep Learning · Large Language Models

Varun Ursekar, Apaar Shanker, Veronica Chatrath, Yuan Xue, Samuel Denton

An important emerging application of coding agents is *agent optimization*: the iterative improvement of a *target agent* through edit–execute–evaluate cycles. Despite its relevance, the community lacks a systematic understanding of coding agent performance on this task. Agent optimization differs fundamentally from conventional software engineering: the target agent interleaves deterministic code with stochastic LLM completions, requiring structured capture of both intermediate reasoning and downstream execution outcomes. To address these challenges, we introduce VeRO (**Ve**rsioning, **R**ewards, and **O**bservations), which provides (1) a reproducible evaluation harness with versioned agent snapshots, budget-controlled evaluation, and structured execution traces, and (2) a benchmark suite of target agents and tasks with reference evaluation procedures. Using VeRO, we conduct an empirical study comparing optimizer configurations across tasks and analyzing which modifications reliably improve target agent performance. We release VeRO to support research on agent optimization as a core capability for coding agents.

Deep Learning · Large Language Models

Yuran Sun, Chuan Wu

Prompt optimization is critical for maximizing the performance of large language models (LLMs). However, it often relies on costly labeled data. Self-supervised methods reduce data dependency, but they suffer from optimization ambiguity or high computational costs. To address these limitations, we propose the Meta-Reasoning Prompt Engineering Agent (MR.PEA), a self-supervised prompt optimization framework that operates with minimal input. MR.PEA leverages meta-reasoning to iteratively build task-specific knowledge, including problem-solving strategies and evaluation criteria, while adaptively retrieving external information to enhance its understanding. This knowledge guides the generation of diverse validation examples, targeted prompt refinement, and comprehensive quality assessments. Experiments on GSM8K and Big-Bench Hard show that MR.PEA outperforms existing baselines, achieving an average performance gain of 7.4% with an optimization cost as low as $0.01 per task.

Theory · Probabilistic Methods

Youheng Zhu, Yiping Lu

Inference-time scaling has recently emerged as a powerful paradigm for improving the reasoning capability of large language models. Among various approaches, \emph{Sequential Monte Carlo (SMC)} has become a particularly important framework, enabling iterative generation, evaluation, rejection, and resampling of intermediate reasoning trajectories. A central component in this process is the \emph{reward model}, which evaluates partial solutions and guides the allocation of computation during inference. However, in practice, true reward models are never available. All deployed systems rely on \emph{approximate reward models}, raising a fundamental question: \emph{Why and when do approximate reward models suffice for effective inference-time scaling?} In this work, we provide a theoretical answer. We identify the \emph{Bellman error} of the approximate reward model as the key quantity governing the effectiveness of SMC-based inference-time scaling. For a reasoning process of length $T$, we show that if the Bellman error of the approximate reward model is bounded by $O(1/T)$, then combining this reward model with SMC reduces the computational complexity of reasoning from exponential in $T$ to polynomial in $T$. This yields an \emph{exponential improvement} in inference efficiency despite using only approximate rewards.

Deep Learning · Large Language Models

Siwei Wu, Yizhi Li, Yuyang Song, Wei Zhang, Yang Wang, Riza Batista-Navarro, Xian Yang, Mingjie Tang, Bryan Dai, Jian Yang 等

Training agentic models for terminal-based tasks critically depends on high-quality terminal trajectories that capture realistic long-horizon interactions across diverse domains. However, constructing such data at scale remains challenging due to two key requirements: \textbf{\emph{Executability}}, since each instance requires a suitable and often distinct Docker environment; and \textbf{\emph{Verifiability}}, because heterogeneous task outputs preclude unified, standardized verification. To address these challenges, we propose \textbf{TerminalTraj}, a scalable pipeline that (i) filters high-quality repositories to construct Dockerized execution environments, (ii) generates Docker-aligned task instances, and (iii) synthesizes agent trajectories with executable validation code. Using TerminalTraj, we curate 32K Docker images and generate 50,733 verified terminal trajectories across eight domains. Models trained on this data with the Qwen2.5-Coder backbone achieve consistent performance improvements on TerminalBench (TB), with gains of up to 20\% on TB 1.0 and 10\% on TB 2.0 over their respective backbones. Notably, \textbf{TerminalTraj-32B} achieves strong performance among models with fewer than 100B parameters, reaching 35.30\% on TB 1.0 and 22.00\% on TB 2.0, and demonstrates improved test-time scaling behavior.

Social Aspects · Accountability, Transparency, and Interpretability

Joseph Paillard, Angel REYERO LOBO, Denis-Alexander Engemann, Thirion Bertrand

Feature-importance methods show promise for transforming machine learning (ML) models from predictive engines into tools for scientific discovery. However, expressive models can be unstable due to data sampling and algorithmic stochasticity, leading to inaccurate variable importance estimates, undermining their utility in critical biomedical applications. While ensembling offers a remedy, the choice between explaining a single ensemble model or aggregating individual model explanations is non-trivial due to the non-linearity of importance measures, and remains largely understudied. Our theoretical analysis, developed under assumptions accommodating complex state-of-the-art ML models, reveals that this choice is governed by a trade-off involving the model's excess risk. In contrast to prior literature, we show that ensembling at the model level provides more accurate variable-importance estimates, particularly for expressive models, by reducing this leading error term. We validate these findings on classical benchmarks and a large-scale proteomic study from the UK Biobank.

Theory · Optimization

Alex Saad-Falcon, Brighton Ancelin, Justin Romberg

We analyze a compressed variant of Oja's algorithm for estimating the principal eigenvector of the data covariance matrix using only two adaptive measurements per sample. At each iteration, we observe one measurement along the current estimate and one in a random orthogonal direction. We prove that after $t$ iterations, the expected sine-squared error to the true eigenvector is $\mathcal{O}(\lambda_1\lambda_2 d^2 / (\Delta^2 t))$, where $d$ is the ambient dimension, $\lambda_1, \lambda_2$ are the leading eigenvalues, and $\Delta = \lambda_1 - \lambda_2$ is the eigengap. We complement this with a matching information-theoretic lower bound of $\Omega(\lambda_1\lambda_2 d^2 / (\Delta^2 t))$ --- the first for compressed eigenvector estimation --- proving that the $d^2$ factor, an additional factor of $d$ compared to full-observation PCA, is the fundamental cost of compression and cannot be improved. Our analysis handles the noisy setting where the covariance has nonzero trailing eigenvalues, providing the first convergence guarantee for adaptive compressed subspace tracking beyond the noiseless case.

Theory · Optimization

Jinyi Huang, Jinlong Lei, Guodong Shi

Distributionally Robust Optimization (DRO) is widely used to improve model robustness, with existing methods addressing either geometric perturbations (e.g., input shifts) or statistical contamination (e.g., heavy-tailed noise and outliers) effectively. However, these uncertainty sources often co-exist. Coupling them through a single divergence or optimal transport constraint conflates geometric displacement with loss-based outlierness, which often leads to the discarding of informative high-leverage samples. We introduce nested DRO, a bilevel formulation that combines optimal transport with an outer $\\phi$-divergence constraint to decouple geometric smoothing from statistical robustness. We prove that this structure naturally induces a geometry-invariant, loss-based reweighting mechanism that separates outlier suppression from transport-induced regularization. We derive a tractable strong dual for the resulting non-convex problem and show its equivalence to variance-regularized risk minimization, providing a rigorous theoretical justification for reweighting gains as a natural consequence of dualization. Empirical results on synthetic and real datasets demonstrate that nested DRO consistently outperforms geometry-coupled DRO baselines, particularly under heavy-tailed contamination where preserving high-leverage structure is crucial.

Social Aspects · Alignment

Advait Yadav, Sidney Black, Oliver Sourbut

Large language model (LLM) agents increasingly coordinate in multi-agent systems, yet we lack an understanding of where and why cooperation failures may arise. In many real-world coordination problems, from knowledge sharing in organizations to code documentation, helping others carries negligible personal cost while generating substantial collective benefits. However, whether LLM agents cooperate when helping neither benefits nor harms the helper, despite being given explicit instructions to do so, remains unknown. We build a multi-agent setup designed to study cooperative behavior in a frictionless environment, removing all strategic complexity from cooperation. We find that capability does not predict cooperation: OpenAI o3 achieves only 17\% of optimal collective performance while OpenAI o3-mini reaches 50\%, despite identical instructions to maximize group revenue. Through a causal decomposition that automates one side of agent communication, we separate cooperation failures from competence failures, tracing their origins through agent reasoning analysis. Testing targeted interventions, we find that explicit protocols double performance for low-competence models, and tiny sharing incentives improve models with weak cooperation. These results demonstrate that even when helping is free and strategically trivial, many LLMs fail to follow the instructed cooperative objectives, requiring interventions based on specific failure modes. Our findings suggest that scaling intelligence alone will not solve coordination problems in multi-agent systems and will require deliberate cooperative design, even when helping others costs nothing.

Deep Learning · Other Representation Learning

Deyu Bo, Xinchao Wang

Multi-modal dataset distillation (MDD) seeks to compress large-scale multi-modal datasets into a compact set of synthetic pairs. Existing methods employ a dual-trajectory matching framework to align the teacher and student models within each modality. While effective, this paradigm incurs non-negligible memory and computational overhead due to the checkpoint storage and bi-level optimization over synthetic data. To address these limitations, we propose analytic parameter matching (APM), which theoretically derives the analytic parameters of modal projectors to replace the inner-loop optimization, and then aligns the analytic projector parameters of teacher and student models. APM offers two key advantages: (1) it replaces checkpoint-intensive storage with only two cached matrices, significantly reducing memory consumption; and (2) it computes analytic parameters in a single forward pass, thereby avoiding costly bi-level optimization. Empirically, APM achieves up to 65$\times$ storage reduction and 9.6$\times$ faster distillation, while scaling to 1,000 synthetic pairs. Extensive experiments on image-text and audio-text benchmarks demonstrate the effectiveness of APM in cross-modal retrieval tasks, \eg, 12.8 IR@1 and 17.8 TR@1 in Flickr30k with 100 synthetic pairs. Moreover, APM exhibits notable generalization performance in cross-architecture evaluation and zero-shot classification tasks.

General Machine Learning · Representation Learning

Yuanyuan Wang, Wenjie Wang, Kun Zhang, Mingming Gong

Bridging the gap between visual realism and physical understanding is a core challenge for video-based world models. We study the structural identifiability of continuous-time physical laws from raw pixels, focusing on whether an encoder-only pipeline can uniquely recover the parameters of second-order linear ODEs. We prove that a level-set slope-coverage condition ensures the learned latent space is locally affine to the true physical state, enabling exact parameter recovery. Our theory provides the first characterization of minimal data requirements across damping regimes, establishing that underdamped systems are identifiable from a single video clip, whereas other regimes require three diverse trajectories. We further introduce a variance-floor regularizer to stabilize the decoder-free objective and prevent latent collapse. Validated on synthetic and real-world data, our approach demonstrates that interpretable physical constants can be reliably estimated from video without the need for compute-intensive pixel reconstruction, ensuring both physical correctness and transparency.