论文检索

Deep Learning · Attention Mechanisms

Kyle Lee, Corentin Delacour, Kevin Callahan-Coray, Kyle Jiang, Can Yaras, Samet Oymak, Tathagata Srimani, Kerem Camsari

Autoregressive decoding becomes bandwidth-limited at long contexts, as generating each token requires reading all $n_k$ key and value vectors from KV cache. We present Stochastic Additive No-mulT Attention (SANTA), a method that sparsifies value-cache access by sampling $S \ll n_k$ indices from the post-softmax distribution and aggregates only those value rows. This yields an unbiased estimator of the post-softmax value aggregation while replacing value-stage multiply-accumulates with gather-and-add. We introduce stratified sampling to design variance-reduced, GPU-friendly variants, demonstrating $1.5\times$ decode-step attention kernel speedup over FlashInfer and FlashDecoding on an NVIDIA RTX 6000 Ada while matching baseline accuracy at 32k-token contexts. Finally, we propose Bernoulli $qK^\mathsf{T}$ sampling as a complementary technique to sparsify the score stage, reducing key-feature access through stochastic ternary queries. Both methods are orthogonal to upstream techniques such as ternary quantization, low-rank projections, and KV-cache compression. Together, they point toward sparse, multiplier-free, and energy-efficient inference.

Applications · Neuroscience, Cognitive Science

Hyun-Jong Lee, Jae-Han Lim

Spiking Neural Networks (SNNs) are regarded as the third generation of neural networks, offering energy-efficient computing for neuromorphic devices. Despite this benefit, hardware-implemented SNNs are vulnerable to hardware faults, which severely degrade their performance. Previous approaches have required direct access to internal SNN circuits to modify weights or monitor internal states, limiting their practicality. Improving robustness to hardware faults without such access remains challenging. To overcome this challenge, we propose a fault-tolerant mechanism that operates only through input data control. The hardware faults reduce the usable learning capacity of SNNs, leading to a mismatch between the instantaneous input load and the degraded network dynamics. Our mechanism mitigates this mismatch by dividing each input sample into multiple fragments, redistributing the input load via a learnable fragmentation strategy. The strategy learns two key fragmentation components: 1) division boundaries and 2) the number of fragments. To our knowledge, this is the first to improve the fault tolerance of SNNs without accessing the internal SNN circuit. Experimental results demonstrate that our mechanism consistently outperforms previous methods in various SNN models, achieving these gains without direct access to internal circuits. Furthermore, we validate its effectiveness on SNNs implemented with a physical FPGA platform, confirming its practicality.

Deep Learning · Large Language Models

Wentao Hu, Zeyu Zhu, Mingkuan Zhao, Zhenhua An, Yanbo Zhai, Shanhong yu, Huilin Zhou, Xin Lai, Xiaoyan Zhu, Jiayin Wang

Sparse Mixture-of-Experts (MoE) architectures scale model capacity efficiently but suffer from massive static parameter footprints, creating significant deployment burdens on memory-constrained hardware. Existing post-training pruning methods often rely on scalar statistics, ignoring the representational geometry of expert feature spaces. This leads to sub-optimal resource allocation across layers and the retention of redundant experts. To address this, we propose a Rank-aware Geometric Expert Pruning (RaGEP) framework to compress MoE models by analyzing the geometric properties of expert activations. First, in the inter-layer allocation stage, we introduce a Rank-aware budget allocation mechanism that adaptively assigns expert budgets based on the effective rank of layer-wise representations. Second, in the intra-layer selection stage, we propose a Spectral-Salience Pruning metric that harmonizes subspace orthogonality and activation magnitude to identify high-energy orthogonal experts. Extensive experiments across MoE models of different scales show that our method consistently outperforms state-of-the-art baselines on a diverse set of zero-shot tasks, while reducing model size and inference cost. Code is available at supplementary material.

Applications · Robotics

Tao Sun, Utkarsh Mishra, Jiaxin Lu, Danfei Xu, Iro Armeni

Compositional diffusion planners enable robotic decision-making beyond the horizon of training trajectories. Yet, current approaches often rely on the heuristic stitching of local predictions. We demonstrate that this induces a non-conservative vector field that does not mathematically correspond to any valid global trajectory log-density function. We propose Energy-based Compositional Diffuser (ECD), a framework that formulates the global trajectory as the minimizer of the sum of local bridge potentials. This energy-based perspective guarantees a conservative update field by construction and reveals a critical endpoint reaction term that is missing in heuristic stitching methods. To enable efficient inference, we further introduce a Markov-based score approximation that computes the reaction term though a single block-tridiagonal solve, maintaining time complexity linear in the planning horizon. Empirically, ECD achieves state-of-the-art success rates on a range of OGBench stitching tasks, while nearly matching the inference speed of heuristic stitching methods.

Applications · Neuroscience, Cognitive Science

Sicheng Shen, Mingyang Lv, Bing Han, Dongcheng Zhao, Guobin Shen, Feifei Zhao, Yi Zeng

In recent years, Spiking Neural Networks (SNNs) have achieved remarkable progress, with Spiking Transformers emerging as a promising architecture for energy-efficient sequence modeling. However, existing Spiking Transformers still lack a principled mechanism for effective temporal fusion, limiting their ability to fully exploit spatiotemporal dependencies. Inspired by feedforward–feedback modulation in the human visual pathway, we propose **TEFormer**, the first Spiking Transformer framework that achieves bidirectional temporal fusion by decoupling temporal modeling across its core components. Specifically, TEFormer employs a lightweight and hyperparameter-free **forward temporal fusion mechanism in the attention module**, enabling fully parallel computation, while incorporating a **backward gated recurrent structure in the MLP** to aggregate temporal information in reverse order and reinforce temporal consistency. Extensive experiments across a wide range of benchmarks demonstrate that TEFormer consistently and significantly outperforms strong SNN and Spiking Transformer baselines under diverse datasets. Moreover, through the first systematic evaluation of Spiking Transformers under different neural encoding schemes, we show that the performance gains of TEFormer remain stable across encoding choices, indicating that the improved temporal modeling directly translates into reliable accuracy improvements across varied spiking representations. These results collectively establish TEFormer as an effective and general framework for temporal modeling in Spiking Transformers. Code: [https://anonymous.4open.science/r/TEFormer-6EB7](https://anonymous.4open.science/r/TEFormer-6EB7)

Applications · Energy

Chao Shen, Zihan Guo, Xu Wan, Zhenghao Yang, Yifan Zhang, Wenqi Huang, Jie Song, Zongyan Zhang, Mingyang Sun

Growing renewable penetration introduces substantial uncertainty into power system operations, necessitating frequent adaptation of dispatch objectives and constraints and challenging expertise-intensive, near-real-time modeling workflows. Large Language Models (LLMs) provide a promising avenue for automating this process by translating natural-language (NL) operational requirements into executable optimization models via semantic reasoning and code synthesis. Yet existing LLM datasets and benchmarks for optimization modeling primarily target coarse-grained cross-domain generalization, offering limited, rigorous evaluation in powersystem settings, particularly for Optimal Power Flow (OPF). We therefore introduce ProOPFD and ProOPF-B, a dataset and benchmark for professional-grade OPF modeling: ProOPF-D contains 12K instances pairing NL requests with parameter adjustments and structural extensions to a canonical OPF, together with executable implementations; ProOPF-B provides 121 expertannotated test cases with ground-truth code, enabling end-to-end evaluation under both concrete and abstract OPF modeling regimes.

Applications · Robotics

Yanbiao Ji, Qiuchang Li, Yuting Hu, Shaokai Wu, Wenyuan XIE, Guodong ZHANG, Qichen He, Deyi Ji, Yue Ding, Hongtao Lu

This paper introduces EnergyFlow, a framework that unifies generative action modeling with inverse reinforcement learning by parameterizing a scalar energy function whose gradient is the denoising field. We establish that under maximum-entropy optimality, the score function learned via denoising score matching recovers the gradient of the expert's soft Q-function, enabling reward extraction without adversarial training. Formally, we prove that constraining the learned field to be conservative reduces hypothesis complexity and tightens out-of-distribution generalization bounds. We further characterize the identifiability of recovered rewards and bound how score estimation errors propagate to action preferences. Empirically, EnergyFlow achieves state-of-the-art imitation performance on various manipulation tasks while providing an effective reward signal for downstream reinforcement learning that outperforms both adversarial IRL methods and likelihood-based alternatives. These results show that the structural constraints required for valid reward extraction simultaneously serve as beneficial inductive biases for policy generalization. The code is available at https://anonymous.4open.science/r/EnergyFlow-FAE1.

Applications · Time Series

Linxiao Yang, Xue Jiang, Gezheng Xu, Tian Zhou, Min Yang, Zhaoyang Zhu, Linyuan Geng, Zhipeng Zeng, Qiming Chen, Xinyue Gu 等

Transformers enable in-context learning (ICL) for rapid, gradient-free adaptation in time series forecasting, yet most ICL-style approaches rely on tabularized, hand-crafted features, while end-to-end sequence models lack inference-time adaptation. We bridge this gap with a unified framework, Baguan-TS, which integrates the raw-sequence representation learning with ICL, instantiated by a 3D Transformer that attends jointly over temporal, variable, and context axes. To make this high-capacity model practical, we tackle two key hurdles: (i) calibration and training stability, improved with a feature-agnostic, target-space retrieval-based local calibration; and (ii) output oversmoothing, mitigated via context-overfitting strategy. On public benchmark with covariates, Baguan-TS consistently outperforms established baselines, achieving the highest win rate and significant reductions in both point and probabilistic forecasting metrics. Further evaluations across diverse real-world energy datasets demonstrate its robustness, yielding substantial improvements.

Applications · Health / Medicine

Sai Advaith Maddipatla, Anar Rzayev, Marco Pegoraro, Ailie Marx, Martin Pacesa, Paul Schanda, Sanketh Vedula, Alexander Bronstein

Protein function relies on dynamic conformational ensembles, yet current generative models like AlphaFold3 (AF3) often fail to produce ensembles that match experimental data. Recent experiment-guided generators attempt to address this by steering the reverse diffusion process. However, these methods are limited by fixed sampling horizons and sensitivity to initialization, often yielding thermodynamically implausible results. We introduce a general inference-time optimization framework to solve these challenges. First, we optimize over latent representations to maximize ensemble log-likelihood, rather than perturbing structures post hoc. This approach eliminates dependence on diffusion length, removes initialization bias, and easily incorporates external constraints. Second, we present novel sampling schemes for drawing Boltzmann-weighted ensembles. By combining structural priors from AF3 with force-field–based priors, we sample from their product distribution while balancing experimental likelihoods. Our results show that this framework consistently outperforms state-of-the-art guidance, improving diversity, physical energy, and agreement with data in X-ray crystallography and NMR, sometimes fitting the experimental data better than deposited PDB structures. Finally, inference-time optimization experiments maximizing iPTM scores reveal that perturbing MSA embeddings can artificially inflate model confidence. This exposes a vulnerability in current design metrics, whose mitigation could offer a pathway to reduce false discovery rates in binder engineering.

Applications · Neuroscience, Cognitive Science

Kartikay Agrawal, Vaishnavi N, Abhijeet Vikram, Vedant Sharma, Ayon Borthakur

Spiking neural networks have attracted increasing attention for their energy efficiency, multiplication-free computation, and sparse event-based processing. In parallel, state space models have emerged as a scalable alternative to transformers for long-range sequence modelling by avoiding quadratic dependence on sequence length. We propose here a spiking heterogeneous harmonic resonate-and-fire state space model (S$H^2$RFSSM), a second-order spiking SSM for classification and regression on ultra-long sequences. S$H^2$RFSSM outperforms transformers and first-order SSMs on average while eliminating matrix multiplications, making it highly suitable for resource-constrained applications. Furthermore, we introduce a kernel-based spiking regressor that enables accurate modelling of dependencies in sequences of up to 50k steps. We also observe a reduction in spiking operations and improved performance with heterogeneity and discretisation in harmonic resonate-and-fire neuronal layers. Overall, we evaluate Harmonic Resonate-and Fire layers across 17 diverse datasets, spanning sensors, time series, and classification to long-term forecasting. Our results demonstrate that S$H^2$RFSSM achieves superior long-range modelling capability with energy efficiency, positioning it as a strong candidate for signal processing on resource-constrained devices for human activity recognition, time series classification, and regression.

Applications · Chemistry, Physics, and Earth Sciences

Xiaochen Du, Juno Nam, Jaemoo Choi, Wei Guo, Sathya Edamadaka, Junyi Sha, Elton Pan, Yongxin Chen, Molei Tao, Rafael Gomez-Bombarelli

Sampling from discrete distributions with multiple modes and energy barriers is fundamental to machine learning and computational physics. Recent discrete neural samplers like MDNS suffer from mode collapse and fail to sample high-energy barrier regions between modes, which is critical for free energy estimation and understanding phase transitions. We propose Metadynamics Discrete Neural Sampler (MetaDNS), a general framework integrating well-tempered metadynamics into discrete diffusion or autoregressive samplers. By maintaining an adaptive, history-dependent bias potential along selected low-dimensional coordinates, MetaDNS forces exploration of previously inaccessible regions, enabling free energy reconstruction infeasible with standard neural samplers due to a lack of high-energy samples. On challenging low-temperature benchmarks including Ising, Potts, and the copper-gold binary alloy, MetaDNS reproduces the thermodynamic distribution. Compared to MCMC-based metadynamics, MetaDNS also achieves comparable exploration requiring fewer energy evaluations.

Social Aspects · Security

Zhuangzhuang Zhang, MingXin Li, Libing Wu, Wei-Bin Lee, Jianping Wang

Collaborative perception (CP) significantly extends the sensing range of connected and autonomous vehicles (CAVs). However, its reliance on data fusion among multiple CAVs makes it inherently vulnerable to adversarial attacks from malicious participants. Existing defenses primarily rely on output-level consensus, assuming that malicious messages manifest as statistical outliers, while suffering from poor adaptability to environmental noise. This makes them vulnerable to stealthy adversarial attacks and prone to high false positive rates. To address this challenge, we shift the defense paradigm from superficial output-level consensus to deeper consistency within the internal feature space. Guided by this principle, we propose \texttt{Cerberus}, a novel defense framework against adversarial attacks in CP systems by leveraging multi-dimensional consistency in the feature space. By quantifying conflicts in topological structure, semantic direction, and energy distribution within feature maps, \texttt{Cerberus} effectively detects adversarial perturbations and provides dynamic protection against adversarial attacks. Experimental results demonstrate that \texttt{Cerberus} significantly outperforms state-of-the-art methods, effectively limiting the attack success rate to as low as 0.05\% while restoring the mAP to 0.88.

Deep Learning · Attention Mechanisms

Zhouzhiming Zhou, Yong He, Qiaoyun Wu, Chaoxu Mu, Ajmal Mian

The unordered nature of point cloud data poses significant challenges to conventional architectures primarily designed for structured data. Spiking neural networks (SNN), by virtue of their inherent sparsity and dynamics, are particularly well-suited for processing point clouds to effectively extract meaningful features. We propose SpikeNet, a novel spiking neural network architecture for energy-efficient and robust point cloud analysis. We introduce spiking-driven sparse attention mechanism coined the Spiking Vector Mask Transformer (SVMT). By dynamically aligning the sparsity of point cloud data through binary spiking masks, SVMT eliminates the need for softmax and multiplication operations, significantly improving computational efficiency. We also propose a Dynamic Sparse Spiking Residual (DSSR) structure and integrate it with SVMT to form the Spiking Neural Network (SpikeNet) for point cloud classification and segmentation. SpikeNet overcomes the trade-off between accuracy and efficiency in previous SNN methods, achieving collaborative optimization of performance and energy-efficiency. Experiments on benchmark datasets show that SpikeNet achieves state-of-the-art performance in shape classification and segmentation tasks, comparable to artificial neural network (ANN) based methods. Our source code is in supplementary material and will be made publicly available.

Applications · Time Series

Fangxu Yu, Xingang Guo, Lingzhi Yuan, Haoqiang Kang, Hongyu Zhao, Lianhui Qin, Furong Huang, Bin Hu, Tianyi Zhou

Time series data is ubiquitous in real-world scenarios and crucial for critical applications ranging from energy management to traffic control. Consequently, the ability to reason over time series is a fundamental skill for generalist models to solve complex problems. However, current benchmarks for generalist models largely overlook this dimension. To bridge this gap, we introduce TSRBench, a comprehensive multi-modal benchmark designed to stress-test the full spectrum of time series reasoning capabilities. TSRBench features: i) a diverse set of 4125 problems from 14 domains, and is categorized into 4 major dimensions: Perception, Reasoning, Prediction, and Decision-Making. ii) 15 tasks from the 4 dimensions evaluating essential reasoning capabilities (e.g., numerical reasoning). Through extensive experiments, we evaluated over 30 leading proprietary and open-source LLMs, VLMs, and TSLLMs within TSRBench. Our findings reveal that: i) scaling laws hold for perception and reasoning but break down for prediction; ii) strong reasoning does not guarantee accurate context-aware forecasting, indicating a decoupling between semantic understanding and numerical prediction; and iii) despite the complementary nature of textual and visual forms of time series as inputs, current multimodal models fail to effectively fuse them for reciprocal performance gains. TSRBench provides a standardized evaluation platform that not only highlights existing challenges but also offers valuable insights to advance generalist models.

Applications · Chemistry, Physics, and Earth Sciences

Yitian Wang, Zhao Yang, Angxiao Yue, Wentao Guo, Yaning Cui, Hongteng Xu

Modeling peptide cyclization is critical for the virtual screening of candidate peptides with desirable physical and pharmaceutical properties. This task is challenging because a cyclic peptide often exhibits diverse, ring-shaped conformations, which cannot be well captured by deterministic prediction models derived from linear peptide folding. In this study, we propose MuCO (Multi-stage Conformation Optimization), a generative peptide cyclization method that models the distribution of cyclic peptide conformations conditioned on the corresponding linear peptide. In principle, MuCO decouples the peptide cyclization task into three stages: topology-aware backbone design, generative side-chain packing, and physics-aware all-atom optimization, thereby generating and optimizing conformations of cyclic peptides in a coarse-to-fine manner. This multi-stage framework enables an efficient parallel sampling strategy for conformation generation and allows for rapid exploration of diverse, low-energy conformations. Experiments on the large-scale CPSea dataset demonstrate that MuCO significantly outperforms state-of-the-art methods in consistently in physical stability, structural diversity, secondary structure recovery, and computational efficiency, making it a promising computational tool for exploring and designing cyclic peptides.

Optimization · Everything Else

Guillaume Houry, Jean Feydy, François-Xavier Vialard

A fundamental challenge in data science is to match disparate point sets with each other. While optimal transport efficiently minimizes point *displacements* under a bijectivity constraint, it is inherently sensitive to rotations. Conversely, minimizing *distortions* via the Gromov-Wasserstein (GW) framework addresses this limitation but introduces a non-convex, computationally demanding optimization problem. In this work, we identify a broad class of distortion penalties that reduce to a simple alignment problem within a lifted feature space. Leveraging this insight, we introduce an iterative GW solver with a linear memory footprint and quadratic (rather than cubic) time complexity. Our method is differentiable, comes with strong theoretical guarantees, and scales to hundreds of thousands of points in minutes. This efficiency unlocks a wide range of geometric applications and enables the exploration of the GW energy landscape, whose local minima encode the symmetries of the matching problem.

Deep Learning · Generative Models and Autoencoders

Riccardo De Santi, Malte Franke, Ya-Ping Hsieh, Andreas Krause

Recent progress in large-scale flow and diffusion models raised two fundamental algorithmic challenges: $(i)$ control-based reward adaptation of pre-trained flows, and $(ii)$ integration of multiple models, i.e., flow merging. While current approaches address them separately, we introduce a unifying probability-space framework that subsumes both as limit cases, and enables *reward-guided flow merging*, allowing principled, task-aware combination of multiple pre-trained flows (e.g., merging priors while maximizing drug-discovery utilities). Our formulation renders possible to express a rich family of *operators over generative models densities*, including intersection (e.g., to enforce safety), union (e.g., to compose diverse models), interpolation (e.g., for discovery), their reward-guided counterparts, as well as complex logical expressions via *generative circuits*. Next, we introduce Reward-Guided Flow Merging (RFM), a mirror-descent scheme that reduces reward-guided flow merging to a sequence of standard fine-tuning problems. Then, we provide first-of-their-kind theoretical guarantees for reward-guided and *pure* flow merging via RFM. Ultimately, we showcase the capabilities of the proposed method on illustrative settings providing visually interpretable insights, and apply our method to high-dimensional de-novo molecular design and low-energy conformer generation.

Applications · Time Series

Qingwen Zeng, Dajun Guo, Zhaoge Bi, lining chen, Jushang Qiu, Yitian Yang, Carl Yang, Huaming Chen, Ling Chen

Time series forecasting is critical in domains such as finance, energy, and healthcare, yet real-world datasets often exhibit non-stationarity, noise, missing values, and distribution shifts, posing severe challenges for generalization. In practice, industry solutions typically rely on customized forecasting frameworks that combine imputation, decomposition, and specialized models. However, such frameworks are costly to engineer and maintain. Moreover, we observe that many frameworks suffer from the impacts of distribution shifts, which degrade their respective performance. It motivates a paradigm that transfers reliably across heterogeneous datasets while accumulating reusable strategy knowledge for large-scale, dynamic environments. Although large language model-based agents have recently shown strong reasoning and tool-use capabilities, existing approaches do not consistently adapt forecasting workflows across diverse time series. We identify two primary factors, including limited strategy-level supervision and the inherent complexity of mapping dataset-specific meta-features to effective forecasting strategies. To address these challenges, we propose BECRA, a novel agent training paradigm that learns forecasting intelligence through contrast-aware exploration and agent-level causal lesson extraction, without human-annotated supervision. BECRA distills symbolic strategy lessons that support in-context planning on unseen datasets, enabling zero-shot training adaptation.

Applications · Energy

Katherine Lambert, Sasha Luccioni

The rise in deployment of large language models has driven a surge in GPU demand and datacenter scaling, raising concerns about electricity use, grid stress, and the impacts of modern AI workloads. Distillation is often promoted as one of the most effective paths to obtain cheaper, more efficient models, yet these claims rarely account for the full end-to-end energy and resource costs, including crucial teacher-side workloads such as data generation, logit caching, and evaluation. We present a comprehensive energy accounting framework that measures the complete computational cost of distillation pipelines via detailed stage-wise tracking of GPU device power consumption. In our experiments, we separate and log empirical energy use across distinct phases and systematically measure the energy and emissions of two common distillation methods: the classic logit-based knowledge distillation and synthetic-data supervised fine-tuning, constructing energy–quality–throughput Pareto frontiers that expose the previously ignored costs. From these measurements and analyses, we derive practical design rules for selecting distillation methods and hyperparameters under energy and budget constraints, and release an open-source measurement harness and accounting protocol to provide a standardized foundation for comparable, reproducible distillation research, explicitly accountable for complete pipeline energy impact.

Optimization · Discrete and Combinatorial Optimization

Yixin Huang, Shengyu Feng, Yiming Yang

Existing neural combinatorial optimization (CO) solvers often rely heavily on expensive labeled data and additional post-processing to produce feasible solutions. Research into mixed integer linear programs (MILPs) is particularly limited due to the lack of effective heuristics for feasibility and the challenge of modeling mixed-type variables for neural solvers. To address these issues, we propose a novel unsupervised Langevin sampler for solving MILPs. Our framework learns only integer variables, while continuous variables are solved using an exact linear programming solver, thus isolating the combinatorial hardness of the problem and avoiding unnecessary modeling complexity. The sampler is based on Langevin dynamics and incorporates both objective optimization and constraint satisfaction into a unified energy function, enabling the model to jointly learn feasibility and optimality. Experiments demonstrate that our method achieves 100\% feasibility without expensive post-processing and matches or outperforms supervised solvers on benchmark datasets, highlighting its effectiveness and scalability.