论文检索

Applications · Time Series

YingHao Ai, Yukai Zhou, Ruoxi Jiang, Junyi An, Chao Qu, Zhijian Zhou, Shiyu Wang, Fenglei Cao, Zenglin Xu, Furao Shen 等

Spatio-temporal forecasting is critical for real-world applications like traffic management, yet capturing complex interactions under high-noise conditions remains challenging. While current methods have shown improved accuracy using spatial physical priors, they often struggle with evolving temporal correlations and systematic errors. In this work, we propose a nested forecasting framework that couples future macro-level regional trends with micro-level historical observations, enabling top-down guidance from abstract future representations for fine-grained forecasting. Specifically, we construct semantically coherent regions via spectral clustering and design a progressive coarse-to-fine predictor to inject macro-dynamics into node-level forecasting. Extensive experiments on multiple real-world datasets demonstrate that our method consistently outperforms state-of-the-art baselines, validating the effectiveness of future macro-guided nested forecasting.

Optimization · Convex

Junhyeong Lee, Sangjin Jin, Yongjae Lee

Decision-Focused Learning (DFL) trains predictors to improve downstream decision quality, but computing regret gradients typically requires differentiating through solvers or relying on surrogate losses, which can be computationally expensive or deviate from the true objective. We show that, under standard regularity with locally stable active constraints, the regret gradient admits a closed-form geometric characterization, equivalent to the prediction error projected onto the tangent space of active constraints, scaled by lo- cal curvature. This reveals that regret gradients can be obtained by filtering decision-irrelevant components from the MSE gradient, providing a simpler and more direct alternative to existing approaches. Based on this, we propose PEAR (Projected Error As Regret-gradient), which computes regret gradients via a reduced linear system over active constraints, avoiding differentiation through solver iterations or additional optimization solves. Experiments on LP benchmarks and a real-world QP task show that PEAR achieves the best decision quality among all baselines while being the most computationally efficient, with gains that persist under constraint shifts.

Deep Learning · Other Representation Learning

Jianqiao Zeng, Ruocheng Wang, Yanzhi Liu, Hao Xiong, Junchi Yan

Despite the fast progress in neural operator learning, long-sequence modeling still is a standing challenge whereby latent states have been introduced with techniques well derived. Diverging from existing methods that treat latent states as transient variables or decoupled representations, CoEvol-NO introduces a {persistent state} to establish a {co-evolutionary framework}, where the latent state and mesh sequence are updated jointly and bidirectionally. Inspired by classical numerical methods, we model the layer-wise state evolution as a {Predictor-Corrector (PC)} process. Specifically, a ``Predictor'' generates a tentative target, followed by a ``Corrector'' that refines the persistent state via an {error-driven update mechanism}. Furthermore, our theoretical analysis reveals that the widely used \textit{direct substitution} and \textit{residual update} paradigms are essentially {first-order approximations} of this error-driven correction under different loss assumptions. We theoretically prove that CoEvol-NO achieves strict {linear time complexity}. Extensive experiments on five standard benchmarks and two large-scale industrial design tasks demonstrate that CoEvol-NO consistently achieves {state-of-the-art (SOTA)} performance.

Applications · Health / Medicine

Jiahao Kuang, Nuowei Liu, Changzhi Sun, Jie Wang, Tao Ji, Yuanbin Wu

Function-guided protein design is a crucial task with significant applications in drug discovery and enzyme engineering. However, the field lacks a unified and comprehensive evaluation framework. Current models are assessed using inconsistent and limited subsets of metrics, which prevents fair comparison and a clear understanding of the relationships between different evaluation criteria. To address this gap, we introduce **PDFBench**, the first comprehensive benchmark for function-guided de novo protein design. Our benchmark systematically evaluates eight state-of-the-art models on 16 metrics across two key settings: description-guided design, for which we repurpose the Mol-Instructions dataset, originally lacking quantitative benchmarking, and keyword-guided design, for which we introduce a new test set, SwissTest, created with a strict datetime cutoff to ensure data integrity. By benchmarking across a wide array of metrics and analyzing their correlations, **PDFBench** enables more reliable model comparisons and provides key insights to guide future research.

Reinforcement Learning · Deep RL

Lirui Luo, Guoxi Zhang, Hongming Xu, Cong Fang, Qing Li

In DRL, an agent is trained from a stream of experience. In a continual learning setting, such agents can suffer from \emph{plasticity loss}: their ability to learn new skills from new experiences diminishes over training. Recently, Mixture-of-Experts (MoE) networks have been reported to enable scaling laws and facilitate the learning of diverse skills. However, in continual reinforcement learning settings, their performance can degenerate as learning proceeds, indicating a loss of plasticity. To address this, building on Neural Tangent Kernel (NTK) theory, we formalize the plasticity loss in MoE policies as a loss of \emph{spectral plasticity}. We then derive a tractable proxy for spectral plasticity, one expressible in terms of individual expert feature matrices. Leveraging this proxy, we introduce \emph{SPHERE}, a practical Parseval penalty tailored for MoE-based policies that alleviates the loss of spectral plasticity. On MetaWorld and HumanoidBench, SPHERE improves average success under continual RL by 133\% and 50\% over an unregularized MoE baseline, while maintaining higher spectral plasticity throughout training.

Social Aspects · Everything Else

Harry Jiang, Baris Askin, Gauri Joshi, Carlee Joe-Wong

Machine learning (ML) systems have grown significantly in size and popularity over recent years. However, the data and computation power supply chains which have helped fuel this growth have not been built without controversy. In particular, some of the data used to train these models may have been used without permission, while the growing appetite for compute power in model training increasingly incentivizes consolidation of access to larger players. As some stakeholders, such as data owners and everyday consumers of the Internet, have felt left behind by the emerging ML ecosystem, we seek to use federated learning paradigm as a model and motivation to develop a more democratized future for the ML community: one that is more decentralized, cooperative, and accountable. This position paper argues that the original proposition of federated learning as a framework enabling cooperation, privacy, and decentralization is still relevant today, even after the emergence of large foundation model- and scaling law-driven ML research, and that FL can inspire alternative ML ecosystems which alleviate and avoid the current frictions of large ML systems.

Deep Learning · Large Language Models

Dezhuang Miao, Xiaoming Zhang, Bo Zhang, Yibin Du, Xiang Li, Ruilin Zeng, Yirui QI

LLM-to-KG systems frequently fail on exclusion-rich questions because natural-language negation is both scope-sensitive and evidence-dependent: it may constrain only one subgoal/branch and only certain supporting paths, yet such attachment is rarely explicit in text. We propose the Executable Exchange Contract (EEC) to bridge this gap, specifying scope-bound exclusions as executable control metadata exchanged between a specifier and an executor. Our executor, MatLogic, compiles exclusions into scope-local masks applied during multi-hop propagation and executes requests under a unified P$\rightarrow$N$\rightarrow$C/D schedule, ensuring exclusions are enforced before witness loss and branch entanglement. The system can also return compact witness pointers to keep support types distinguishable when needed. We evaluate on both structured complex queries and end-to-end natural-language KGQA, and introduce contract-aligned diagnostics that isolate errors from specification versus execution and verify the necessity of scoped enforcement.

Deep Learning · Theory

William Merrill, Hongjian Jiang, Yanhong Li, Anthony Lin, Ashish Sabharwal

The community is increasingly exploring linear RNNs (LRNNs) as language models, motivated by their expressive power and parallelizability. While prior work establishes the expressivity benefits of LRNNs over transformers, it is unclear what makes LRNNs---but not traditional, *nonlinear* RNNs---as easy to parallelize in practice as transformers. We answer this question by providing a tight connection between types of RNNs and standard complexity classes. We show that LRNNs can be viewed as log-depth (bounded fan-in) arithmetic circuits, which represents only a slight depth overhead relative to log-depth boolean circuits that transformers admit. Furthermore, we show that nonlinear RNNs can solve $\mathsf L$-complete problems (and even $\mathsf P$-complete ones, under polynomial precision), revealing a fundamental barrier to parallelizing them as efficiently as transformers. Our theory also identifies fine-grained expressivity differences between recent popular LRNN variants: permutation-diagonal LRNNs are $\mathsf{NC}^1$-complete whereas diagonal-plus-low-rank LRNNs are more expressive ($\mathsf{PNC}^1$-complete). We provide further insight by associating each type of RNN with a corresponding automata-theoretic model that it can simulate. Together, our results reveal fundamental tradeoffs between nonlinear RNNs and different variants of LRNNs, providing a foundation for designing LLM architectures that achieve an optimal balance between expressivity and parallelism.

Applications · Computer Vision

Antoine Schnepf, Karim Kassab, Flavian Vasile, Andrew Comport

The generation of immersive and navigable 3D environments is increasingly prevalent with the growing adoption of virtual reality and 3D content. However, recent methods face a fundamental limitation: they cannot produce 3D worlds that simultaneously (i) are navigable over long-range spatial extents and (ii) cover the complete omnidirectional field of view ($360^\circ$ horizontally and $180^\circ$ vertically). To address this challenge, we introduce SphericalDreamer, a method for generating fully immersive and long-range 3D environments from textual prompts. Our approach is built on the generation of multiple panoramic images, which are subsequently lifted into 3D and fused together while maintaining visual and geometric consistency. SphericalDreamer produces highly detailed, fully immersive 3D environments, while substantially improving scale and navigability compared to prior approaches.

Deep Learning · Graph Neural Networks

Xuan Minh Vuong Nguyen, Nissrine Akkari, Fabien Casenave, Jonathan Viquerat, Elie Hachem

Graph Neural Networks (GNNs) on meshes have emerged as promising surrogates for computational mechanics, but standard local message passing struggles to propagate information across unstructured meshes, leading to large errors in regions with complex physics (e.g., shocks, wakes, boundary layers). Existing approaches enlarge connectivity with long-range edges chosen *a priori* via geometric heuristics or random sampling, which lack a mechanism to prioritize high-error regions and often introduce redundant communication. We propose **MiSe-GNN**, a dual-head architecture that adaptively augments graph connectivity using model-predicted *a posteriori* errors. MiSe-GNN jointly predicts physical fields and a node-wise error indicator; the predicted error is periodically converted into a hierarchy of additional edges via an adaptive tree that links high-error nodes to spatial pivots at multiple scales. This error-guided connectivity concentrates message passing where the surrogate is uncertain while keeping the graph sparse elsewhere, yielding a transparent and physically interpretable graph-space analogue of adaptive mesh refinement. Across industrial CFD and CSD benchmarks, MiSe-GNN consistently improves accuracy and accuracy–compute trade-offs over strong baselines, and qualitative analyses show that it routes communication toward physically challenging regions. These results establish *error-guided edge augmentation* as a robust and general design principle for long-range message passing in physics-aware GNNs.

Applications · Computer Vision

Hao Liu, Yi Zhou, Yanni Ma

Voxel-based 3D object detectors typically discretize the spatial domain using a uniform Cartesian grid, which allocates the same voxel size to both near-range and far-range regions. However, this uniform discretization is suboptimal for small objects such as pedestrians and cyclists, as they occupy only a few voxels and thus struggle to capture fine-grained geometric details. Although increasing the global voxel resolution can alleviate this problem, it inevitably increases substantial memory consumption and computational overhead. In this paper, we propose Radial Scaling Voxelization (RSV), a simple yet effective non-uniform discretization strategy that adaptively modulates the effective voxel size based on the radial distance from the LiDAR sensor. Unlike previous cylindrical or polar discretization schemes, RSV preserves the Cartesian grid topology by applying a continuous radial scaling function to the input coordinates before standard voxelization. This operation yields a near-high, far-unchanged resolution pattern: the effective voxel size becomes finer in near regions, where the geometric structures of small objects are difficult to capture, while remaining nearly unchanged in far regions to avoid unnecessary computational cost. Importantly, RSV is architecture-agnostic and can directly replace the discretization module in any voxel-based detector without modifying the backbone, network design, or training pipeline. Extensive experiments on the KITTI and nuScenes datasets demonstrate that integrating our RSV into several voxel-based baselines consistently enhances small-object detection performance, especially for the Pedestrian and Cyclist categories, while incurring only marginal additional computational overhead.

Deep Learning · Attention Mechanisms

QIUHAO Zeng, Jerry Huang, Peng Lu, Ruiyi Fang, Gezheng Xu, Zihao Jing, Yufei Cui, Charles X. Ling, Gang Niu, Boyu Wang

Despite advances in long-context inference, large language models (LLMs) remain fundamentally limited by the key-value (KV) caching mechanisms that are necessary for stable computation. Management techniques, such as selective token eviction and pruning, have vastly mitigated the issues that have arisen, but often discard potentially useful information to manage the growing memory requirements of the cache. In this paper, we build upon these approaches to propose Attention with Routed Memory ARM, a novel KV caching structure that introduces a fully differentiable, fixed-size memory system organized as a hierarchical routing structure that learns to select memory slots via Gumbel-Softmax and performs sigmoid-gated updates that softly combine new and stored information, avoiding hard eviction and thereby reducing information loss. By combining this with a policy to dynamically select varying amounts of memory at inference, ARM adapts its accesses for simple contexts and expanding retrieval for inputs that require deeper reasoning, enabling more scalable and effective retrieval on both short and long contexts. Experimental results on standard commonsense and long-context reasoning benchmarks demonstrate that ARM achieves superior performance and efficiency compared to fixed KV-caching approaches, while remaining efficient and scalable in terms of both memory and generation latency.

Deep Learning · Attention Mechanisms

Rahul Vashisht, Harish Ramaswamy

A standard self-attention layer consists of two interacting circuits: the query-key circuit that governs attention allocation, and the output-value circuit that maps attended representations to predictions. Collapsed and factorized parameterizations of the query-key and output-value circuits lead to qualitatively different attention patterns. In particular, some parameterizations give sharper attention to task-relevant tokens, at a similar training loss. We analyze how the parameterizations of these circuits shape the parameter trajectories in single-layer self-attention models trained on next-token prediction. Through gradient-flow analysis, we show that factorization induces implicit rescaling of the two circuits' learning rates. We derive closed-form dynamics showing that output-value and query-key parameters move along a line, with relative speeds determined by their learning rates. Faster query-key learning relative to output-value learning thus produces sharper attention, as the model compensates for slower output-value learning by increasing attention mass on relevant tokens. Experiments show that differences in the relative learning rates of the two circuits govern attention concentration. This improves attention interpretability while maintaining comparable predictive performance.

Applications · Computer Vision

Jiahao Li, Qingwang Zhang, Qiuyu Chen, Guozhan Qiu, Yunzhong Lou, Xiangdong Zhou

The field of Computer-Aided Design (CAD) generation has made significant progress in recent years. Existing methods typically fall into two separate categorie: parametric CAD modeling and direct boundary representation (B-Rep) synthesis. In modern feature-based CAD systems, parametric modeling and B-Rep are inherently intertwined, as advanced parametric operations (e.g., *fillet* and *chamfer*) require explicit selection of B-Rep geometric primitives, and the B-Rep itself is derived from parametric operations. Consequently, this paradigm gap remains a critical factor limiting AI-driven CAD modeling for complex industrial product design. This paper present *FutureCAD*, a novel text-to-CAD framework that leverages large language models (LLMs) and a B-Rep grounding transformer (*BRepGround*) for high-fidelity CAD generation. Our method generates executable CadQuery scripts, and introduces a text-based query mechanism that enables the LLM to specify geometric selections via natural language, which *BRepGround* then grounds to the target primitives. To train our framework, we construct a new dataset comprising real-world CAD models. For the LLM, we apply supervised fine-tuning (SFT) to establish fundamental CAD generation capabilities, followed by reinforcement learning (RL) to improve generalization. Experiments show that *FutureCAD* achieves state-of-the-art CAD generation performance.

Deep Learning · Large Language Models

Ren-Biao Liu, Li Xin-Ye, Hui Sun, Yali Du, Jiang-Tian Xue, Ming Li

Training large language models for code generation requires selecting high-quality data from solution pools where each problem admits multiple correct implementations. Conventional studies on data selection hold that sophisticated strategies that employ various optimization objectives, such as diversity maximization or difficulty ranking, should outperform naive random sampling. However, research on *whether sophisticated selection methods truly benefit code generation* remains limited. In this work, we systematically evaluate multiple selection strategies across different representation spaces, including continuous embeddings, discrete tokens, and syntactic structures, using various base language models. We observe counterintuitive phenomena, suggesting that sophisticated methods may not yield robust benefits. Instead, simple random sampling achieves consistently competitive performance across all models, exhibiting greater stability and transferability than sophisticated methods. We attribute that an *implicit knowledge consensus* exists among diverse correct solutions, such that random selection already covers the common algorithmic knowledge required for training. Our findings provide practical insights into data selection for code generation, suggesting practitioners can adopt simple random sampling without sacrificing performance.

Probabilistic Methods · Bayesian Models and Methods

Kenyon Ng, Edwin Fong, David Frazier, Jeremias Knoblauch, Susan Wei

Bayesian inference provides principled uncertainty quantification but is often limited by challenges of prior and likelihood elicitation. The martingale posterior (MGP) (Fong et al, 2023) offers an alternative by replacing these requirements with a predictive rule. Additionally MGP focuses inference on parameters defined through a loss function. This framework is especially resonant in the era of foundation transformers; practitioners increasingly leverage models like TabPFN for their state-of-the-art capabilities, yet often require epistemic uncertainty for a scientific estimand $\theta$ that need not parametrize the model’s implicit latent model. The MGP provides the mechanism to recover these posterior distributions. We introduce TabMGP, an MGP built on TabPFN for tabular data. TabMGP produces credible sets with near-nominal coverage and often outperforms both handcrafted MGP constructions and standard Bayesian baselines.

General Machine Learning · Kernel methods

Louis Allain, Sébastien Da Veiga, Brian Staber

Conformal prediction (CP) is a distribution-free method to construct reliable prediction intervals that has gained significant attention in recent years. Despite its success and various proposed extensions, a significant practical feature which has been overlooked in previous research is the potential skewed nature of the noise, or of the residuals when the predictive model exhibits bias. In this work, we leverage recent developments in CP to propose a new asymmetric procedure that bridges the gap between skewed and non-skewed noise distributions, while still maintaining adaptivity of the prediction intervals. We introduce a new statistical learning problem to construct adaptive and asymmetric prediction bands, with a unique feature based on a penalty which promotes symmetry: when the penalty intensity varies, the intervals smoothly change from symmetric to asymmetric ones. This statistical learning problem is based on reproducing kernel Hilbert spaces and the recently introduced kernel sum-of-squares framework. First, we establish representer theorems to make our problem tractable in practice, and derive dual formulations which are essential for scalability to larger datasets. Second, the intensity of the penalty is chosen using a novel data-driven method which automatically identifies the symmetric nature of the noise. We show that consenting to some asymmetry can let the learned prediction bands better adapt to small sample regimes, outliers or biased predictive models. Finally, our experiments illustrate the efficiency of such penalized kernel sum-of-squares to construct adaptive prediction bands.

Deep Learning · Large Language Models

Yun Qu, Qi Wang, Yixiu Mao, Heming Zou, Yuhang Jiang, Weijie Liu, Clive Bai, Kai Yang, Yangkun Chen, Saiyong Yang 等

Reinforcement learning enhances the reasoning capabilities of large language models but often involves high computational costs due to rollout-intensive optimization. Online prompt selection presents a plausible solution by prioritizing informative prompts to improve training efficiency. However, current methods either depend on costly, exact evaluations or construct prompt-specific predictive models lacking generalization across prompts. This study introduces Generalizable Predictive Prompt Selection (GPS), which performs Bayesian inference towards prompt difficulty using a lightweight generative model trained on the shared optimization history. Intermediate-difficulty prioritization and history-anchored diversity are incorporated into the batch acquisition principle to select informative prompt batches. The small predictive model also generalizes at test-time for efficient computational allocation. Experiments across varied reasoning benchmarks indicate GPS's substantial improvements in training efficiency, final performance, and test-time efficiency over superior baseline methods. The code is available at https://anonymous.4open.science/r/GPS-ICML.

Deep Learning · Other Representation Learning

Fabian Gröger, Shuo Wen, Maria Brbic

The Platonic Representation Hypothesis suggests that representations from neural networks are converging to a common statistical model of reality. We show that the existing metrics used to measure representational similarity are *confounded by network scale*: increasing model depth or width can systematically inflate representational similarity scores. To correct these effects, we introduce a permutation-based null-calibration framework that transforms any representational similarity metric into a calibrated score with statistical guarantees. We revisit the Platonic Representation Hypothesis with our calibration framework, which reveals a nuanced picture: the apparent convergence reported by global spectral measures largely disappears after calibration, while local neighborhood similarity, but not local distances, retains significant agreement across different modalities. Based on these findings, we propose the *Aristotelian Representation Hypothesis*: representations in neural networks are converging to shared local neighborhood relationships.

Deep Learning · Large Language Models

Longhui Zhang, Jiahao Wang, Chenhao Hu, Bingyu Liang, Jing Li, Min zhang

While large language models (LLMs) have greatly advanced the functional correctness of automated code translation systems, the runtime efficiency of translated programs has received comparatively little attention. With the waning of Moore’s law, runtime efficiency has become increasingly important for program quality, alongside functional correctness. Our preliminary study reveals that LLM-translated programs often run slower than human-written ones, and this issue cannot be remedied through prompt engineering alone. Therefore, our work proposes SwiftTrans, a code translation framework comprising two key stages: (1) Multi-Perspective Exploration, where MpTranslator leverages parallel in-context learning (ICL) to generate diverse translation candidates; and (2) Difference-Aware Selection, where DiffSelector identifies the optimal candidate by explicitly comparing differences between translations. We further introduce Hierarchical Guidance for MpTranslator and Ordinal Guidance for DiffSelector, enabling LLMs to better adapt to these two core components. To support the evaluation of runtime efficiency in translated programs, we extend existing benchmarks, CodeNet and F2SBench, and introduce a new benchmark, SwiftBench. Experimental results across all three benchmarks show that SwiftTrans achieves consistent improvements in both correctness and runtime efficiency.