论文检索

Applications · Robotics

Zhiyu Huang, Yun Zhang, Johnson Liu, Rui Song, Chen Tang, Jiaqi Ma

Robots in dynamic, human-centric environments must follow language instructions while maintaining real-time reactive control. Vision-language-action (VLA) models offer a promising framework, but they assume temporally aligned reasoning and control, despite semantic inference being inherently delayed relative to real-time action. We introduce Think-in-Control (TIC)-VLA, a latency-aware framework that explicitly models delayed semantic reasoning during action generation. TIC-VLA defines a delayed semantic-control interface that conditions action generation on delayed vision-language semantic states and explicit latency metadata, in addition to current observations. We further propose a latency-consistent training pipeline that injects reasoning inference delays during imitation learning and online reinforcement learning, aligning training with asynchronous deployment. To support realistic evaluation, we present DynaNav, a physics-accurate, photo-realistic simulation suite for language-guided navigation in dynamic environments. Extensive experiments in simulation and on a real robot show that TIC-VLA consistently outperforms prior VLA models while maintaining robust real-time control under multi-second reasoning latency. Code, data, and benchmarks will be released for reproducibility.

Deep Learning · Generative Models and Autoencoders

Yu-Yang Qian, Junda Su, Lanxiang Hu, Peiyuan Zhang, Zhijie Deng, Peng Zhao, Hao Zhang

Diffusion large language models (dLLMs) offer capabilities beyond those of autoregressive (AR) LLMs, such as parallel decoding and random-order generation. However, realizing these benefits in practice is non-trivial, as dLLMs inherently face an *accuracy-parallelism trade-off*. Despite increasing interest, existing methods typically focus on only one-side of the coin, targeting either efficiency or performance. To address this limitation, we propose d3LLM (*Pseudo-Distilled Diffusion Large Language Model*), striking a balance between accuracy and parallelism: (i) during training, we introduce *pseudo-trajectory distillation* to teach the model which tokens can be decoded confidently at early steps, thereby improving parallelism; (ii) during inference, we employ *entropy-based multi-block decoding* with a KV-cache refresh mechanism to achieve high parallelism while maintaining accuracy. To better evaluate dLLMs, we also introduce AUP (*Accuracy Under Parallelism*), a new metric that jointly measures accuracy and parallelism. Experiments demonstrate that our d3LLM achieves up to $10\times$ speedup over vanilla LLaDA/Dream, and up to $5\times$ speedup over the AR models (Qwen-2.5-7B) without much accuracy degradation.

Probabilistic Methods · Spectral Methods

Genki Osada

While diffusion models enable new approaches for estimating Local Intrinsic Dimension (LID), existing methods fail in high-dimensional spaces where noise from vast normal directions overwhelms the tangent signal. We propose Local Hessian Spectral Dimension (LHSD), which resolves this by applying spectral filtering to the log-density Hessian, explicitly cutting off large eigenvalues associated with normal directions to count zero-curvature tangent directions. Implemented using Stochastic Lanczos Quadrature (SLQ), LHSD avoids full Hessian construction, achieving linear scalability with dimension $D$. Experiments on synthetic and real data confirm LHSD’s superior robustness and its utility in detecting memorization in large-scale diffusion models.

Theory · Reinforcement Learning and Planning

Vagul Mahadevan, Claire Chen, Shuze Liu, Shangtong Zhang

Stochastic approximations (SA)--algorithms which derive their power through the use of random, incremental updates--are at the heart of reinforcement learning (RL). Expanding the theory of SA has established rigorous results concerning the most important algorithms in RL, including stochastic gradient descent and temporal difference learning. In this work, we focus on two-timescale stochastic approximations, a class which notably includes temporal difference learning with gradient correction (TDC) and actor-critic methods. Prior work has developed stability (boundedness) and convergence criteria for two-timescale SA under i.i.d. noise, but analogous results for Markovian noise have remained elusive--a critical issue since RL data are generated by a Markov chain, making i.i.d. assumptions unrealistic. To address this gap, we present the first stability result and the first asymptotic convergence result for two-timescale schemes with Markovian noise under general, verifiable conditions--notably, without resorting to projected variants of the schemes or requiring the noise to be in a compact space. As a key application, we contribute the first asymptotic convergence proof of TDC, an off-policy prediction algorithm with linear approximation and eligibility traces. Together, our results extend SA theory, establishing the first theoretical foundation for analysis of two-timescale algorithms with the realistic noise models inherent to RL.

General Machine Learning · Causality

Shimeng Huang, Matthew Robinson, Francesco Locatello

Mendelian Randomization (MR) is a prominent observational epidemiological research method, designed to address unobserved confounding when estimating causal effects. It is closely related to instrumental variable (IV) methods, where genetic variants serve as instruments to infer causal relationships from observational data. However, the core assumptions required for valid IV analysis---particularly the independence between instruments and unobserved confounders---are untestable and often violated in practice. In MR, such violations commonly arise when genetic variants are correlated with environmental factors (e.g., population stratification and assortive mating), leading to confounding between instruments and outcomes. At the same time, MR studies increasingly include data collected across multiple environments or populations, providing an opportunity to address these violations. Leveraging this setting, we propose a representation learning framework that exploits multi-environment data to recover latent exogenous components of genetic instruments suitable for causal inference. We provide theoretical insights into when and how the learned components can act as valid instruments, and we demonstrate the effectiveness of our approach through simulations and semi-synthetic experiments using genetic data from the All of Us Biobank.

Deep Learning · Foundation Models

Jiaxing Qiu, Kaihua Hou, Roxana Daneshjou, Ahmed Alaa, Thomas Hartvigsen

Model editing aims to correct errors in large, pretrained models without altering unrelated behaviors. While some recent works have edited vision–language models (VLMs), no existing editors tackle reasoning-heavy tasks, which typically require humans and models to reason about images. We therefore propose ReasonEdit, the first VLM editor to let users explain their reasoning during editing, introducing a new, practical model editing setup. ReasonEdit continuously stores human reasoning in a codebook, and retrieves only relevant facts during inference using a novel topology-balanced multimodal embedding method inspired by network science. Across four VLMs on multiple rationale-based visual question answering datasets, ReasonEdit achieves state-of-the-art editing performance, ultimately showing that using human reasoning during editing greatly improves edit generalization.

Deep Learning · Robustness

Ruize Zhang, Yu Li, Zhang Wan, Juan Cao, Jie Zhang, Sheng Tang

Recent test-time defenses for CLIP claim to preserve zero-shot clean accuracy while improving adversarial robustness. However, we find the reported robustness of six recent proposed state-of-the-art methods substantially overestimated: they fail under basic adaptive attacks. We further observe that these defenses share a common reliance on an indicative measurement that is assumed to capture the distributional difference between clean and adversarial samples and to determine whether the defense should preserve or alter the static model’s prediction. We argue that this assumption is the fundamental weakness, and we propose CLIP-MAD (Manipulating Assumed Difference), an adaptive attack strategy designed to break it. CLIP-MAD efficiently expands the adversarial distribution without costly full gradient calculations and can be flexibly combined with existing attack baselines to further boost attack strength. Experiments across 13 datasets demonstrate that CLIP-MAD produces strong adversarial samples that markedly reduce the robustness of diverse test-time defenses, revealing a false sense of security in CLIP’s zero-shot robustness.

Deep Learning · Generative Models and Autoencoders

Nicolas Dufour, Lucas Degeorge, Arijit Ghosh, Vicky Kalogeiton, David Picard

The default paradigm of post-training text-to-image generators includes post-hoc selection of generated images, and subsequent training with one reward model to align the generator to the reward, typically user preference. This discards informative data as well as optimizes only for a single reward, hence harming diversity, semantic fidelity and efficiency. Instead, we propose MIRO, a method that conditions the model on multiple rewards during training, thus letting the model learn user preferences directly. MIRO pre-training both improves the visual quality of the generated images and speeds up the training, achieving state of the art on the GenEval compositional benchmark and user-preference scores (PickAScore, ImageReward, HPSv2).

Applications · Robotics

Rufeng Chen, Yue Chang, Xiaqiang Tang, Hechang Chen, Sihong Xie

Open-vocabulary navigation requires embodied agents to manage significant perception uncertainty stemming from semantic ambiguity and model errors. However, most existing works settle for local optimal deterministic approaches, depriving complex navigation decision-making over multiple composite possibilities that are critical for globally better solutions. In this paper, we propose Probabilistic Scene Graph Navigation (PSG-Nav), which constructs a 3D Probabilistic Scene Graph that uses full semantic categorical distributions to account for perception uncertainty. To efficiently use the local distributions to compose and reason about the optimal navigation landmarks, we propose Multiverse Decision to sample multiple most likely world settings from the joint distribution, and evaluate navigation landmarks based on the compatibility between landmarks and multiverses. To mitigate false positives due to epistemic uncertainty in open-vocabulary navigation, we introduce the Evidential Experience Calibrator, which enables online lifelong adaptation by cross-validating detections against memories of past successes and failures. Extensive experiments on widely-used benchmarks MP3D, HD3D, and HSSD demonstrate that PSG-Nav establishes new state-of-the-art results, achieving a Success Rate of 66.1%, 44.8%, and 67.9%, respectively.

Theory · Reinforcement Learning and Planning

Muhammed Emrullah Ildiz, Halil Alperen Gozeten, Ege Onur Taga, Samet Oymak

State-of-the-art reasoning models can utilize long chain-of-thought to solve sophisticated coding and math problems. During this process, the model often attemps at a solution multiple times by utilizing verification and self-reflection capabilities. In this work, we view a long CoT as a process where the model makes K attempts at solving a problem in which each attempt is allowed to build on earlier solutions. This way, we formalize long CoT as a pass@K problem with dependent samples. Under this formalism, we derive the policy gradient and RL algorithms for optimizing long CoT reward and derive how each attempt should be weighed for unbiased gradient computation while maintaining small variance. Our theory reveals how the self-correction capability and dense feedback influence the training and eventual performance of long CoT-based reasoning. We provide both synthetic and real experiments corroborating our theory and the benefits of the associated algorithms. As a by product, our research also reveals when verification and long chain-of-thought is beneficial over parallel sampling strategies and the role of the model capability.

General Machine Learning · Kernel methods

Johannes Teutsch, Oleksii Molodchyk, Marion Leibold, Timm Faulwasser, Armin Lederer

Providing non-conservative uncertainty quantification for function estimates derived from noisy observations remains a fundamental challenge in statistical machine learning, particularly for applications in safety-critical domains. In this work, we propose novel non-asymptotic probabilistic uniform error bounds for kernel-based regression. Compared to related bounds in the literature that are restricted to (conditionally) independent sub-Gaussian noise, our bounds allow to consider a broad class of non-Gaussian distributions, such as sub-Gaussian, bounded, sub-exponential, and variance-bounded noise. Moreover, our results apply to correlated and uncorrelated noise. We compare our proposed error bounds with existing results in terms of the induced uncertainty region and their performance in safe control, demonstrating the tightness of the proposed bounds.

General Machine Learning · Evaluation

Drew Keller, Kweku Kwegyir-Aggrey, Ryan Steed, Anita Rao, Julia Sharp, A. Bergman

Benchmarks are widely used to evaluate and compare the performance of artificial intelligence systems. However, some approaches to computing benchmark metrics produce invalid uncertainty estimates or make unrecognized assumptions about the evaluation setting. We leverage statistical modeling to make two contributions to the practice of AI benchmarking. First, we formally distinguish measurements of benchmark accuracy from generalized accuracy (performance on all potential test items similar to those included in the benchmark). Then, in a simulated setting and with large-scale evaluation of 22 API-access frontier large language models on 3 popular benchmarks, we show how analysis via generalized linear mixed model can estimate generalized accuracy while more efficiently quantifying uncertainty compared to existing regression-free approaches. We also show how this approach can equip evaluators with important context on evaluation results, including variance decomposition and item difficulty estimates that illuminate important aspects of LLM performance and benchmark construction.

Deep Learning · Other Representation Learning

Yuwei Wang, Guikun Chen, Xiruo Jiang, Yazhou Yao, Di Liu, Xiangbo Shu, Fumin Shen, Wenguan Wang

Recent advances in visual representation learning have seen the rise of clustering-based vision backbones, which adopt clustering as a core paradigm for feature extraction. However, existing clustering-based backbones typically rely on a single clustering algorithm, whose inherent inductive bias limits their representational capacity. To address this, we propose EnFormer, which embeds ensemble clustering as a core component of feature extraction. EnFormer structures feature extraction around two steps: (i) Ensemble Generation, where several differentiable base clustering methods are introduced to capture diverse semantic structures; and (ii) Consensus Aggregation, which employs a differentiable mechanism to fuse the results of all base clusterings to reconstruct refined visual features. Extensive experiments show that EnFormer consistently outperforms existing clustering-based backbones across core vision tasks, with higher performance and significantly improved throughput.

Deep Learning · Generative Models and Autoencoders

Gabriel Loaiza-Ganem, Kevin Zhang, Wei Cui, Marc Law, Kin Kwan Leung

Conformal prediction (CP) and its extension, conformal risk control (CRC), are established frameworks for quantifying uncertainty in supervised machine learning through formal guarantees. However, recent breakthroughs in artificial intelligence (AI) have been driven by unsupervised generative models, such as large language models (LLMs) and image generators, which are not directly compatible with CP or CRC. In this work we introduce conformal generation (Conf-Gen), a general framework adapting CRC to generative tasks while relaxing its theoretical assumptions. Conf-Gen unifies and generalizes previous attempts to apply CP to LLMs, and extends conformal methodology to entirely new domains. We demonstrate the flexibility of Conf-Gen through some novel applications, including obtaining conformal guarantees on: image generators producing non-memorized images, conversational AI systems having asked enough clarifying questions, and the output of AI agents being correct.

Deep Learning · Large Language Models

Ali Taghibakhshi, Ruisi Cai, Saurav Muralidharan, Sharath Turuvekere Sreenivas, Ameya Mahabaleshwarkar, Marcin Chochowski, Akhiad Bercovich, Ran Zilberstein, Ran El-Yaniv, Yonatan Geifman 等

Training a family of large language models (LLMs), either from scratch or via iterative compression, is prohibitively expensive and inefficient, requiring separate training runs for each model in the family. In this paper, we introduce Star Elastic, a novel LLM post-training method that adds N nested submodels to a given parent reasoning model using the compute of one run (Nx savings) via a single post-training job. Beyond reducing training costs, Star Elastic also addresses a fundamental limitation in efficient reasoning: the rigidity of static architectures, which forces the allocation of constant resources regardless of token difficulty. By unlocking elastic budget control, Star Elastic enables a novel approach that uses different submodels for each reasoning phase (thinking and answering). Star Elastic supports (1) nesting along the SSM, embedding channel, MoE and FFN axes, (2) learning nested submodels via an end-to-end trainable router, and (3) curriculum-based knowledge distillation. We apply Star Elastic to the NVIDIA Nemotron Nano models; in particular, we demonstrate its effectiveness on hybrid MoE architectures with Nemotron Nano v3 (30B/3.6A), generating 23B (2.8A) and 12B (2.0A) variants with 160B training tokens. For Nemotron Nano v2 (12B), we produce 9B and 6B nested models using only 110B training tokens, achieving a 360x reduction versus training from scratch and a 7x reduction over state-of-the-art compression methods. All nested models match or outperform independently trained baselines of comparable size. Crucially, elastic budget control advances the accuracy--latency Pareto frontier, achieving up to 16% higher accuracy and 1.9x lower latency via dynamic per-phase model selection.

General Machine Learning · Evaluation

Yuanzhe Hu, Xiaopeng Wang, Yuxin Wang, Xiaokun Zhong, Haiquan Lu, Tianyu Pang, Michael Mahoney, Yujun Yan, Pu Ren, Yaoqing Yang

Neural networks (NNs) trained under different hyperparameters can fall into distinct training ``regimes'', with models in the same regime showing homogeneous properties and models across regimes differing qualitatively. In this paper, we analyze multi-regime patterns in scientific machine learning (SciML) models by characterizing these regimes and the transitions between them. We show how different regimes affect trainability and generalization, and we demonstrate that loss-landscape analysis enables regime-based diagnostics to understand, evaluate, and improve SciML model training. Our analysis yields three key insights: (1) compared with computer vision (CV) tasks, SciML models exhibit significantly more pathological loss landscapes; (2) optimization methods are regime-specific -- different optimization strategies help in different regimes, but none is uniformly effective; and (3) SciML models exhibit fine-grained failure modes that challenge conventional interpretations of standard loss-landscape metrics. Using this study, we aim to unify our understanding of seemingly different failure modes across SciML tasks and obtain task-oblivious insights and methodologies for addressing these failures. We validate these findings across widely used SciML models, including physics-informed neural networks (PINNs), Fourier neural operators (FNOs), and Neural Ordinary Differential Equations (NeuralODEs), on benchmarks spanning representative ordinary and partial differential equations.

Reinforcement Learning · Multi-agent

Tian Xia, Tianrun Gao, Wenhao Deng, Long Wei, Xiaowei Qian, Chenglei Yu, Tailin Wu

Engineering construction automation aims to transform natural language specifications into physically viable structures, requiring complex integrated reasoning under strict physical constraints. While modern LLMs possess broad knowledge and strong reasoning capabilities that make them promising candidates for this domain, their construction competencies remain largely unevaluated. To address this gap, we introduce BuildArena, the first physics-aligned interactive benchmark designed for language-driven engineering construction. It takes a first step towards engineering automation using LLMs. Technically, it contributes to the community in two aspects: (1) an extendable task design strategy spanning static and dynamic mechanics across multiple difficulty tiers; (2) a 3D Spatial Geometric Computation Library for supporting construction based on language instructions. On nine frontier LLMs, BuildArena comprehensively evaluates their capabilities for language-driven and physics-grounded construction automation.

Deep Learning · Large Language Models

Jiyeon Kim, Sungik Choi, Yongrae Jo, Moontae Lee, Minjoon Seo

Diffusion-based language models(dLLMs) have emerged as a promising alternative to autoregressive language models, offering the potential for parallel token generation and bidirectional context modeling. However, harnessing this flexibility for fully non-autoregressive decoding remains an open question, particularly for reasoning and planning tasks. In this work, we investigate non-autoregressive decoding in dLLMs by systematically analyzing its inference dynamics along the temporal axis. Specifically, we uncover an inherent failure modes in confidence-based non-autoregressive generation stem from a strong proximity bias—the denoising order tends to concentrate on spatially adjacent tokens. This local dependency leads to spatial error propagation, rendering the entire trajectory critically contingent on the initial unmasking position. Leveraging this insight, we present a minimal-intervention approach that guides early token selection, employing a lightweight planner and end-of-sequence temperature annealing. We thoroughly evaluate our method on various reasoning and planning tasks and observe substantial overall improvement over existing heuristic baselines without significant computational overhead.

Deep Learning · Other Representation Learning

Yeqin Zhang, Yunfei Wang, Jiaxuan Chen, Ke Qin, Yizheng Zhao, Cam-Tu Nguyen

Sentence representations are foundational to many Natural Language Processing (NLP) applications. While recent methods leverage Large Language Models (LLMs) to derive sentence representations, most rely on final-layer hidden states, which are optimized for next-token prediction and thus often fail to capture global, sentence-level semantics. This paper introduces a novel perspective, demonstrating that attention value vectors capture sentence semantics more effectively than hidden states. We propose *Value Aggregation (VA)*, a simple method that pools token values across multiple layers and token indices. In a training-free setting, VA outperforms other LLM-based embeddings, even matches or surpasses the ensemble-based MetaEOL. Furthermore, we demonstrate that when paired with suitable prompts, the layer attention outputs can be interpreted as aligned weighted value vectors. Specifically, the attention scores of the last token function as the weights, while the output projection matrix ($W_O$) aligns these weighted value vectors with the common space of the LLM residual stream. This refined method, termed *Aligned Weighted VA (AlignedWVA)*, achieves state-of-the-art performance among training-free LLM-based embeddings, outperforming the high-cost MetaEOL by a substantial margin. Finally, we highlight the potential of obtaining strong LLM embedding models through fine-tuning Value Aggregation.

Deep Learning · Everything Else

Sangwoo Hwang, Yeeun Hong, Jaeha Kung

The rapid growth of deep neural networks (DNNs) has intensified the demand for efficient hardware acceleration under quantization. While prior research has successfully reduced weight and activation precision, partial sums generated during accumulation often retain high precision, resulting in significant energy overhead. In this work, we analyze psum distributions in tiled architectures and reveal that within-tile outliers are input-dependent. We propose PsumQuant, a post-training, input-aware quantization that predicts psum scales on-the-fly. By leveraging the crest factor of input activations, our learnable scale predictor effectively bounds the psum bit-width while handling the extreme outliers in DNNs. Experimental results on a $128 \times 128$ systolic array demonstrate that PsumQuant compresses psum precision down to 8-bit within only a 1\% accuracy drop on ResNet-18 and a marginal 0.04 perplexity increase on Llama-3.1. Furthermore, bit-width reduction with PsumQuant results in a 45\% reduction in total energy with minimal accuracy loss, demonstrating that PsumQuant provides a highly efficient solution for actual NPU architectures.