论文检索

Deep Learning · Large Language Models

Xiaoxin Shi, Jiaxin Wan, Linkang Dong, Wei Jiang, Yue Liu, Zengfeng Huang

LLM-based function calling enables intelligent agents to interact with external tools and environments, yet autoregressive decoding imposes a fundamental latency bottleneck that limits real-time applications such as embodied intelligence, game AI, and interactive avatars (e.g., 10 Hz control frequency). We observe that function calling differs fundamentally from free-form text generation: structured outputs exhibit substantial token redundancy (delimiters, parameter names), and arguments exhibit weak causal dependencies. Crucially, these two properties must be exploited jointly to achieve real-time performance. We present RealtimeTool, which introduces special tokens that serve a dual role: compressing low-entropy tokens (4–6× reduction) while acting as mode selectors that enable independent parallel generation of function name and arguments. This synergistic design achieves 3–6× end-to-end speedup (up to 9.6×) with only +8.2% parallelization overhead. Experiments on five benchmarks across Qwen-series models (0.5B–14B) demonstrate substantial speedup while maintaining competitive or improved accuracy. On Mobile Actions, RT-Qwen-0.5B outperforms Google's FunctionGemma in both accuracy and latency consistency. With quantization on consumer-grade GPU, RealtimeTool achieves 61.2ms P50 latency, enabling 16 Hz real-time control at 4B model scale, bridging the gap between LLM function calling and latency-critical real-world deployment.

Deep Learning · Large Language Models

Yuli Chen, Shuhao Zhang, Jiale Han, Fanshen Meng, Haishen Jiang, Bo Cheng, Qiang Tong, Xiulei Liu

The exponential growth in the parameter scale of Large Language Models (LLMs) has precipitated an urgent demand for efficient compression techniques to facilitate practical deployment. To address this challenge, low-rank decomposition based on Singular Value Decomposition (SVD) offers a principled, hardware-friendly pathway for compressing LLMs without retraining. However, existing training-free approaches predominantly rely on uniform rank allocation, implicitly assuming homogeneous redundancy across the model depth and thereby neglecting the inherent non-uniformity of representational evolution. To bridge this gap, we introduce \textbf{CGSVD}, a \uline{\textbf{C}}ascaded \uline{\textbf{G}}ranular \uline{\textbf{S}}ingular \uline{\textbf{V}}alue \uline{\textbf{D}}ecomposition framework that leverages a dual-level non-uniform allocation strategy to maximize semantic preservation. Specifically, we quantify inter-layer significance via angular distance and assess intra-layer compressibility through spectral entropy, enabling precise identification of critical architectural components. Furthermore, we propose an Iterative Residual Filling (IRF) mechanism to bridge the parameter gap caused by integer-rank truncation and ensure strict adherence to global compression targets. Extensive experiments on representative LLM families ranging from 3B to 13B parameters verify the superiority of our approach. Notably, under a 30\% compression ratio on the LLaMA3.1-8B model, CGSVD achieves a remarkable average zero-shot accuracy boost of 6.08\% and reduces perplexity by 33.39 compared to the baseline. We release the code\footnote{The code is available at: \url{https://anonymous.4open.science/r/CGSVD-BD6E}.} to facilitate future research.

Applications · Computer Vision

Keyun Zhao, Guangchang Li, Yunpeng Bai, shao jiang, Ying Li

Hyperspectral image change detection (HSI-CD) suffers from severe annotation scarcity and complex change patterns, which fundamentally limit the effectiveness of directly fine-tuning pre-trained foundation models. Although semi-supervised learning provides a promising direction, existing approaches mainly rely on confidence-based pseudo-label selection, leading to limited data diversity or severe error propagation. In this paper, we propose Influence-Aware Semi-supervised Fine-tuning (IA-SFT), a novel framework that evaluates the influence of pseudo-labels on model decision behavior to identify truly valuable supervision signals. Instead of confidence-based selection, IA-SFT evaluates each low-confidence pseudo-label by measuring its impact on labeled data, enabling reliable filtering of high-value pseudo-labels with minimal noise. To further adapt foundation models to HSI-CD, we design an Adaptive Fusion Change Decoder (AFCD) that jointly models global semantic consistency and local change details. Extensive experiments on three benchmark datasets demonstrate that IA-SFT consistently improves pseudo-label quality and detection performance, achieving superior accuracy compared to state-of-the-art methods. Additional analyses validate the transferability of IA-SFT when integrated into different frameworks in a plug-and-play manner. Code will be released.

Applications · Health / Medicine

Noah Flynn

Designing genetic circuits, which are biological systems capable of programmed behaviors within living cells, remains a laborious, expert-driven process despite decades of progress in synthetic biology. We introduce GenCircuit-RL, a reinforcement learning framework that trains language models to reason about genetic circuit design through code generation, where models produce Python code using PySBOL to construct circuits in the standardized Synthetic Biology Open Language (SBOL) format. Our approach addresses the challenge of sparse feedback in biological design through hierarchical verification rewards that decompose correctness into five levels, from code execution through structural validity to functional behavior, providing dense learning signal while multiplicative dependencies prevent reward hacking. We contribute SynBio-Reason, a benchmark of approximately 4,753 circuits spanning six canonical circuit types and nine tasks from code repair to de novo design, with held-out biological parts enabling rigorous out-of-distribution evaluation. A four-stage curriculum progressively shifts optimization pressure from basic code generation toward functional correctness, enabling models to acquire compositional reasoning capabilities incrementally. Our framework demonstrates that hierarchical verification combined with curriculum learning enables compact language models to generate functionally correct genetic circuits, including generalization to novel biological parts and rediscovery of canonical designs from synthetic biology literature.

Applications · Computer Vision

Siwoo Lim, Sunjae Yoon, Gwanhyeong Koo, Chang D. Yoo

Gaussian Splatting has achieved significant improvements by incorporating warping-based techniques. These approaches enhance synthesis quality by warping images from source views into the target viewpoint to compensate for missing or residual pixels. However, such methods suffer from pixel-level inaccuracies due to uncertain geometry. This uncertainty leads to spatial misalignments in the warped images, which disrupt residual learning used in warping-based methods and fundamentally limit the gains of correction, particularly on thin structures and high-frequency details. Driven by our insight that useful visual cues are not lost but locally preserved under slight displacement, we propose Geometry-Aware Deformable Aggregation (GADA). This method introduces an iterative refinement module with deformable offsets to actively correct spatial misalignments and recover these displaced cues. Furthermore, to address the limitations of standard pipelines where visibility checks (i.e., thresholding) often discard valid pixels and multi-view warped image fusion relies on naive mean aggregation, our module is coupled with an implicit confidence weighting mechanism that selectively suppresses unreliable evidence. Consequently, our approach outperforms prior warping-based Gaussian Splatting, preserving high-frequency quality while achieving 2.13times faster FPS.

Applications · Computer Vision

Mamoona Javaid, Mubashir Noman, Abdul Hannan, Shah Nawaz, Mustansar Fiaz, Sajid Ghuffar

Rapid expansion of urban areas and population growth is causing an immense increase in waste production, which demands the need for efficient and automated waste management. In this scenario, automated waste recycling (AWR) that utilizes deep learning methods to separate the recyclable waste objects may emerge as a savior to humanity. Recent deep learning approaches for AWR provide promising waste segmentation performance, however, these methods rely on large backbone networks that are inefficient for AWR systems and suffer from performance deterioration in cluttered scenes. To this end, an optimal waste segmentation network is introduced which effectively utilizes the spatial domain to capture localized structural dependencies and the spectral domain to efficiently extract global contextual relationships. This cascaded design allows the network to progressively leverage both local and global representations across complementary domains to highlight the semantic information necessary for effective segmentation of various waste objects. Furthermore, auxiliary feature enhancement module (AFEM) is introduced to enhance the target objects' boundaries and blob amplification for better segmentation in cluttered scenarios. Extensive experimentation on ZeroWaste-aug, ZeroWaste-f and SpectralWaste reveals the merits of the proposed method.

Applications · Chemistry, Physics, and Earth Sciences

Orestis Oikonomou, Levi Lingsch, Dana Grund, Siddhartha Mishra, Georgios Kissas

Analytical solutions to differential equations offer exact, interpretable insight but are rarely available because discovering them requires expert intuition or exhaustive search in combinatorial spaces. We introduce SIGS, a neuro-symbolic framework that automates this process. SIGS uses a formal grammar to generate only syntactically valid building blocks, embeds these expressions into a continuous space, and then searches this space to assemble, score, and refine candidate closed-form solutions by minimizing a physics-based residual. This design unifies symbolic reasoning with numerical optimization; the grammar constrains candidate solution blocks to be proper by construction, while the latent search makes exploration tractable and data-free. SIGS is the first neuro-symbolic method to (i) analytically solve coupled systems of nonlinear PDEs, (ii) discover solutions under grammar misspecification, and (iii) produce accurate symbolic approximations for PDEs lacking known closed-form solutions. Overall, SIGS achieves orders-of-magnitude improvements in accuracy and efficiency over existing symbolic methods on standard benchmarks.

General Machine Learning · Online Learning, Active Learning and Bandits

Pascal Janetzky, Dr. Tobias Schlagenhauf, Stefan Feuerriegel

Continual learning (CL) aims to train models sequentially over multiple domains without forgetting previously learned knowledge. However, existing CL methods optimize for in-domain performance and are therefore prone to learning spurious, domain-specific cues (``shortcut learning''), which limits generalization to unseen domains after deployment. In this paper, we address this limitation through *continual learning of domain-invariant representation*. We introduce a broad class of CL methods that sequentially learn representations capturing invariant structures across domains. Our methods are motivated by the observation that such invariant structures often preserve the underlying causal mechanisms, which can reduce the risk of overfitting to domain-specific cues and thus offer better out-of-domain generalization. Our proposed CL methods combine replay-based training with a tailored sequential invariance alignment to learn---and preserve---invariant structures over time. We evaluate our methods under a deployment-oriented protocol that measures performance on unseen target domains. Across six benchmark and real-world datasets spanning vision, medicine, manufacturing, and ecology, our methods consistently outperform existing CL baselines in terms of generalization to unseen target domains. As an ablation, we further show that na\"ive extensions of sequential training with existing domain-invariant representation learning (DIRL) methods provide only limited benefits. To the best of our knowledge, this is the first work to develop domain-invariant representation methods for CL.

Applications · Computer Vision

Zheng Wang, Yanwen Wang, Tianyu Fang, Jiaxing Shen, Yisen Kang, Di Wu, Yuanqing Zheng

The quantity and 3-D shape of wear particles are essential indicators for assessing the health of jet engines, enabling early detection of potential damage and preventing accidents caused by catastrophic failures. However, capturing wear particles is difficult due to their minute sizes and ultra high-speed movement within intense jet flows. Existing technologies struggle with the extreme background noise and low resolution in such harsh environments. In this paper, we propose QuantWear, the first quantum sensing system designed to directly quantify and profile wear particles on the sub-millimeter scale. QuantWear innovatively tracks wear particles by monitoring the spectral signatures of Sodium (Na) and Potassium (K) atoms within jet flow, which naturally adhere to particle surfaces due to electrochemical reactions in high-temperature combustion. We construct a custom atomic detector that leverages quantum jump and Faraday rotation effects to isolate these specific atomic signals, effectively suppressing the broad-spectrum flame noise. Next, we apply a deep learning framework to effectively measure the quantity of wear particles in dynamic vaporous backgrounds. Finally, we generate a fully reconstructed 3-D model of the wear particles from multiple 2-D images. Extensive field tests and high-fidelity simulations demonstrate that QuantWear achieves an imaging Signal-to-Noise Ratio (SNR) of 22.5 dB and a 3-D reconstruction similarity exceeding 95%, significantly outperforming state-of-the-art technologies.

Deep Learning · Large Language Models

Hippolyte Pilchen, Romain Fabre, Franck SIGNE TALLA, Patrick Perez, Edouard Grave

Large language models (LLMs) are typically trained on shuffled corpora, yielding models whose knowledge is frozen at training time and whose temporal grounding remains poorly understood. In this work, we study the impact of pretraining dynamics on the acquisition of time-sensitive factual knowledge, focusing specifically on data ordering. Our main contributions are twofold. First, we introduce a comprehensive benchmark of over 7,000 temporally grounded questions and an evaluation protocol that enables analysis of whether models correctly associate facts with their corresponding time periods. Second, we pretrain 6B-parameter language models on temporally ordered Common Crawl snapshots and compare them against standard shuffled pretraining. Our results show that sequentially trained models match shuffled baselines on general language understanding and common knowledge while consistently exhibiting more up-to-date and temporally precise knowledge. Temporally ordered pretraining yields improved factual freshness, while shuffled pretraining peaks on older data, possibly due to increased factual repetition. These findings, along with the release of our checkpoints and datasets, provide a foundation for future research on continual learning for large language models.

Applications · Language, Speech and Dialog

Tom Labiausse, Romain Fabre, Yannick Estève, Alexandre Défossez, Neil Zeghidour

Simultaneous speech translation is the task of translating source speech into a target language in real-time. Given that the dependencies between source and target words are non-monotonic (e.g. the word order can change between German and English), this means learning to jointly align and translate. This task has been traditionally tackled through supervised training on aligned data, and as collecting such data is challenging, this relies on synthetic data with automatic alignment. The latter relies on heuristics that are language-specific and suboptimal. We instead propose Hibiki-Zero, a model for simultaneous speech translation trained without word-level alignments between source and target speech. To do so, we train on sentence-level aligned data so that the model learns to perform speech translation but with high latency. We then introduce a novel reinforcement learning strategy relying on GRPO to optimize the translation latency of the model while retaining its translation capabilities. After supervised and post-training, Hibiki-Zero performs multilingual simultaneous translation with state-of-the-art translation accuracy, latency, voice transfer and naturalness across five X-to-English tasks. Moreover, we demonstrate that our model can be easily finetuned to support another language as input with less than 1000h of speech data. We provide examples ([hibiki-zero-s2st.github.io](https://hibiki-zero-s2st.github.io)) as well as models and release a benchmark containing 15h of multilingual data for speech translation evaluation.

Applications · Computer Vision

Saman Forouzandeh, Wei Peng, Xinghuo Yu, Mahdi Jalili

VideoQA models can be accurate yet often fail to align answers with the correct video segments (the \emph{grounding gap}). We introduce \textbf{LINGUA} (\textbf{L}anguage-based \textbf{IN}ference for \textbf{G}rounded Video \textbf{U}nderstanding \textbf{A}gent), a memory-based agent that performs grounded VideoQA by reasoning in an explicit \emph{linguistic belief state}. LINGUA uses five mechanisms: (1) event-driven perception (retains 8--12\% of frames while preserving 94\% of question-relevant events); (2) typed memory for episodic narratives, semantic affordances, and procedural scripts; (3) Belief-Action-Verification loops with postcondition and temporal checks; (4) meta reflection with contrastive refinement; and (5) Bayesian reliability tracking for continual learning without gradient updates. Built with Gemma3-4B (Ollama, 4-bit), LINGUA outperforms strong baselines on five VideoQA benchmarks, reaching 82.4\% on NExT-QA and 42.3\% Acc@GQA on NExT-GQA (answer + IoU$\geq$0.5 temporal localization), while running 2.6$\times$ faster than dense-frame methods. In continual learning over 100 videos, accuracy rises from 45.2\% (first 10) to 61.8\% (last 10) without catastrophic forgetting, indicating online adaptation via memory refinement.

Tom Labiausse, Romain Fabre, Yannick Estève, Alexandre Défossez, Neil Zeghidour

Simultaneous speech translation is the task of translating source speech into a target language in real-time. Given that the dependencies between source and target words are non-monotonic (e.g. the word order can change between German and English), this means learning to jointly align and translate. This task has been traditionally tackled through supervised training on aligned data, and as collecting such data is challenging, this relies on synthetic data with automatic alignment. The latter relies on heuristics that are language-specific and suboptimal. We instead propose Hibiki-Zero, a model for simultaneous speech translation trained without word-level alignments between source and target speech. To do so, we train on sentence-level aligned data so that the model learns to perform speech translation but with high latency. We then introduce a novel reinforcement learning strategy relying on GRPO to optimize the translation latency of the model while retaining its translation capabilities. After supervised and post-training, Hibiki-Zero performs multilingual simultaneous translation with state-of-the-art translation accuracy, latency, voice transfer and naturalness across five X-to-English tasks. Moreover, we demonstrate that our model can be easily finetuned to support another language as input with less than 1000h of speech data. We provide examples ([hibiki-zero-s2st.github.io](https://hibiki-zero-s2st.github.io)) as well as models and release a benchmark containing 15h of multilingual data for speech translation evaluation.

Applications · Computer Vision

Xinge Peng, Yiting Lu, Xin Li, Zhibo Chen

We present IQA-Spider, the first image quality assessment (IQA) framework that unifies reasoning, grounding, and referring within a LMM-based system for multi-granularity quality understanding. Existing LMM-based IQA methods typically support only partial perception dimensions, \egno, quality description and question answering~(\ieno, reasoning) or pixel-level grounding, largely due to the absence of (i) a unified task-and-data formulation and (ii) effective optimization paradigms for multi-granularity learning. To address these limitations, we formulate a rigorous four-task paradigm covering global and local quality description, pixel-level grounding, and region-level referring. Based on this formulation, we construct a corresponding IQA dataset with a scalable and automatic annotation pipeline, thereby providing a solid foundation for unified multi-granularity learning. To further enable unified perception, we adopts a conflict-free two-stage design that progressively extends textual multi-granularity understanding to pixel-level grounding: (i) the first stage equips the model with fine-grained textual reasoning across multiple IQA tasks, and (ii) the second stage introduces a training-free text-to-point grounding paradigm, which bridges textual semantics and pixel-level perception by mapping token logits to spatial coordinates. Based on these efforts, we achieve IQA-Spider with unified multi-granularity explainable image quality assessment. Extensive experiments across multiple benchmarks demonstrate strong performance, validating the effectiveness and versatility of the proposed formulation and framework. Code and datasets will be released upon acceptance.

Applications · Robotics

Wenhao Li, Xiu Su, Dan Niu, Yichao Cao, Hongyan Xu, Zhe Qu, Lei Fan, Shan You, Chang Xu

Vision-language-action (VLA) models have advanced the field of embodied manipulation by harnessing broad world knowledge and strong generalization. However, current VLA models still face several key challenges, including limited reasoning capability, lack of status monitoring, and difficulty in self-correction. In this paper, we introduce \textbf{Sentinel-VLA}, a metacognitive VLA model equipped with an active ``sentinel'' module to monitor real-time execution status. Only when necessary, such as during initial planning or upon detecting an error, the model triggers a dynamic reasoning or formulate error recovery solutions. This on-demand reasoning mechanism ensures robust decision-making while minimizing computational overhead. Notably, all training data (spanning 44 tasks and over 2.6 million transitions) is automatically generated and annotated through our designed pipeline. We also propose the Self-Evolving Continual Learning (SECL) algorithm, which allows Sentinel-VLA to identify its capability boundaries and automatically collect data for expansion, paired with Orthogonal Continual Adapter (OC-Adapter) to constrain parameter updates to an orthogonal space, thereby preventing catastrophic forgetting. Real-world experiments demonstrate that Sentinel-VLA boosts the task success rate by over 30\% compared to the SOTA model, PI0. We will open-source all the code, weights, and data generation pipeline.

Reinforcement Learning · Batch/Offline

Thanh Nguyen, Tri Ton, Hongbin Choe, Minh-Tung Luu, Chang D. Yoo

Diffusion-based Q-learning has emerged as a powerful paradigm for offline reinforcement learning, but its reliance on multi-step denoising makes both training and inference computationally expensive and brittle. Recent efforts to accelerate diffusion Q-learning toward single-step action generation typically introduce auxiliary networks, policy distillation, or multi-phase training, which frequently compromise simplicity, stability, or performance. To address these limitations, we introduce Bootstrapped Flow Q-Learning (BFQ), a novel framework that enables accurate single-step action generation during both training and inference—without auxiliary networks or distillation procedures. BFQ adopts a divide-and-conquer view of the displacement vector along the flow path: it begins by learning short-range displacements that can be accurately estimated from the Flow Matching marginal velocity, and bootstraps these components to directly learn a noise-to-action mapping in a single step. This formulation eliminates multi-step denoising, resulting in a learning procedure that is substantially faster, simpler, and more robust. Extensive D4RL evaluations show that BFQ improves performance while significantly reducing computational cost compared to multi-step diffusion baselines, demonstrating that single-step action generation suffices for high-performance offline Reinforcement Learning.

Deep Learning · Large Language Models

Shanshan Zhong, Kate Shen, Chenyan Xiong

Agentic Web is an emerging paradigm where autonomous agents help users use online information. As the paradigm develops, content providers are also deploying agents to manage their data and serve it through controlled interfaces. This shift moves information access from centralized retrieval to decentralized coordination. To study this setting, we introduce AgentWebBench, a benchmark that evaluates how well a user agent synthesizes answers by interacting with website-specific content agents. We evaluate four tasks that cover common web information needs, spanning ranked retrieval (web search, web recommendation) and open-ended synthesis (question answering, deep research). Across seven advanced LLMs and three coordination strategies, multi-agent coordination generally lags behind centralized retrieval as expected, because user agent cannot directly access the corpus, but the gap shrinks with model scale and can even outperform centralized retrieval on question answering. This benchmark also enables us to study properties of the emerging paradigm of the digital world. We find that decentralized access concentrates traffic toward a small set of websites, test time scaling improves both interaction reliability and task performance, and strong results require sufficient interactions guided by careful planning. Finally, our failure analysis suggests that user agents need better planning and answer synthesis, while content agents need more reliable retrieval and evidence quality. Code, data, and APIs will be released online.

Applications · Computer Vision

Sanghyuk Chun, Olga Russakovsky

Multimodal learning has seen remarkable progress, particularly with large-scale pre-training across various modalities. Most current approaches are built on the assumption of a deterministic one-to-one alignment between modalities. However, this oversimplifies real-world multimodal relationships, where their nature is inherently many-to-many. The many-to-many property, or \emph{multiplicity}, is not a side-effect of noise or annotation error, but an inevitable outcome of intra-modal variability, representational asymmetry, and task-dependent ambiguity in multimodal tasks. We argue that multiplicity is a fundamental bottleneck that affects all stages of the multimodal learning pipeline: from data construction to model training and evaluation benchmarks. By formalizing its causes and consequences, we demonstrate how ignoring multiplicity leads to training uncertainty, unreliable evaluation, and degraded dataset quality. This position paper calls for new research directions on multimodal learning, including multiplicity-aware learning frameworks and dataset construction and evaluation protocols.

Theory · Reinforcement Learning and Planning

Runze Zhao, Yue Yu, Ruhan Wang, Chunfeng Huang, Dongruo Zhou

Continuous-time reinforcement learning (CTRL) provides a natural framework for sequential decision-making in dynamic environments where interactions evolve continuously over time. While CTRL has shown growing empirical success, its ability to adapt to varying levels of problem difficulty remains poorly understood. In this work, we investigate the instance-dependent behavior of CTRL and introduce a simple, model-based algorithm built on maximum likelihood estimation (MLE) with a general function approximator. Unlike existing approaches that estimate system dynamics directly, our method estimates the state marginal density to guide learning. We establish instance-dependent performance guarantees by deriving a regret bound that scales with the total reward variance and measurement resolution. Notably, the regret becomes independent of the specific measurement strategy when the observation frequency adapts appropriately to the problem’s complexity. To further improve performance, our algorithm incorporates a randomized measurement schedule that enhances sample efficiency without increasing measurement cost. These results highlight a new direction for designing CTRL algorithms that automatically adjust their learning behavior based on the underlying difficulty of the environment.

Social Aspects · Alignment

Avinash Reddy, Thayne Walker, Jaime Ide, Amrit Singh Bedi

Large language models (LLMs) are increasingly used to generate executable outputs, JSON objects, and API calls, where a single syntax error can make the output unusable. Constrained decoding enforces validity token-by-token via masking and renormalization, but it can distort generation when the model assigns low probability mass to valid continuations, pushing decoding toward locally valid yet semantically incorrect trajectories. We propose \emph{Draft-Conditioned Constrained Decoding (DCCD)}, a simple two-step, training-free inference procedure that decouples semantic planning from structural enforcement: an unconstrained draft is generated first, and constrained decoding is then applied, conditioned on this draft, to guarantee validity. We analyze DCCD through a KL-projection view, showing that draft conditioning increases feasible mass and reduces the cumulative “projection tax” induced by hard constraints, with an optional best-of-$K$ draft selection. Across structured reasoning benchmarks, DCCD improves strict structured accuracy by up to +24 percentage points over standard constrained decoding (e.g., 15.2\% to 39.0\% on GSM8K with a 1B model), and enables smaller model pairs to match or exceed much larger constrained baselines, yielding substantial gains in parameter efficiency.