论文检索

Applications · Language, Speech and Dialog

Yuwei Zhang, Chengyu Dong, Shuowei Jin, Changlong Yu, Hejie Cui, Hongye Jin, Xinyang Zhang, Hamed Bonab, Colin Lockard, Jianshu Chen 等

Context management enables agentic models to solve long-horizon tasks through iterative summarization of previous interaction histories. However, this process typically incurs substantial decoding overhead for the extra summarization tokens, which significantly affect the end-to-end response latency at deployment. In this paper, we introduce CoMem, a novel framework that decouples memory management from the primary agent workflow, enabling these processes to execute in parallel. We propose a $k$-step-off asynchronous pipeline that overlaps the memory model's summarization with the agent's inference, effectively masking the latency of context processing. To ensure robustness under this asynchronous setting, we introduce a reward-driven training strategy that aligns the memory model to capture sufficient statistics for the agent's decision-making. Theoretical analysis confirms that CoMem offers a superior efficiency-effectiveness trade-off compared to coupled architectures. Our extensive experimental results on SWE-Bench-Verified show that CoMem provides 1.4x latency improvements upon vanilla long-context solutions while preserving most of the performance. Furthermore, we demonstrate that these latency gains scale favorably with increased system throughput, offering a modular path forward for the independent optimization of agent reasoning and memory compression.

Applications · Language, Speech and Dialog

Sanidhya Vijayvargiya, Vijay Viswanathan, Graham Neubig

Humans do not always express what they need perfectly. Helpful assistants must be able to ask for clarification to handle real-world user commands which may be underspecified or poorly stated. Yet, optimal clarification remains challenging. The space of possible tasks is large, and not all missing information is equally valuable. We study real software engineering tasks. These tasks require the user to provide many details, some of which are more important than others, and an accurate assistant should be able to determine which clarification questions to ask when necessary information is missing. We systematically quantify which types of information in prompts types most impact task success and what types of clarification questions enable productive answers by simulated users. Using Shapley attribution and distributional comparisons, we identify two learnable properties of effective clarification: task relevance (which information impacts success) and user answerability (what users can realistically provide). We operationalize these as multi-stage reinforcement learning rewards to train an 8B parameter module matching GPT-5's performance while generating 41\% fewer questions. Our work demonstrates that effective clarification emerges from grounding reward design in empirical analysis of information impact and user answerability, establishing a generalizable reward framework applicable across interactive task domains.

Applications · Language, Speech and Dialog

Ui-Hyeop Shin, Jaehyun Ko, Woocheol Jeong, Hyung-Min Park

Speech restoration in real-world conditions is challenging due to compounded distortions and mismatches between input and desired output rates. Most existing systems assume a fixed and shared input–output rate, relying on external resampling that incurs redundant computation and limits generality. We address this setting by formulating speech restoration under decoupled input–output rates, and propose TF-Restormer, a query-based asymmetric modeling framework. The encoder concentrates analysis on the observed input bandwidth using a time–frequency dual-path architecture, while a lightweight decoder reconstructs missing spectral content via frequency extension queries. This design enables a single model to operate consistently across arbitrary input–output rate pairs without redundant resampling. Experiments across diverse sampling rates, degradations, and operating modes show that TF-Restormer maintains stable restoration behavior and balanced perceptual quality, including in real-time streaming scenarios.

Applications · Language, Speech and Dialog

Xiaoyu Yang, Yifan Yang, Zengrui Jin, Ziyun Cui, Wen Wu, Baoxiangli, Chao Zhang, Phil Woodland

Self-supervised learning (SSL) has significantly advanced acoustic representation learning. However, most existing models are optimised for either speech or audio event understanding, resulting in a persistent gap between these two domains. We address this gap with SPEAR (SPEech and Audio Representations), a self-supervised framework that distils complementary knowledge from a speech-focused SSL teacher and a general-audio SSL teacher into a single unified model. SPEAR applies multi-codebook vector quantisation to continuous teacher representations to produce fine-grained discrete tokens that capture both semantic and acoustic information. To effectively integrate these heterogeneous representations, SPEAR jointly predicts them given a masked input with an asymmetric pre-training loss. We further improve robustness in complex sound scenes through a novel token mixing mechanism. Extensive experiments demonstrate that SPEAR consistently outperforms existing unified speech and audio models. SPEAR establishes a new state-of-the-art on the SUPERB benchmark, surpassing WavLM Large on 12 of 15 tasks, while achieving competitive performance on the HEAR benchmark. These results position SPEAR as a versatile foundation for general-purpose speech and audio representation learning. The code and pre-trained models will be released.

Applications · Language, Speech and Dialog

Yixian Shen, Zhiheng Yang, Qi Bi, Changshuo Wang, JIA-HONG HUANG, Shuai Wang, Prayag Tiwari, George Floros, Anuj Pathania

Multimodal reasoning often relies on long chains of intermediate textual and visual thoughts, where accumulating visual tokens and dense cross-modal attention incur substantial computation and memory overhead. To address this challenge, we propose Spectral-Progressive Thought Flow (*SpecFlow*), a *novel* lightweight multimodal reasoning framework that represents intermediate visual thoughts in a fixed-size discrete cosine space. By exploiting strong energy compaction, *SpecFlow* preserves global layout and relational structure while introducing high-frequency details only when increased spatial precision is required. To align visual state evolution with linguistic intent, classifier-free guidance enables autoregressive textual thoughts to steer flow-based updates of the visual workspace without expanding the context. As a result,*SpecFlow* maintains a bounded visual workspace whose updates depend only on the current visual state and accumulated textual trace, enabling long-horizon inference with stable latency and memory usage independent of reasoning depth. Empirical results show that *SpecFlow* achieves competitive or superior reasoning performance while reducing computation and memory costs by up to *$2.1\times$*.

Applications · Language, Speech and Dialog

Haoran Luo, Haihong E, Guanting Chen, Qika Lin, Yikai Guo, Fangzhi Xu, Zemin Kuang, Meina Song, Xiaobao Wu, Yifan Zhu 等

Retrieval-Augmented Generation (RAG) mitigates hallucination in LLMs by incorporating external knowledge, but relies on chunk-based retrieval that lacks structural semantics. GraphRAG methods improve RAG by modeling knowledge as entity-relation graphs, but still face challenges in high construction cost, fixed one-time retrieval, and reliance on long-context reasoning and prompt design. To address these challenges, we propose Graph-R1, an agentic GraphRAG framework via end-to-end reinforcement learning (RL). It introduces lightweight knowledge hypergraph construction, models retrieval as a multi-turn agent-environment interaction, and optimizes the agent process via an end-to-end reward mechanism. Experiments on standard RAG datasets show that Graph-R1 outperforms traditional GraphRAG and RL-enhanced RAG methods in reasoning accuracy, retrieval efficiency, and generation quality. Our code is available.

Applications · Language, Speech and Dialog

Yangbo Wei, Zhen Huang, Shaoqiang Lu, Junhong Qian, Chen Wu, Lei He

In open-ended domains, natural language instructions are often *underspecified*, mapping to multiple valid yet functionally distinct latent intents. While Large Language Models (LLMs) excel at generation, their ability to resolve such *task ambiguity* through interaction is currently hampered by *semantic blindness*—a tendency to squander interaction budgets on distinguishing trivial syntactic variants rather than fundamental intent differences. To address this, we propose *Topological Active Inference (TAI)*, a geometric framework that recasts disambiguation as a process of *intent-manifold contraction*. TAI first leverages *Persistent Homology* to recover the topological skeleton of the solution space, theoretically guaranteeing the separation of semantic signal from syntactic noise. Subsequently, it synthesizes clarifying questions as *separating hyperplanes* designed to efficiently bisect the probability mass of the intent manifold. We introduce *Topological Expected Information Gain (TEIG)* for question selection and prove that maximizing TEIG reduces query complexity from linear $\mathcal{O}(N)$ to logarithmic $\mathcal{O}(\log K)$, where $K$ is the number of latent intents. Extensive experiments demonstrate that TAI recovers user intent with significantly fewer turns, achieving state-of-the-art disambiguation efficiency.

Applications · Language, Speech and Dialog

Artem Dementyev, Wazeer Zulfikar, Sinan Hersek, Pascal Getreuer, Anurag Kumar, Vivek Kumar

Current multimodal LLMs process audio as a mono stream, ignoring the rich spatial information essential for embodied AI. Existing spatial audio models, conversely, are constrained to fixed microphone geometries, preventing deployment across diverse devices. We present PhaseCoder, a transformer-only spatial audio encoder that is agnostic to microphone geometry. PhaseCoder takes raw multichannel audio and microphone coordinates as inputs to perform localization and produces robust spatial embeddings. We integrate PhaseCoder with the Gemma 3n LLM by finetuning it to reason over ``spatial audio tokens''. We show our encoder achieves state-of-the-art results on microphone-invariant localization benchmarks and, for the first time, enables an LLM to perform complex spatial reasoning and targeted transcription tasks from an arbitrary microphone array.

Applications · Language, Speech and Dialog

Bin Kang, Shaoguo Wen, Yang Fan, Shunlong Wu, Junjie Wang, Yulin Li, Junzhi Zhao, Junle Wang, Zhuotao Tian

While existing text-to-speech (TTS) models exhibit high expressiveness, fine-grained control over composite instructions remains challenging due to the structural mismatch between discrete textual intents and continuous acoustic realizations. Inspired by human cognitive decoupling, we introduce AgentSteerTTS, a multi-agent closed-loop framework designed for intent-faithful expressive control of composite instructions. First, in our framework, an adversarial disentanglement module mitigates speaker-emotion leakage through gradient reversal and cross-covariance regularization. Next, a Dual-Stream Anchoring Controller grounds abstract intents using a large-scale acoustic prototype library: a Retrieval Agent selects expressive anchors, while a Synthesis Agent fuses them into continuous control vectors via gated attention. Finally, a Fast–Slow Feedback Agent refines output intensity through latent gradient correction and resolves semantic–acoustic mismatches using high-level perceptual critique. Experiments on a composite-instruction benchmark and public test sets show that AgentSteerTTS yields consistent and significant improvements to the baselines, demonstrating the effectiveness of the proposed method. Our code and models will be released.

Applications · Language, Speech and Dialog

Yifan Yang, Hui Wang, Bing Han, Shujie Liu, Jinyu Li, Yong Qin, Xie Chen

Recent advances in text-to-speech (TTS) technology have enabled systems to generate speech that is often indistinguishable from human speech, bringing benefits to accessibility, content creation, and human-computer interaction. However, current evaluation practices are increasingly inadequate for capturing the full range of capabilities, limitations, and societal impacts of modern TTS systems. This position paper introduces the concept of Responsible Evaluation and argues that it is essential and urgent for the next phase of TTS development, structured through three progressive levels: (1) ensuring the faithful and accurate reflection of a model's true capabilities and limitations, with more robust, discriminative, and comprehensive objective and subjective scoring methodologies; (2) enabling comparability, standardization, and transferability through standardized benchmarks, transparent reporting, and transferable evaluation metrics; and (3) assessing and mitigating ethical risks associated with forgery, misuse, privacy violations, and security vulnerabilities. Through this concept, we critically examine current evaluation practices, identify systemic shortcomings, and propose actionable recommendations. We hope this concept will not only foster more reliable and trustworthy TTS technology but also guide its development toward ethically sound and societally beneficial applications.

Applications · Language, Speech and Dialog

Ke Lei, Yu Zhang, Changhao Pan, Xueyi Pu, Wenxiang Guo, Ruiqi Li, Zhou Zhao

Real-time and accurate spatial audio generation is pivotal for delivering an immersive experience. However, existing spatial audio synthesis technologies are often encumbered by a tradeoff between generation quality and high inference latency, as well as difficulty in capturing precise spatial information from multimodal inputs. To address these challenges, we propose S3Audio, a unified streaming framework for high-fidelity spatial audio generation from panoramic videos and text prompts. S3Audio mainly makes the following contributions: 1) We introduce a causal autoregressive diffusion transformer architecture that enables streaming high-quality spatial audio generation. 2) We design a Spatial Video–Audio Contrastive (SVAC) learning strategy to align the video encoder with the acoustic domain, and further employ a multi-objective online direct preference optimization~(ODPO) scheme, resulting in strong spatial perception and robust multimodal spatial audio synthesis. 3) To alleviate the current scarcity of spatial audio datasets, we also develop an automated annotation pipeline for generating detailed spatial captions. Experimental results demonstrate that S3Audio achieves superior performance in both video-to-spatial and text-to-spatial audio generation tasks. Demos can be found at: \url{https://s3audio.github.io}

Applications · Language, Speech and Dialog

Tae Soo Kim, Yoonjoo Lee, Jaesang Yu, John Chung, Juho Kim

To handle ambiguous and open-ended requests, Large Language Models (LLMs) are increasingly trained to interact with users to surface intents they have not yet expressed (e.g., ask clarification questions). However, users are often ambiguous because they have not yet formed their intents: they must observe and explore outcomes to discover what they want. Simply asking "what kind of tone do you want?" fails when users themselves do not know. We introduce DiscoverLLM, a novel and generalizable framework that trains LLMs to help users form and discover their intents. Central to our approach is a novel user simulator that models cognitive state with a hierarchy of intents that progressively concretize as the model surfaces relevant options---where the degree of concretization serves as a reward signal that models can be trained to optimize. Resulting models learn to collaborate with users by adaptively diverging (i.e., explore options) when intents are unclear, and converging (i.e., refine and implement) when intents concretize. Across proposed interactive benchmarks in creative writing, technical writing, and SVG drawing, DiscoverLLM achieves over 10% higher task performance while reducing conversation length by up to 40%. In a user study with 75 human participants, DiscoverLLM improved conversation satisfaction and efficiency compared to baselines.

Applications · Language, Speech and Dialog

Haolin Yang, Jipeng Zhang, Zhitao He, Alexander Zhou, Yi Fung

Large Language Models (LLMs) often struggle with the precise logic and schema alignment required for complex Text-to-SQL tasks. While current methods rely heavily on static prompting, they lack the ability to dynamically adapt and self-correct through environmental interaction. To bridge this gap, we propose **MARS-SQL**, a multi-agent architecture that leverages interactive Reinforcement Learning (RL) to optimize SQL generation. Unlike monolithic approaches, our method decomposes the problem into three specialized roles: schema linking, query generation, and solution validation. Central to our approach is a generation agent trained via a multi-turn RL policy, which operates within a ReAct-style loop. This agent learns to iteratively reason, execute intermediate SQL actions on a live database, and refine its strategy based on execution feedback. To ensure robustness, we introduce a validation mechanism that treats solution selection as a generative modeling task, identifying the optimal interaction trajectory through next-token prediction probabilities. Empirical evaluations demonstrate the effectiveness of coupling interactive learning with trajectory ranking. **MARS-SQL** achieves state-of-the-art performance, recording an execution accuracy of 77.84\% on the BIRD development dataset and 89.75\% on the Spider test dataset.

Applications · Language, Speech and Dialog

Zhenliang Zhang, Wenqing Wang, Yong Hu, Yaming Yang, Jiaheng Gao, Chen Shen, Xiaojun Wan

Long-Text Understanding (LTU) at million-token scale requires balancing reasoning fidelity with computational efficiency. Frontier long-context LLMs can process millions of token contexts end-to-end, but they suffer from high token consumption and attention dilution. In parallel, specialized LTU agents often sacrifice fidelity through task-agnostic abstractions like graph construction or indexing. We identify a key insight for LTU: query-relevant information is typically sparse relative to the full document, so effective reasoning should rely on a query-sufficient subset rather than the entire context. To address this, we propose SCOUT, a new paradigm for LTU that **shifts from passive processing to active information foraging**. It treats the document as an explorable environment and answers from a compact, provenance-grounded epistemic state. Guided by state-level gap diagnosis, SCOUT adaptively alternates between coarse-to-fine exploration and anchored state updates that progressively *contract* its epistemic state toward query sufficiency. Experiments show that SCOUT matches state-of-the-art proprietary models while reducing token consumption by up to 8 times. Moreover, SCOUT remains stable as context length scales, substantially alleviating the practical cost--capability trade-off in long-context reasoning. **Code.** An anonymous repository link is provided in the Supplementary Material.

Applications · Language, Speech and Dialog

Yiqing Xie, Emmy Liu, Gaokai Zhang, Nachiket Kotalwar, Shubham Gandhi, Acharya, Xingyao Wang, Carolyn Rose, Graham Neubig, Daniel Fried

Coding agents are increasingly used for a wide range of real-world tasks, from adding features and documentation to creating programs from scratch. Ideally, the agent should perform well across all the diverse tasks. However, most prior work concentrates on issue solving, and such single-task training does not transfer reliably to other coding tasks. In this work, we aim to train coding agents that generalize across tasks. We first analyze task transferability from two axes: the commonalities shared among coding tasks and the capabilities of current agents. Guided by these findings, we derive a set of principles for training task design and verify them through a series of controlled experiments. We then present Hybrid-Gym, a training dataset built on four scalable training tasks that follow these principles. Experiments show that, under zero-shot task transfer, Hybrid-Gym achieves performance comparable to in-domain training, and further improves existing datasets when combined with them (e.g., +4.85% on SWT-Bench-Verified).

Applications · Language, Speech and Dialog

Xiang Li, Pin-Yu Chen, Wenqi Wei

Recent advances in audio deepfake detection have been driven by increasingly large speech foundation models and growing amounts of synthetic data. Despite steady improvements on different benchmarks, it remains unclear how detection performance scales with model capacity and training data under realistic deployment conditions, where detectors operate under distribution shift, signal corruption, and unseen synthesis pipelines. In this work, we present the first systematic study of scaling laws in post-training audio deepfake detection, focusing on fine-tuning regimes rather than large-scale pretraining. Using a controlled family of speech foundation models with shared architecture and pretraining, we analyze how detection performance, robustness, and generalization evolve as a function of model size and training data scale. Our evaluation covers multiple dimensions, including out-of-distribution datasets, common audio corruptions, cross-language generalization, and cross-TTS (Text-to-Speech) generalization to unseen speech synthesis systems. Across settings, we observe consistent but highly non-uniform scaling behavior: while larger models are more sample-efficient and generalize better overall, scaling benefits weaken under corruptions and linguistic shift, and persistent error gaps remain even at the largest scales. Our results reveal a fundamental asymmetry between performance scaling and robustness scaling in audio deepfake detection. While larger detectors consistently improve in-distribution detection performance, gains in robustness and generalization, particularly under cross-language and cross-TTS evaluation, are substantially weaker and exhibit persistent error gaps.

Applications · Language, Speech and Dialog

Yaochen Han, Ke Fan, Hongxu Jiang, Wanqi Xu, Weiyu Xie, Runhua Zhang, Chenhui Zhu, Yixiang Zhang

High-performance GPU kernels are critical for reducing the exponentially growing computational costs of large language models (LLMs), but their development heavily relies on manual tuning by domain experts. While recent advances in LLM-based approaches show promise for automating kernel generation, they still struggle to achieve both correctness and high performance. This limitation primarily arises from the lack of domain-specific optimization guidance, hindering effective exploration of the optimization space. We propose $\textbf{EGG}$, an $\underline{E}$xpert-$\underline{G}$uided Agent Framework for Kernel $\underline{G}$eneration, which incorporates expert optimization principles to guide LLMs’ decisions. Inspired by expert workflows, we decompose kernel generation into two hierarchical stages: 1) algorithmic structure design, which establishes a high-quality computational structure foundation; 2) hardware-specific tuning, which performs targeted adjustments through parallel mapping, tensor tiling, and memory optimization. This staged decomposition defines explicit optimization objectives, structuring the design space to achieve progressive refinements. To this end, a stage-aware multi-agent collaboration mechanism is designed for inter and intra-stage context management, ensuring stable optimization trajectories. Experiments on KernelBench and real-world workloads show that EGG achieves a $2.13\times$ average speedup over PyTorch, outperforming existing agent-based and RL-based approaches.

Applications · Language, Speech and Dialog

Eitan Wagner, Elisha Rosensweig, Omri Abend

Recent advances in large language models and their widespread adoption have prompted claims that natural language could entirely replace formal languages, such as programming languages, for software design. In this position paper, we argue that this perspective overlooks fundamental linguistic properties of natural language, specifically that it is optimized for underspecification in open-ended contexts. We introduce a formal framework centered on *task specificity*, defining it as the information-theoretic reduction of uncertainty—in an output space, such as all possible images—given a user's specific requirements. We prove a *specificity crossover theorem*, showing the existence of a threshold beyond which the cost to express formal requirements into natural language exceeds the cost of direct formal specification. By analyzing case studies across modalities, such as image generation, code synthesis, and audio production, we demonstrate that natural language excels at low specificity tasks, while formal languages are advantageous on tasks with stricter requirements. We conclude that natural and formal languages are complementary tools and advocate the development of hybrid systems that allow users to move across the specificity spectrum.

Applications · Language, Speech and Dialog

Chenyang Shao, Jiahe Liu, Fengli Xu, Yong Li

Scientific illustration figures are essential for depicting research works' conceptual designs, methodology, and experimental workflows, playing a pivotal role in communicating complex academic insights. However, creating high-quality scientific illustrations remains a labor-intensive task for human scientists. While recent generative image models have advanced prompt-based editing, the synthesis of fully **editable** figures remains a fundamental challenge. Valid editability involves structured transformations of graphical elements, scales, attributes, and text, rather than simple pixel-level changes. Existing models generate raster outputs that do not support manual correction or layout adjustment, limiting their utility in scientific publishing, where editable vector figures are typically required for submission. To address this challenge, we introduce **LiveFigure**, an agentic framework driven by VLM agents that imitates the multi-step drawing workflow of human researchers. It first plans figure blueprints by drawing inspiration from high-quality references in previous works, then generates executable scripts that produce figures via the PowerPoint interface based on skills and experience, and finally refines the outputs with targeted visual diagnostics, producing fully vectorized, editable figures that meet publication standards. Extensive experiments demonstrate that LiveFigure generates inherently editable figures that are both visually clear and aesthetically appealing, achieving 80% publication-readiness within just 17 manual edits—far surpassing the 24% rate of the strongest baseline, NanoBanna. Human preference studies further validate this advantage, with LiveFigure securing a 60% win rate against NanoBanna. Our code is available at [https://anonymous.4open.science/r/LiveFigure](https://anonymous.4open.science/r/LiveFigure).

Applications · Language, Speech and Dialog

Marcus Min, Deyuan Mike He, Zhaoyu Li, Zixuan Yi, Sharad Malik, Aarti Gupta, Xujie Si, Osbert Bastani

Autoformalization, translating informal natural language into formal, machine-verifiable languages, has been framed as a tool to generate training data for neural theorem provers, with most work focusing on individual statements. This position paper argues for theory-level autoformalization: formalizing complete theories, including axioms, definitions, theorems, proofs, tactics, and their inter-dependencies as structured libraries. We examine the significance of this shift, address 3 alternative views, identify 5 open challenges, and propose 3 promising paths forward.