论文检索

Applications · Language, Speech and Dialog

Yixian Shen, Zhiheng Yang, Qi Bi, Changshuo Wang, JIA-HONG HUANG, Shuai Wang, Prayag Tiwari, George Floros, Anuj Pathania

Multimodal reasoning often relies on long chains of intermediate textual and visual thoughts, where accumulating visual tokens and dense cross-modal attention incur substantial computation and memory overhead. To address this challenge, we propose Spectral-Progressive Thought Flow (*SpecFlow*), a *novel* lightweight multimodal reasoning framework that represents intermediate visual thoughts in a fixed-size discrete cosine space. By exploiting strong energy compaction, *SpecFlow* preserves global layout and relational structure while introducing high-frequency details only when increased spatial precision is required. To align visual state evolution with linguistic intent, classifier-free guidance enables autoregressive textual thoughts to steer flow-based updates of the visual workspace without expanding the context. As a result,*SpecFlow* maintains a bounded visual workspace whose updates depend only on the current visual state and accumulated textual trace, enabling long-horizon inference with stable latency and memory usage independent of reasoning depth. Empirical results show that *SpecFlow* achieves competitive or superior reasoning performance while reducing computation and memory costs by up to *$2.1\times$*.

Applications · Neuroscience, Cognitive Science

Fenghao Liu, Yipeng Shen, Peng Chen, Qian Zheng, Peng Lin, Gang Pan

Processing long sequence data such as speech requires models to maintain long-term dependencies, which is challenging for recurrent spiking neural networks due to high temporal dynamics in neuron models that leak stored information in their membrane potentials, and due to vanishing gradients during backpropagation through time. These issues can be mitigated by employing more complex neuron designs, such as ALIF and TC-LIF, but these neuron-level solutions often incur high computational costs and complicate hardware implementation, undermining the efficiency advantages of Spiking neural networks. Here we propose an architectural-level solution that leverages the dynamical interactions of a few leaky integrate-and-fire (LIF) neurons to enhance long-term information storage. The memory capability of this LIF-based micro-circuit is adaptively modulated by global recurrent connections of the recurrent spiking neural network, contributing to selective enhancement of temporal information retention, and promoting stable gradient propagation through time. The proposed model outperforms previous methods including LSTM, ALIF, and TC-LIF in long sequence tasks, achieving 96.52\% accuracy on the PS-MNIST dataset. Furthermore, our method also provides a compelling efficiency advantage, yielding up to 277$\times$ computational efficiency improvement compared to conventional models such as LSTM. This work paves the way for building cost-effective, hardware-friendly, and interpretable spiking neural networks for long sequence modeling.

Applications · Language, Speech and Dialog

Haoran Luo, Haihong E, Guanting Chen, Qika Lin, Yikai Guo, Fangzhi Xu, Zemin Kuang, Meina Song, Xiaobao Wu, Yifan Zhu 等

Retrieval-Augmented Generation (RAG) mitigates hallucination in LLMs by incorporating external knowledge, but relies on chunk-based retrieval that lacks structural semantics. GraphRAG methods improve RAG by modeling knowledge as entity-relation graphs, but still face challenges in high construction cost, fixed one-time retrieval, and reliance on long-context reasoning and prompt design. To address these challenges, we propose Graph-R1, an agentic GraphRAG framework via end-to-end reinforcement learning (RL). It introduces lightweight knowledge hypergraph construction, models retrieval as a multi-turn agent-environment interaction, and optimizes the agent process via an end-to-end reward mechanism. Experiments on standard RAG datasets show that Graph-R1 outperforms traditional GraphRAG and RL-enhanced RAG methods in reasoning accuracy, retrieval efficiency, and generation quality. Our code is available.

Applications · Language, Speech and Dialog

Yangbo Wei, Zhen Huang, Shaoqiang Lu, Junhong Qian, Chen Wu, Lei He

In open-ended domains, natural language instructions are often *underspecified*, mapping to multiple valid yet functionally distinct latent intents. While Large Language Models (LLMs) excel at generation, their ability to resolve such *task ambiguity* through interaction is currently hampered by *semantic blindness*—a tendency to squander interaction budgets on distinguishing trivial syntactic variants rather than fundamental intent differences. To address this, we propose *Topological Active Inference (TAI)*, a geometric framework that recasts disambiguation as a process of *intent-manifold contraction*. TAI first leverages *Persistent Homology* to recover the topological skeleton of the solution space, theoretically guaranteeing the separation of semantic signal from syntactic noise. Subsequently, it synthesizes clarifying questions as *separating hyperplanes* designed to efficiently bisect the probability mass of the intent manifold. We introduce *Topological Expected Information Gain (TEIG)* for question selection and prove that maximizing TEIG reduces query complexity from linear $\mathcal{O}(N)$ to logarithmic $\mathcal{O}(\log K)$, where $K$ is the number of latent intents. Extensive experiments demonstrate that TAI recovers user intent with significantly fewer turns, achieving state-of-the-art disambiguation efficiency.

Applications · Language, Speech and Dialog

Artem Dementyev, Wazeer Zulfikar, Sinan Hersek, Pascal Getreuer, Anurag Kumar, Vivek Kumar

Current multimodal LLMs process audio as a mono stream, ignoring the rich spatial information essential for embodied AI. Existing spatial audio models, conversely, are constrained to fixed microphone geometries, preventing deployment across diverse devices. We present PhaseCoder, a transformer-only spatial audio encoder that is agnostic to microphone geometry. PhaseCoder takes raw multichannel audio and microphone coordinates as inputs to perform localization and produces robust spatial embeddings. We integrate PhaseCoder with the Gemma 3n LLM by finetuning it to reason over ``spatial audio tokens''. We show our encoder achieves state-of-the-art results on microphone-invariant localization benchmarks and, for the first time, enables an LLM to perform complex spatial reasoning and targeted transcription tasks from an arbitrary microphone array.

Applications · Language, Speech and Dialog

Bin Kang, Shaoguo Wen, Yang Fan, Shunlong Wu, Junjie Wang, Yulin Li, Junzhi Zhao, Junle Wang, Zhuotao Tian

While existing text-to-speech (TTS) models exhibit high expressiveness, fine-grained control over composite instructions remains challenging due to the structural mismatch between discrete textual intents and continuous acoustic realizations. Inspired by human cognitive decoupling, we introduce AgentSteerTTS, a multi-agent closed-loop framework designed for intent-faithful expressive control of composite instructions. First, in our framework, an adversarial disentanglement module mitigates speaker-emotion leakage through gradient reversal and cross-covariance regularization. Next, a Dual-Stream Anchoring Controller grounds abstract intents using a large-scale acoustic prototype library: a Retrieval Agent selects expressive anchors, while a Synthesis Agent fuses them into continuous control vectors via gated attention. Finally, a Fast–Slow Feedback Agent refines output intensity through latent gradient correction and resolves semantic–acoustic mismatches using high-level perceptual critique. Experiments on a composite-instruction benchmark and public test sets show that AgentSteerTTS yields consistent and significant improvements to the baselines, demonstrating the effectiveness of the proposed method. Our code and models will be released.

Applications · Language, Speech and Dialog

Yifan Yang, Hui Wang, Bing Han, Shujie Liu, Jinyu Li, Yong Qin, Xie Chen

Recent advances in text-to-speech (TTS) technology have enabled systems to generate speech that is often indistinguishable from human speech, bringing benefits to accessibility, content creation, and human-computer interaction. However, current evaluation practices are increasingly inadequate for capturing the full range of capabilities, limitations, and societal impacts of modern TTS systems. This position paper introduces the concept of Responsible Evaluation and argues that it is essential and urgent for the next phase of TTS development, structured through three progressive levels: (1) ensuring the faithful and accurate reflection of a model's true capabilities and limitations, with more robust, discriminative, and comprehensive objective and subjective scoring methodologies; (2) enabling comparability, standardization, and transferability through standardized benchmarks, transparent reporting, and transferable evaluation metrics; and (3) assessing and mitigating ethical risks associated with forgery, misuse, privacy violations, and security vulnerabilities. Through this concept, we critically examine current evaluation practices, identify systemic shortcomings, and propose actionable recommendations. We hope this concept will not only foster more reliable and trustworthy TTS technology but also guide its development toward ethically sound and societally beneficial applications.

General Machine Learning · Supervised Learning

Zikun Jin, Yuhua Qian, Xinyan Liang, Jiaqian Zhang, Haijun Geng

Robust signal enhancement at extremely low SNR is fundamentally challenging because noise becomes strongly entangled with the signal and corrupts local time–frequency (TF) evidence. In this regime, fixed resolution short-time Fourier transform (STFT) enhancement with purely data driven convolutional biases can become overconfident in unreliable TF regions, causing unstable suppression or residual artifacts. We propose FracKGMF, which couples Fractional Distance Decay Convolution (FracConv) with Knowledge Guided Multi-view Fusion (KGMF) for expressive TF modeling and reliability aware decisions under heavy corruption. FracConv introduces a lightweight fractional distance decay family that reshapes local interactions into long tailed receptive patterns, enabling aggregation of weak but globally consistent cues when per-bin observations are ambiguous. KGMF further injects a Wiener inspired reliability view derived from noise statistics to calibrate multi-view fusion, avoiding over-suppression in uncertain regions while exploiting confident structure for effective denoising. Across speech and electromagnetic (EM) benchmarks, FracKGMF improves perceptual quality and intelligibility over state-of-the-art baselines, with especially robust gains at extremely low SNR; on EM dataset at -20 dB, it achieves an average improvement of 33 dB where conventional TF methods performed poorly. The code will be released upon acceptance.

Applications · Language, Speech and Dialog

Ke Lei, Yu Zhang, Changhao Pan, Xueyi Pu, Wenxiang Guo, Ruiqi Li, Zhou Zhao

Real-time and accurate spatial audio generation is pivotal for delivering an immersive experience. However, existing spatial audio synthesis technologies are often encumbered by a tradeoff between generation quality and high inference latency, as well as difficulty in capturing precise spatial information from multimodal inputs. To address these challenges, we propose S3Audio, a unified streaming framework for high-fidelity spatial audio generation from panoramic videos and text prompts. S3Audio mainly makes the following contributions: 1) We introduce a causal autoregressive diffusion transformer architecture that enables streaming high-quality spatial audio generation. 2) We design a Spatial Video–Audio Contrastive (SVAC) learning strategy to align the video encoder with the acoustic domain, and further employ a multi-objective online direct preference optimization~(ODPO) scheme, resulting in strong spatial perception and robust multimodal spatial audio synthesis. 3) To alleviate the current scarcity of spatial audio datasets, we also develop an automated annotation pipeline for generating detailed spatial captions. Experimental results demonstrate that S3Audio achieves superior performance in both video-to-spatial and text-to-spatial audio generation tasks. Demos can be found at: \url{https://s3audio.github.io}

Deep Learning · Attention Mechanisms

Shogo Yamauchi, Tohru Nitta, Hideaki Tamori

Quaternion Neural Networks are parameter-efficient and model multidimensional dependencies by representing four related features as a single entity. However, existing quaternion self-attention computes component-wise scores and applies independent softmax operations to each component, which increases computational cost and allows attention distributions to diverge across components. We propose a Shared-Score Quaternion Self-Attention mechanism that computes a single real-valued score using the quaternion inner product and applies a shared attention distribution across all components. This reduces score-computation multiplications by 75\% and the number of softmax operations from four to one. We prove that the component-wise and shared scores lie in the same interaction subspace—the linear span of bilinear terms induced by quaternion linear projections. This indicates that independent component-wise attention primarily re-parameterizes the same interactions rather than fundamentally expanding the feature interaction space. In speech enhancement, where phase information is crucial, our method reduces the inference time by 45--61\% while maintaining enhancement quality, making quaternion attention a more practical approach. These findings provide a systematic approach to efficient hypercomplex attention.

Applications · Language, Speech and Dialog

Tae Soo Kim, Yoonjoo Lee, Jaesang Yu, John Chung, Juho Kim

To handle ambiguous and open-ended requests, Large Language Models (LLMs) are increasingly trained to interact with users to surface intents they have not yet expressed (e.g., ask clarification questions). However, users are often ambiguous because they have not yet formed their intents: they must observe and explore outcomes to discover what they want. Simply asking "what kind of tone do you want?" fails when users themselves do not know. We introduce DiscoverLLM, a novel and generalizable framework that trains LLMs to help users form and discover their intents. Central to our approach is a novel user simulator that models cognitive state with a hierarchy of intents that progressively concretize as the model surfaces relevant options---where the degree of concretization serves as a reward signal that models can be trained to optimize. Resulting models learn to collaborate with users by adaptively diverging (i.e., explore options) when intents are unclear, and converging (i.e., refine and implement) when intents concretize. Across proposed interactive benchmarks in creative writing, technical writing, and SVG drawing, DiscoverLLM achieves over 10% higher task performance while reducing conversation length by up to 40%. In a user study with 75 human participants, DiscoverLLM improved conversation satisfaction and efficiency compared to baselines.

Deep Learning · Generative Models and Autoencoders

Gilad Nurko, Roi Benita, Yehoshua Dissen, Tomohiro Nakatani, Marc Delcroix, Shoko Araki, Joseph Keshet

Robust classification in noisy environments remains a fundamental challenge in machine learning. Standard approaches typically treat signal enhancement and classification as separate, sequential stages: first enhancing the signal and then applying a classifier. This approach fails to leverage the semantic information in the classifier's output during denoising. In this work, we propose a general, domain-agnostic framework that integrates two interacting diffusion models: one operating on the input signal and the other on the classifier's output logits, without requiring any retraining or fine-tuning of the classifier. This coupled formulation enables mutual guidance, where the enhancing signal refines the class estimation and, conversely, the evolving class logits guide the signal reconstruction towards discriminative regions of the manifold. We introduce three strategies to effectively model the joint distribution of the input and the logit. We evaluated our joint enhancement method for image classification and automatic speech recognition. The proposed framework surpasses traditional sequential enhancement baselines, delivering robust and flexible improvements in classification accuracy under diverse noise conditions.

Yitian Gong, Kuangwei Chen, Zhaoye Fei, Xiaogui Yang, Ke Chen, Yang Wang, Kexin Huang, Mingshu Chen, Ruixiao Li, Qinyuan Cheng 等

Discrete audio tokenizers are fundamental to empowering large language models with native audio processing and generation capabilities. Despite recent progress, existing approaches often rely on pretrained encoders, semantic distillation, or heterogeneous CNN-based architectures. These designs introduce fixed inductive biases that limit reconstruction fidelity and hinder effective scaling. In this paper, we argue that discrete audio tokenization should be learned fully end-to-end using a homogeneous and scalable architecture. Based on this perspective, we propose $\textbf{TAC}$, a Transformer-based audio tokenizer that jointly optimizes the encoder, quantizer, and decoder from scratch for high-fidelity reconstruction of general audio. We show that a simple, fully end-to-end learned tokenizer built from homogeneous, causal Transformer blocks scales gracefully and supports high-fidelity reconstruction across diverse audio domains. Across speech, sound, and music, the proposed tokenizer consistently outperforms prior codecs over a wide range of bitrates, while exhibiting predictable improvements with increased scale. Notably, leveraging TAC’s discrete tokens, we develop the first purely autoregressive TTS model that surpasses prior non-autoregressive and cascaded systems. Furthermore, TAC enables competitive ASR performance without auxiliary encoders. Our findings position TAC as a unified, scalable interface for the next generation of native audio foundation models.

Applications · Language, Speech and Dialog

Haolin Yang, Jipeng Zhang, Zhitao He, Alexander Zhou, Yi Fung

Large Language Models (LLMs) often struggle with the precise logic and schema alignment required for complex Text-to-SQL tasks. While current methods rely heavily on static prompting, they lack the ability to dynamically adapt and self-correct through environmental interaction. To bridge this gap, we propose **MARS-SQL**, a multi-agent architecture that leverages interactive Reinforcement Learning (RL) to optimize SQL generation. Unlike monolithic approaches, our method decomposes the problem into three specialized roles: schema linking, query generation, and solution validation. Central to our approach is a generation agent trained via a multi-turn RL policy, which operates within a ReAct-style loop. This agent learns to iteratively reason, execute intermediate SQL actions on a live database, and refine its strategy based on execution feedback. To ensure robustness, we introduce a validation mechanism that treats solution selection as a generative modeling task, identifying the optimal interaction trajectory through next-token prediction probabilities. Empirical evaluations demonstrate the effectiveness of coupling interactive learning with trajectory ranking. **MARS-SQL** achieves state-of-the-art performance, recording an execution accuracy of 77.84\% on the BIRD development dataset and 89.75\% on the Spider test dataset.

Applications · Language, Speech and Dialog

Zhenliang Zhang, Wenqing Wang, Yong Hu, Yaming Yang, Jiaheng Gao, Chen Shen, Xiaojun Wan

Long-Text Understanding (LTU) at million-token scale requires balancing reasoning fidelity with computational efficiency. Frontier long-context LLMs can process millions of token contexts end-to-end, but they suffer from high token consumption and attention dilution. In parallel, specialized LTU agents often sacrifice fidelity through task-agnostic abstractions like graph construction or indexing. We identify a key insight for LTU: query-relevant information is typically sparse relative to the full document, so effective reasoning should rely on a query-sufficient subset rather than the entire context. To address this, we propose SCOUT, a new paradigm for LTU that **shifts from passive processing to active information foraging**. It treats the document as an explorable environment and answers from a compact, provenance-grounded epistemic state. Guided by state-level gap diagnosis, SCOUT adaptively alternates between coarse-to-fine exploration and anchored state updates that progressively *contract* its epistemic state toward query sufficiency. Experiments show that SCOUT matches state-of-the-art proprietary models while reducing token consumption by up to 8 times. Moreover, SCOUT remains stable as context length scales, substantially alleviating the practical cost--capability trade-off in long-context reasoning. **Code.** An anonymous repository link is provided in the Supplementary Material.

General Machine Learning · Evaluation

Huanzhi Mao, Aditya Ghai, Imra Dawoodani, Tony Ginart, Shishir G. Patil, John Emmons, Joseph E Gonzalez

Audio agents are increasingly deployed to execute tools from spoken requests, yet audio tool use poses challenges beyond text-only function calling: perception errors (e.g., homophones, noise, disfluencies) can corrupt entities and arguments, and natural interactions often require clarification that changes the tool-calling protocol. We introduce MFCL-Audio, a large-scale benchmark for audio function calling with 6.2K expert-verified tasks across two suites that mirror common deployments: MFCL Text Audio \(pipelined ASR$\rightarrow$LLM$\rightarrow$tools via transcripts) and MFCL True Audio \(end-to-end audio-in$\rightarrow$tool calls). MFCL-Audio includes controlled speech and acoustic perturbations (accent and speaking-rate variation, content disfluencies, and background noise) generated through a controllable audio synthesis/augmentation pipeline. We provide automatic grading for both function names and argument values using AST-based matching for single-turn calls and response/state-based metrics for multi-turn interactions, enabling scalable evaluation without LLM judges. Across a broad set of models, we propose a failure-mode taxonomy and analyze which speech and noise factors most strongly impact tool-calling accuracy. We release the benchmark, evaluation harness, and audio pipeline to support research on reliable speech-based agents.

Esther Sun, Bo-Hao Su, Abinay Reddy Naini, Shinji Watanabe, Carlos Busso

Speech Large Language Models (SLLMs) enable high-level emotion reasoning, but often produce ungrounded, text-biased judgments without verifiable acoustic evidence. In contrast, SSL encoders such as WavLM yield strong acoustic representations yet remain opaque discriminative models that offer limited interpretability. To bridge this gap, we introduce the Agentic Decoding of Emotion via Probing Tools (ADEPT) framework, which reframes emotion recognition as a multi-turn inquiry process rather than a single-pass prediction. ADEPT transforms an SLLM into an agent that maintains an evolving candidate set and adaptively invokes dedicated semantic and acoustic probing tools within a structured pipeline of candidate generation, evidence collection, and adjudication. Crucially, ADEPT enables a paradigm shift from consensus learning to ambiguity-driven emotion reasoning. Since human affect exhibits complexity and co-occurrence of emotions, we leverage minority annotations as informative signals instead of discarding them as noise. Finally, we integrate Group Relative Policy Optimization (GRPO) with the Evidence Trust Gate to explicitly couple tool-usage behaviors with prediction quality and enforce evidence-based reasoning. Experiments demonstrate that ADEPT improves in most cases the primary emotion accuracy while substantially improving minor emotion characterization, producing explanations grounded in auditable evidence.

Deep Learning · Robustness

Tianyi Xu, Cheng'an Wei, Yue Zhao, Kai Chen

Automatic Speech Recognition (ASR) systems, such as those in intelligent assistants, are vulnerable to adversarial examples (AEs). Benign audio clips like music, when embedded with small perturbations, can trick ASR models into recognizing attacker-specified commands. Prior studies focus on minimizing perturbation magnitude to craft AEs. However, they fails to achieve high attack stealthiness against black-box ASR systems in the physical world. In this paper, we introduce the first music carrier selection algorithm and an attention-aware stealthiness loss function to generate stealthy AEs. Extensive evaluations on five commercial ASR APIs and three widely-used voice assistants demonstrate that our method significantly outperforms state-of-the-art techniques in both effectiveness and stealthiness. Notably, in a user study involving 200 participants, 55.6\% of participants perceived our physical adversarial examples as benign audio, which is an improvement of over 20\% compared to existing methods.

Applications · Language, Speech and Dialog

Yiqing Xie, Emmy Liu, Gaokai Zhang, Nachiket Kotalwar, Shubham Gandhi, Acharya, Xingyao Wang, Carolyn Rose, Graham Neubig, Daniel Fried

Coding agents are increasingly used for a wide range of real-world tasks, from adding features and documentation to creating programs from scratch. Ideally, the agent should perform well across all the diverse tasks. However, most prior work concentrates on issue solving, and such single-task training does not transfer reliably to other coding tasks. In this work, we aim to train coding agents that generalize across tasks. We first analyze task transferability from two axes: the commonalities shared among coding tasks and the capabilities of current agents. Guided by these findings, we derive a set of principles for training task design and verify them through a series of controlled experiments. We then present Hybrid-Gym, a training dataset built on four scalable training tasks that follow these principles. Experiments show that, under zero-shot task transfer, Hybrid-Gym achieves performance comparable to in-domain training, and further improves existing datasets when combined with them (e.g., +4.85% on SWT-Bench-Verified).

Deep Learning · Sequential Models, Time series

Ruben Solozabal, Velibor Bojkovic, Hilal AlQuabeh, Klea Ziu, Kentaro Inui, Martin Takac

State-space models (SSMs) have emerged as a powerful foundation for long-range sequence modeling, with the HiPPO framework showing that continuous-time projection operators can be used to derive stable, memory-efficient dynamical systems that encode the past history of the input signal. However, existing projection-based SSMs often rely on polynomial bases with global temporal support, whose inductive biases are poorly matched to signals exhibiting localized or transient structure. In this work, we introduce \emph{WaveSSM}, a collection of SSMs constructed over wavelet frames. Our key observation is that wavelet frames yield a localized support on the temporal dimension, useful for tasks requiring precise localization. Empirically, we show that on equal conditions, \textit{WaveSSM} outperforms orthogonal counterparts as S4 on real-world datasets with transient dynamics, including physiological signals on the PTB-XL dataset and raw audio on Speech Commands.