论文检索

Deep Learning · Large Language Models

Wanlong Fang, Tianle Zhang, Wen Tao, Alvin Chan

Understanding \emph{modality interaction} in multimodal large language models (MLLMs) remains a central challenge for reliable and interpretable deployment. We introduce Partial Information Decomposition (PID) as a unified, decision-level framework that separates \emph{unique}, \emph{redundant}, and \emph{synergistic} contributions of sensory and linguistic inputs, moving beyond representation alignment and outcome-based evaluation. Across vision–language benchmarks, PID reveals stable \emph{interaction regimes}: reasoning-oriented tasks consistently exhibit high cross-modal synergy, whereas knowledge-oriented tasks are dominated by language-unique information. These regimes generalize across architectures and scales and predict causal sensitivity to modality-level interventions. We extend this framework to tri-modal systems with Sensory PID, treating language as a control variable to decompose information gain from video and audio. Applied to omni-modal models, this analysis uncovers a persistent \emph{sensory synergy bottleneck}, where decisions remain dominated by visual information even on fusion-dependent tasks. Layer-wise analysis further show that sensory integration emerges late and is instruction-gated, following early visual saturation.

Deep Learning · Foundation Models

Xiangyu Zeng, Zhiqiu Zhang, Yuhan Zhu, Xinhao Li, Zikang Wang, Changlian Ma, Qingyu Zhang, Zizheng Huang, Kun Ouyang, Tianxiang Jiang 等

Existing multimodal large language models for long-video understanding predominantly rely on uniform sampling and single-turn inference, limiting their ability to identify sparse yet critical evidence amid extensive redundancy. We introduce VideoSeeker, a novel framework that supports iterative discovery of salient visual clues, fine-grained inspection of key segments, and adaptive termination once sufficient evidence is acquired. Technically, we address two core challenges in interleaved tool invocation. First, to mitigate attention dispersion induced by the heterogeneity of reasoning and tool-calling, we propose Task-Decoupled Attention Masking, which isolates per-step concentration while preserving shared global context. Second, to control context length growth in multi-turn interactions, we introduce a Verifiable Trajectory-Guided Reward that balances exploration coverage with reasoning efficiency. To support training at scale, we further develop a data synthesis pipeline and construct Seeker-173K, comprising 173K high-quality tool-interaction trajectories for effective supervised and reinforcement learning. Extensive experiments show that VideoSeeker substantially outperforms state-of-the-art methods, achieving 72.1% accuracy on MLVU and 46.5% on Video-Holmes. These results demonstrate VideoSeeker's strong multi-hop evidence-seeking and reasoning capabilities, and validate the effectiveness of native tool invocation in long-video scenarios.

Shenyuan Gao, William Liang, Kaiyuan Zheng, Ayaan Malik, Seonghyeon Ye, Sihyun Yu, Wei-Cheng Tseng, Yuzhu Dong, Kaichun Mo, Chen-Hsuan Lin 等

Being able to simulate the outcomes of actions in varied environments will revolutionize the development of generalist agents at scale. However, modeling these world dynamics, especially for dexterous robotics tasks, poses significant challenges due to limited data coverage and scarce action labels. As an endeavor towards this end, we introduce DreamDojo, a foundation world model that learns diverse interactions and dexterous controls from 44k hours of egocentric human videos. Our data mixture represents the largest video dataset to date for world model pretraining, spanning a wide range of daily scenarios with diverse objects and skills. To address the scarcity of action labels, we introduce continuous latent actions as unified proxy actions, enhancing interaction knowledge transfer from unlabeled videos. After post-training on small-scale target robot data, DreamDojo demonstrates a strong understanding of physics and precise action controllability. We also devise a distillation pipeline that accelerates DreamDojo to a real-time speed of 10.93 FPS and further improves consistency to the context. Our work enables several important applications based on generative world models, including live teleoperation, policy evaluation, and model-based planning. Systematic evaluation on multiple challenging out-of-distribution (OOD) benchmarks verifies the significance of our method for simulating open-world, contact-rich tasks, paving the way for general-purpose robot world models.

Applications · Computer Vision

Zhenghao Xing, Ruiyang Xu, Yuxuan Wang, Jinzheng He, Ziyang Ma, Qize Yang, Yunfei Chu, Jin Xu, Junyang Lin, Chi Wing Fu 等

Passive models for long video understanding typically rely on a ``watch-it-all'' paradigm, processing data uniformly regardless of query difficulty, causing input complexity to scale linearly with video duration. Although interactive frameworks have emerged, they often rely on global pre-scanning, failing to strictly decouple perception from temporal length. We introduce OmniAgent, a POMDP-based active perception framework. OmniAgent executes on-demand actions to selectively distill audio-visual signals into a persistent textual memory, imposing an information bottleneck that fundamentally decouples reasoning complexity from raw video duration. To operationalize this, we bootstrap the agent via success-driven trajectory synthesis and optimize its policy using TAURA (Turn-aware Adaptive Uncertainty Rescaled Advantage). TAURA addresses credit assignment ambiguity in long-horizon reasoning by leveraging token entropy as a proxy for decision criticality, explicitly steering gradients toward pivotal discovery steps. Crucially, OmniAgent demonstrates a positive test-time scaling property, where performance improves as the reasoning turns increase, validating the efficacy of adaptive perception. Empirical results across ten benchmarks (e.g., VideoMME, LVBench) demonstrate that OmniAgent establishes new state-of-the-art performance. Notably, on LVBench, our 7B agent outperforms the 10$\times$ larger Qwen2.5-VL-72B (50.5 vs. 47.3), validating the effectiveness of query-conditional active perception.

Deep Learning · Large Language Models

Wenqi Liu, Yunxiao Wang, Shijie Ma, Meng Liu, Qile Su, Tianke Zhang, Haonan Fan, Changyi Liu, Kaiyu Jiang, Jiankang Chen 等

In long-video understanding, conventional uniform frame sampling often fails to capture key visual evidence, leading to degraded performance and increased hallucinations. To address this, recent agentic thinking-with-videos paradigms have emerged, adopting a localize–clip–answer pipeline in which the model actively identifies relevant video segments, performs dense sampling within those clips, and then produces answers. However, existing methods remain inefficient, suffer from weak localization, and adhere to rigid workflows. To solve these issues, we propose VideoTemp-o3, a unified agentic thinking-with-videos framework that jointly models video grounding and question answering. VideoTemp-o3 exhibits strong localization capability, supports on-demand clipping, and can refine inaccurate localizations. Specifically, in the supervised fine-tuning stage, we design a unified masking mechanism that encourages exploration while preventing noise. For reinforcement learning, we introduce dedicated rewards to mitigate reward hacking. Besides, from the data perspective, we develop an effective pipeline to construct high-quality long video grounded QA data, along with a corresponding benchmark for systematic evaluation across various video durations. Experimental results demonstrate that our method achieves remarkable performance on both long video understanding and grounding.

Theory · Deep Learning

Song Li, yongping xiong

Large multimodal models (LMMs) have shown strong capabilities in audio-visual understanding by jointly reasoning over visual, auditory, and linguistic inputs. However, processing long-form audio-visual content often requires a large number of tokens, leading to substantial computational and memory overhead during inference. Existing efficiency-oriented methods typically apply uniform compression or pruning strategies, which overlook modality-specific characteristics and instruction-dependent reasoning behaviors in multimodal models. In this work, we present PRIM, an inference framework for efficient multimodal reasoning that systematically compresses audio-visual representations based on attention dynamics and instruction relevance. Motivated by an attention-based analysis revealing modality imbalance and layer-wise redundancy in LMMs, PRIM introduces a cooperative compression pipeline that spans both multimodal encoders and the language model. Specifically, PRIM performs early text-conditioned audio-visual fusion to externalize cross-modal interactions, applies attention-guided and frequency-aware strategies to remove redundant audio and video tokens, and further adapts token retention inside the language model according to task demands. Extensive experiments on multiple audio-visual benchmark datasets demonstrate that PRIM consistently achieves stable and superior efficiency--accuracy trade-offs across diverse tasks and datasets. These results demonstrate that PRIM, a multimodal cooperative compression approach, provides an effective pathway toward scalable and efficient audio-visual reasoning.

General Machine Learning · Representation Learning

Yahya Sattar, Sunmook Choi, Leo Maynard-Zhang, Yassir Jedra, Maryam Fazel, Sarah Dean

Auto-regressive models have emerged as powerful tools for sequential data, from language to video. Understanding how and why these models learn latent representations remains an open theoretical question. In this work, we demonstrate that when trained by empirical risk minimization on data from partially observed linear dynamical systems, two-layer linear auto-regressive models naturally learn to approximate Kalman filtering. In particular, we show that the learned hidden representation coincides, up to a similarity transformation, with the state estimates produced by the optimal (Kalman) filter, even though the model has no explicit knowledge of the underlying dynamics or state. The result follows from three main insights. First, we establish that the Kalman filter is well approximated by an auto-regressive model with bounded truncation error. Second, we show that despite non-convexity, the two-layer optimization landscape is benign, i.e., all stationary points are either strictly saddle or global minima. Finally, as our main contributions, we provide finite-sample guarantees on prediction error, parameter estimation error, and latent state recovery. Numerical simulations support the theoretical results and demonstrate that auto-regressive models automatically represent latent state estimates.

Deep Learning · Large Language Models

Silin Gao, Hao Zhao, Zeming Chen, Sepideh Mamooler, Antara R Bhattacharya, Qiyu Wu, Hiromi Wakaki, Yuki Mitsufuji, Li Mi, Syrielle Montariol 等

Multimodal LLMs lack a systematic understanding of visual dynamics in complex human world activities, which requires the model to predict or simulate multiple levels of dynamic constituents, such as the general progression of actions and the associated changes of low-level details in the world. To address this challenge, we propose a dynamic visual schema-guided world model, DynaVieW, optimized for visual dynamic prediction and simulation. DynaVieW achieves an in-depth understanding of visual dynamics by learning interleaved state-transition sequences, where states cover broad visual scenes from video keyframes, and transitions capture comprehensive dynamic constituents within a hierarchical schema. DynaVieW jointly models transition prediction and state simulation under a mixture-of-experts architecture, with a cross-expert selective attention and a schema token re-weighted loss, to ensure effective and robust learning. DynaVieW's superior visual dynamic understanding boosts its downstream performances on both visual narrative creation and world simulation, showing improved consistency and controllability of visual generation and better instruction-following ability.

Zongsheng Cao, Anran Liu, Jun Xie, Jing Li, Lang Chen, Feng Chen, zhepeng Wang, Zigan Wang

Long-video temporal reasoning remains challenging for Large Video Language Models (LVLMs). Recent reasoning-enhanced models apply reinforcement learning with outcome supervision to improve temporal understanding. However, outcome-only rewards cannot distinguish whether a model arrived at the correct answer through valid temporal reasoning or fabricated claims, a fundamental limitation that undermines trustworthiness. We observe a key structural correspondence: in videos, events form \emph{temporal traces}, ordered sequences of how entities interact over time; in model reasoning, \emph{reasoning traces} capture step-by-step temporal claims. Correct temporal reasoning requires the latter to mirror the former. This correspondence enables us to \emph{verify} reasoning traces against the video's temporal structure. We introduce \textbf{Temporal Reasoning Traces (TRT)}, a unified representation that indexes ordered event chains from videos and serves as a verification oracle for model reasoning. Building on TRT, we propose \textbf{trace-grounded process supervision}: during reinforcement learning, each temporal claim in the model's reasoning trace is programmatically verified against TRT,rewarding grounded reasoning and penalizing fabrications. Unlike neural reward models that may themselves err, our verification is fully deterministic. Extensive experiments show the effectiveness of our model, achieving state-of-the-art performance.

Applications · Robotics

Yu Qi, Haibo Zhao, Ziyu Guo, Siyuan Ma, Ziyan Chen, Yaokun Han, Renrui Zhang, Zitiantao Lin, Yizhe Zhu, Shiji Xin 等

Understanding the capability bottlenecks of embodied multimodal large language models (MLLMs) is crucial for improvement. However, existing embodied benchmarks fail to provide actionable insights because they focus on task-level evaluation rather than discovering capability bottlenecks. To address this, we introduce BEAR, where we divide embodied tasks into 14 atomic skills for skill-level evaluation. BEAR comprises 4,469 interleaved image–video–text entries across 14 skills in 6 categories, ranging from low-level perception to high-level planning. We evaluate 20 MLLMs on BEAR under a hierarchical skill-level diagnosis framework and discover that (1) perceptual capabilities are major bottlenecks behind reasoning failures, and (2) models fail due to unstable spatiotemporal modeling which remain unexposed in previous benchmarks. Furthermore, building on these insights, we propose BEAR-Agent, a multimodal conversable agent that augments MLLMs with visual and spatial tools. It substantially enhances MLLMs’ performance across skills, yielding a relative improvement of 17.5% on GPT-5 on BEAR and outperform baselines by a large margin in both simulation and real-robot experiments across models.

Applications · Computer Vision

Zhihao He, Tieyuan Chen, Kangyu Wang, Ziran Qin, Yang Shao, Chaofan Gan, Shijie Li, Zuxuan Wu, Weiyao Lin

Current Video Large Language Models (Video LLMs) typically encode frames via a vision encoder and employ an autoregressive (AR) LLM for understanding and generation. However, this AR paradigm inevitably faces a dual efficiency bottleneck: strictly unidirectional attention compromises *understanding efficiency* by hindering global spatiotemporal aggregation, while serial decoding restricts *generation efficiency*. To address this, we propose **VidLaDA**, a Video LLM based on Diffusion Language Models (DLMs) that leverages bidirectional attention to unlock comprehensive spatiotemporal modeling and decode tokens in parallel. To further mitigate the computational overhead of diffusion decoding, we introduce **MARS-Cache**, an acceleration strategy that prunes redundancy by combining asynchronous visual cache refreshing with frame-wise chunk attention. Experiments show VidLaDA rivals state-of-the-art AR baselines (e.g., Qwen2.5-VL and LLaVA-Video) and outperforms DLM baselines, with MARS-Cache delivering over 12x speedup without compromising accuracy. *Code and checkpoints will be available in the camera-ready version.*

Applications · Health / Medicine

Danush Kumar Venkatesh, Adam Schmidt, Muhammad Abdullah Jamal, Omid Mohareri

Surgical video datasets are essential for scene understanding, enabling procedural modeling and intra-operative support. However, these datasets are often heavily imbalanced, with rare actions and tools under-represented, which limits the robustness of downstream models. We address this challenge with *SurgiFlowVid*, a sparse and controllable video diffusion framework for generating surgical videos of under-represented classes. Our approach introduces a dual-prediction diffusion module that jointly denoises RGB frames and optical flow, providing temporal inductive biases to improve motion modeling from limited samples. In addition, a sparse visual encoder conditions the generation process on lightweight signals (e.g., sparse segmentation masks or RGB frames), enabling controllability without dense annotations. We validate our approach on three surgical datasets across tasks including action recognition, tool presence detection, and laparoscope motion prediction. Synthetic data generated by our method yields consistent gains of $10$–$20$% over competitive baselines, establishing *SurgiFlowVid* as a promising strategy to mitigate data imbalance and advance surgical video understanding methods.

General Machine Learning · Evaluation

Wayne Chi, Yixiong Fang, Arnav Yayavaram, Siddharth Yayavaram, Seth Karten, Qiuhong Anna Wei, Runkun Chen, Alexander Wang, Valerie Chen, Ameet Talwalkar 等

While coding agents have advanced rapidly, progress on multimodal agents has lagged behind, largely due to a gap between the unimodal nature of code and other multimodal computer applications. Game development bridges the modality gap, mirroring software development's complexity in terms of large codebases and contextual complexity, while simultaneously requiring multimodal understanding. We present GameDevBench, the first benchmark for evaluating agents on game development tasks, consisting of 168 tasks derived from web and video tutorials. Tasks require significant multimodal understanding and are complex---the average solution requires more than three times the amount of changes compared to software development benchmarks. Agents still struggle with game development, with the best agent solving only $50.0$\% of tasks. We further introduce two simple image and video-based feedback methods, nearly doubling performance in one setting from $25.6$\% to $44.4$\%. We find that performance degrades sharply with multimodal complexity, dropping on average from $44.4$\% pass@1 on gameplay oriented tasks to $24.3$\% on graphics tasks.

Deep Learning · Large Language Models

Yiyan Ji, Yiyan Ji, Jungang Li, Xuyang Liu, Xinlong Chen, Junfei Wu, Bozhou Li, Bohan Zeng, Yang Shi, Yushuo Guan 等

Omni-modal Large Language Models (Omni-LLMs) have demonstrated strong capabilities in audio-video understanding tasks. However, their reliance on long multimodal token sequences leads to substantial computational overhead. Despite this challenge, token compression methods designed for Omni-LLMs remain limited. To bridge this gap, we propose OmniSIFT (Omni-modal Spatio-temporal Informed Fine-grained Token compression), a modality-asymmetric token compression framework tailored for Omni-LLMs. Specifically, OmniSIFT adopts a two-stage compression strategy: (i) a spatio-temporal video pruning module that removes video redundancy arising from both intra-frame structure and inter-frame overlap, and (ii) a vision-guided audio selection module that filters audio tokens. The entire framework is optimized end-to-end via a differentiable straight-through estimator. Extensive experiments on five representative benchmarks verify the efficacy and robustness of OmniSIFT. Notably, for Qwen2.5-Omni-7B, OmniSIFT adds 4.85M parameters while still achieving lower latency than training-free baselines such as OmniZip. With only 25% of the original token context, OmniSIFT consistently outperforms all compression baselines and even surpasses the full-token model on several tasks.

Hengrui Hu, Jingyu Li, Juntao Liang, Guanyu Chen, Lan Zhang

Although Multimodal Large Language Models have made remarkable progress, they still struggle with long-video understanding due to the massive memory footprint of KV Caches. Exsiting methods often resort to disjoint retrieval or attention-based static reduction to achieve compression. However, these methods disrupt temporal continuity and ignore the varying information density across network layers. In this work, we reveal that memory allocation should mirror layer-wise semantic density, rather than adhering to a uniform budget. To this end, we introduce EAKV, a training-free entropy-driven adaptive KV compression framework that leverages attention entropy to adaptively allocate compression budgets, selectively preserving critical tokens while distilling redundant contexts into compact contextual anchors, thereby achieving granular memory allocation proportional to semantic density. Extensive experiments on four benchmarks demonstrate that EAKV surpasses existing methods across varying model scales with improvements ranging from 1.5% to 4.8%.

Xiongkun Linghu, Jiangyong Huang, Baoxiong Jia, Siyuan Huang

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a key paradigm for unlocking complex reasoning in Large Language Models (LLMs), yet its potential in 3D scene understanding remains untapped. To bridge this gap, we present Reinforcement Fine-Tuning for Video-based 3D Scene Understanding (3D-RFT), the first framework to extend RLVR to 3D perception and reasoning. Our pipeline operates in two stages: activating 3D-aware Multi-modal Large Language Models (MLLMs) via Supervised Fine-Tuning (SFT), followed by reinforcement fine-tuning using Group Relative Policy Optimization (GRPO) with strictly verifiable reward functions. We design task-specific rewards—such as 3D IoU and F1-score—to provide deterministic signals for spatial alignment. Extensive experiments demonstrate that 3D-RFT achieves state-of-the-art performance on video-based 3D scene understanding benchmarks, significantly outperforming VG LLM-8B on detection and grounding tasks. Moreover, our model surpasses larger mainstream models on VSI-Bench, demonstrating the efficiency of verifiable reinforcement learning. We conclude by offering valuable insights into optimal training strategies .

Applications · Computer Vision

Qishen Yin, Tanghui Jia, Peng Jin, Hao Li, Juntong Wu, Guanlin Lu, Li Yuan

Does \emph{Comprehending the main idea of a 2-hour movie} and \emph{Counting the birds appearing in a 15-second clip} really warrant the same video processing pipeline? We present Task-Aware Mechanism (TAM), a hybrid-gated Mixture-of-Experts (MoE) vision tower that adapts frame count and resolution to the user query and video length. TAM introduces a compact 0.1B text-only \emph{Inductor} trained on our TA-116K dataset to infer task types, enabling task-aware visual budgeting and routing: a soft-gated MoE vision encoder for stability, and hard-gated resolution-specific projectors/pipelines for efficient specialization. Built on Qwen2-7B, TallVA-8B-A7B achieves state-of-the-art performance among models with comparable LLMs across diverse video benchmarks and remains competitive against stronger-LLM baselines, showing that task-aware visual budgeting makes video understanding more holistic. The code is included in the supplementary material.

Yanglin Feng, Yang Qin, Dezhong Peng, Rui Wang, Xiaomin Song, Peng Hu

Multimodal learning aims to integrate multi-sensor data to exploit their complementary information, embracing a more comprehensive real-world perception and understanding. However, heterogeneous discrepancies across modalities consistently trigger imbalanced multimodal optimization, restricting the joint learning performance. Although existing methods mitigate this issue through optimization modulation and conflict alleviation, they still suffer from entangled optimization and uniform learning pace in conventional monolithic frameworks, limiting the effectiveness of multimodal learning. To address this issue, we propose a novel Multimodal Nested Learning Framework (MoNet), which reformulates the monolithic framework into nested sub-processes, decoupling and coordinating multimodal learning. To achieve this, we present a Decoupled Multimodal Stable Memory block (DMSM) as the outermost nested level, which decouples multimodal learning into independent optimization streams for semantic exploitation across modalities. Additionally, we develop an Adaptive Multimodal Coordinated Fusion block (AMCF), which constitutes the inner nested level. It attempts to coordinate multimodal information integration across multi-timescale nested memories, balancing multimodal fusion. Extensive experimental results on eight datasets across three tasks demonstrate the superiority of MoNet.

Deep Learning · Robustness

Yanglin Feng, Yang Qin, Dezhong Peng, Rui Wang, Xiaomin Song, Peng Hu

Multimodal learning aims to integrate multi-sensor data to exploit their complementary information, embracing a more comprehensive real-world perception and understanding. However, heterogeneous discrepancies across modalities consistently trigger imbalanced multimodal optimization, restricting the joint learning performance. Although existing methods mitigate this issue through optimization modulation and conflict alleviation, they still suffer from entangled optimization and uniform learning pace in conventional monolithic frameworks, limiting the effectiveness of multimodal learning. To address this issue, we propose a novel Multimodal Nested Learning Framework (MoNet), which reformulates the monolithic framework into nested sub-processes, decoupling and coordinating multimodal learning. To achieve this, we present a Decoupled Multimodal Stable Memory block (DMSM) as the outermost nested level, which decouples multimodal learning into independent optimization streams for semantic exploitation across modalities. Additionally, we develop an Adaptive Multimodal Coordinated Fusion block (AMCF), which constitutes the inner nested level. It attempts to coordinate multimodal information integration across multi-timescale nested memories, balancing multimodal fusion. Extensive experimental results on eight datasets across three tasks demonstrate the superiority of MoNet.

Haojian Huang, Chuanyu Qin, Yinchuan Li, YINGCONG CHEN

Reinforcement learning has advanced video reasoning in large multi-modal models, yet dominant pipelines either rely on on-policy self-exploration, which plateaus at the model’s knowledge boundary, or hybrid replay that mixes policies and demands careful regularization. Dynamic context methods zoom into focused evidence but often require curated pretraining and two-stage tuning, and their context remains bounded by a small model’s capability. In contrast, larger models excel at instruction following and multi-modal understanding, can supply richer context to smaller models, and rapidly zoom in on target regions via simple tools. Building on this capability, we introduce an observation-level intervention: a frozen, tool-integrated teacher identifies the missing spatiotemporal dependency and provides a minimal evidence patch (e.g. timestamps, regions etc.) from the original video while the question remains unchanged. The student answers again with the added context, and training updates with a chosen-rollout scheme integrated into GRPO. We further propose a Robust Improvement Reward (RIR) that aligns optimization with two goals: outcome validity through correct answers and dependency alignment through rationales that reflect the cited evidence. Advantages are group-normalized across the batch, preserving on-policy exploration while directing it along causally meaningful directions with minimal changes to the training stack. Experiments on general video and video-reasoning benchmarks show consistent accuracy gains and strong generalization.