GraphVideoAgent: Enhancing Long-form Video Understanding with Entity Relation Graphs

摘要

Long-form video understanding (LVU) addresses the challenge of answering complex questions over extended video length, where informative cues are sparse and easily overwhelmed by redundant content. To tackle this, it requires selecting a small set of question-relevant keyframes and reasoning over long-range, temporally dispersed visual evidence. However, current methods typically extract frame-level features with limited temporal context and store them in sequential memory structures. As a result, they struggle to capture the evolving relations among entities and fail to maintain identity consistency when entities temporarily leave and later reappear in the video. These limitations prevent accurate keyframe localization and coherent reasoning. In this paper, we propose GraphVideoAgent, a novel agent-based LVU framework that integrates a dynamic entity relation graph with a large language model (LLM)-based multi-round reasoning. Our framework emulates human cognitive strategies by iteratively retrieving keyframes and explicitly tracking both temporal and semantic interactions among entities. Our GraphVideoAgent iteratively reflects on question cues and visual observations, while the graph memory maintains a structured representation of evolving entity states and their causal relations. This design enables accurate keyframe selection, effective reasoning over sparse visual evidence, and interpretable prediction. Extensive experiments on two LVU benchmarks, EgoSchema and NExT-QA, demonstrate that GraphVideoAgent achieves state-of-the-art performance while using only 8.2 and 8.1 frames on average, significantly improving both accuracy and efficiency.