BiOMamba: Mamba-based Forward-Then-Backward Temporal Modeling for Online Action Detection and Anticipation

摘要

Given that action evolution follows temporal progression, recent studies for Online Action Detection (OAD) and Online Action Anticipation (OAA) generally adopt forward temporal modeling to capture dependencies in observable video sequences. However, the strictly sequential nature of forward temporal modeling prevents subsequent frames from being used to enhance the earlier modeling process. In particular, the current frame, the last observable frame in the online video stream, serves as the direct visual cue for ongoing action recognition and the informative context for future action anticipation. As modeling errors accumulate over time, the resulting representations may progressively deviate from the actual semantics. Findings in cognitive neuroscience show that the hippocampus performs backward replay after observation to reinforce and correct the interpretation of previous observations. Inspired by this, we propose to incorporate backward temporal modeling following forward temporal modeling, enabling the model to leverage backward temporal modeling to enhance forward temporal modeling. Based on this idea, we propose a unified model for OAD and OAA, named Bidirectional Online Mamba (BiOMamba). Specifically, to address the excessive length and relevance imbalance in observable sequences, BiOMamba compresses distant long-term memory and preserves recent short-term memory. Then, BiOMamba sequentially model both forward and backward temporal dependencies in the whole memory. Finally, according to the temporal modeling result, BiOMamba generates representations for current and future actions. BiOMamba achieves state-of-the-art performance on THUMOS'14 (OAD: 73.3% mAP, OAA: 59.7% mAP) and TVSeries (OAD: 89.9% mcAP, OAA: 83.7% mcAP).