Real-world object detection systems, such as those in autonomous driving and surveillance, must continuously learn new object categories and simultaneously adapt to changing environmental conditions. Existing approaches, Class Incremental Object Detection (CIOD) and Domain Incremental Object Detection (DIOD)--only address one aspect of this challenge. CIOD struggles in unseen domains, while DIOD suffers from catastrophic forgetting when learning new classes, limiting their real-world applicability. To overcome these limitations, we introduce Dual Incremental Object Detection (DuIOD), a more practical setting that simultaneously handles class and domain shifts in an exemplar-free manner. We propose DuET, a Task Arithmetic-based model merging framework that enables stable incremental learning while mitigating sign conflicts through a novel Directional Consistency Loss. Unlike prior methods, DuET is detector-agnostic, allowing models like YOLO11 and RT-DETR to function as real-time incremental object detectors. To comprehensively evaluate both retention and adaptation, we introduce the Retention-Adaptability Index (RAI), which combines the Average Retention Index (Avg RI) for catastrophic forgetting and the Average Generalization Index for domain adaptability into a common ground. Extensive experiments on the Pascal Series and Diverse Weather Series demonstrate DuET's effectiveness, achieving a +13.12% RAI improvement while preserving 89.3% Avg RI on the Pascal Series (4 tasks), as well as a +11.39% RAI improvement with 88.57% Avg RI on the Diverse Weather Series (3 tasks), outperforming existing methods.
论文检索
输入标题、作者或关键词，从 2,101 篇学术成果中精准定位
Mamba has shown great potential for computer vision due to its linear complexity in modeling the global context with respect to the input length. However, existing lightweight Mamba-based backbones cannot demonstrate performance that matches Convolution or Transformer-based methods. By observing, we find that simply modifying the scanning path in the image domain is not conducive to fully exploiting the potential of vision Mamba. In this paper, we first perform comprehensive spectral and quantitative analyses, and verify that the Mamba block mainly models low-frequency information under Convolution-Mamba hybrid architecture. Based on the analyses, we introduce a novel Laplace mixer to decouple the features in terms of frequency and input only the low-frequency components into the Mamba block. In addition, considering the redundancy of the features and the different requirements for high-frequency details and low-frequency global information at different stages, we introduce a frequency ramp inception, i.e., gradually reduce the input dimensions of the high-frequency branches, so as to efficiently trade-off the high-frequency and low-frequency components at different layers. By integrating mobile-friendly convolution and efficient Laplace mixer, we build a series of tiny hybrid vision Mamba called TinyViM. The proposed TinyViM achieves impressive performance on several downstream tasks including image classification, semantic segmentation, object detection and instance segmentation. In particular, TinyViM outperforms Convolution, Transformer and Mamba-based models with similar scales, and the throughput is about 2-3 times higher than that of other Mamba-based models. The good balance between the efficiency and performance of TinyViM shows that a properly designed and optimized vision mamba can achieve high performance with a small model size. Code is available at https://github.com/xwmaxwma/TinyViM.
Contrastive Language-Image Pretraining (CLIP) excels at learning generalizable image representations but often falls short in zero-shot inference on certain downstream datasets. Test-time adaptation (TTA) mitigates this issue by adjusting components like normalization layers or context prompts, yet it typically requires large batch sizes and extensive augmentations, leading to high computational costs. This raises a key question: Can VLMs' performance drop in specific test cases be mitigated through efficient, training-free approaches? To explore the solution, we investigate token condensation (TC) techniques, originally designed to enhance vision transformer efficiency by refining token usage during inference. We observe that informative tokens improve visual-text alignment in VLMs like CLIP on unseen datasets. However, existing TC methods often fail to maintain in-distribution performance when reducing tokens, prompting us to ask: How can we transform TC into an effective "free-lunch" adaptation strategy for VLMs? To address this, we propose Token Condensation as Adaptation (TCA), a training-free adaptation method that takes a step beyond standard TC. Rather than passively discarding tokens, TCA condenses token representation by introducing reservoir-based domain anchor tokens for information-preserving token reduction and logit correction. TCA achieves up to a 21.4% performance improvement over the strongest baseline on cross-dataset benchmark and the CIFAR-100-Corrupted dataset while reducing GFLOPs by 12.2% to 48.9%, with minimal hyperparameter dependency on both CLIP and SigLIP series. Code is available at https://github.com/Jo-wang/TCA.
Errors often occur during human-robot interactions, such as failing to respond, interrupting users, or providing answers that do not meet user expectations. Detecting these issues on time is crucial for making human-robot communication more natural and user-friendly. In this technical report, we present the approach proposed by our team, CFM-HRI, for the ERR@HRI 2.0 Challenge 2025, targeting the task of interaction error detection. Specifically, we propose a lightweight and efficient time-series classification approach, empowered by cross-modal alignment, to detect interaction errors more accurately and promptly. To mitigate temporal misalignment across modalities, we adopt an upsampling alignment strategy, followed by feature fusion to obtain unified representations. A sliding-window voting mechanism is then introduced to construct training samples along with corresponding ground truth labels. Several machine learning models are employed to detect errors based on the fused features. Experimental results demonstrate the effectiveness of our approach in capturing cross-modal inconsistencies and improving detection accuracy. Our approach won first place in Sub-Challenge 2 and second place in Sub-Challenge 1 of the ERR@HRI 2.0 Challenge, held in conjunction with ACM MM 2025. We provide detailed descriptions of our data processing and experimental setup, along with an analysis of the limitations of our approach and potential directions for future work. Our code is available on https://github.com/setsaile/CFM-HRI-ERR-HRI2.0.
Reliable detection of conversational errors and user-initiated corrections is critical for effective human-robot interaction (HRI). In this study, we present a comprehensive multimodal approach leveraging temporal window processing, targeted feature engineering, and a MiniRocket + Ridge classification pipeline to address the challenges introduced by the ERR@HRI 2.0 dataset. Our methodology systematically integrates multimodal data streams, including facial expressions, acoustic features, and linguistic embeddings, to predict robot failures and user reactions. Experimental results demonstrate significant improvements over baseline models in event-level detection performance. Notably, linguistic features derived from transcript embeddings emerged as the most informative modality, substantially enhancing model performance. However, we observed challenges associated with managing false positives at the event level, suggesting avenues for future refinement in adaptive thresholding and sequential post-processing techniques. Our findings underscore the importance of careful feature selection and robust temporal modelling in developing effective real-time error detection systems for conversational robots. Our code is available online. https://github.com/Ruddy202/err-hri-2.0-armas.git.
Multivariate time-series (MTS) classification tasks play a key role in data-driven applications spanning healthcare, finance, and mobile communication. As MTS data are typically collected from multiple interdependent sensors, the resulting temporal patterns inherently reflect the characteristics of the underlying sensing systems. Despite this connection, conventional MTS classification models predominantly focus on raw time-series data while disregarding valuable sensor-specific prior knowledge, which fundamentally constrains their classification accuracy. The emergence of large language models (LLMs) has encoded extensive sensor-related knowledge within their parameter spaces. However, effectively harnessing such knowledge to enhance MTS classification networks remains an open challenge. To address this, we propose Foresail, a status-guided neural framework that bridges this gap through systematic integration of LLM-derived sensor knowledge via the status relationship matrix and fine-grained status labels. Foresail can be seamlessly integrated with existing MTS networks to optimize performance and generate interpretable intermediate results. Experiments on irregularly and regularly sampled MTS data demonstrate that Foresail outperforms state-of-the-art approaches, achieving a notable improvement in F1-score of up to 10.9% compared to the basic MTS network.
Time series analysis is crucial in diverse scenarios. Beyond forecasting, considerable real-world tasks are categorized into classification, imputation, and anomaly detection, underscoring different capabilities termed time series understanding in this paper. While GPT-style models have been positioned as foundation models for time series forecasting, the BERT-style architecture, which has made significant advances in natural language understanding, has not been fully unlocked for time series understanding, possibly attributed to the undesirable dropout of essential elements of BERT. In this paper, inspired by the shared multi-granularity structure between multivariate time series and multisentence documents, we design TimesBERT to learn generic representations of time series including temporal patterns and variate-centric characteristics. In addition to a natural adaptation of masked modeling, we propose a parallel task of functional token prediction to embody vital multi-granularity structures. Our model is pre-trained on 260 billion time points across diverse domains. Leveraging multi-granularity representations, TimesBERT achieves state-of-the-art performance across four typical downstream understanding tasks, outperforming task-specific models and language pre-trained backbones, positioning it as a versatile foundation model for time series understanding.
Real-world time series typically exhibit complex temporal variations, making the time series classification task notably challenging. Recent advancements have demonstrated the potential of multi-scale analysis approaches, which provide an effective solution for capturing these complex temporal patterns. However, existing multi-scale analysis-based time series prediction methods fail to eliminate redundant scale-shared features across multi-scale time series, resulting in the model over- or under-focusing on scale-shared features. To address this issue, we propose a novel end-to-end Disentangled Multi-Scale framework for Time Series classification (DisMS-TS). The core idea of DisMS-TS is to eliminate redundant shared features in multi-scale time series, thereby improving prediction performance. Specifically, we propose a temporal disentanglement module to capture scale-shared and scale-specific temporal representations, respectively. Subsequently, to effectively learn both scale-shared and scale-specific temporal representations, we introduce two regularization terms that ensure the consistency of scale-shared representations and the disparity of scale-specific representations across all temporal scales. Extensive experiments conducted on multiple datasets validate the superiority of DisMS-TS over its competitive baselines, with the accuracy improvement up to 9.71%.
Human activity recognition (HAR) is an evolving technique that offers innovative solutions across various domains, such as healthcare, sports training, and human-computer interactions. This paper addresses the novel challenge of video-based activity recognition, focusing on detecting and classifying athletes' actions to enable precision sports training. Conventional HAR methods based on direct video analysis incur excessive computational overhead and constrained applicability. In contrast, our novel transformer-based framework, namely RSFomer, converts videos into multivariate time series, and then detects and classifies the athletes' actions. However, sports videos often suffer from severe occlusion, which introduces significant noise to the converted time series and thus deteriorates recognition performance. To address this challenge, we implement several innovative strategies to improve the robustness of our framework. First, we propose a dual-scale filtering mechanism that leverages the unscented Kalman filter and kinematic constraints to reduce noise and outliers in the converted time series. Second, we incorporate the masking mechanism and temporal slicing mechanism to enhance the transformer's ability to handle anomalies and extract multi-scale features for accurate action recognition. We perform extensive evaluations on our Boxing dataset as well as the UEA and FineGym datasets. The results demonstrate that our RSFomer is effective, outperforming existing state-of-the-art methods with significant advantages.
Medical time series, such as Electroencephalogram (EEG) and Electrocardiogram (ECG), are widely used for disease detection, with multiple electrodes or sensors recording simultaneously. Accurately modeling inter-channel relationships is crucial for improving detection performance. Current methods mainly rely on data-driven approaches to model channel relationships, facing two challenges: (1) insufficient integration of medical prior knowledge, hindering the accurate representation of physiological correlations between channels, and (2) high temporal pattern similarity across channels, leading to feature redundancy and degraded classification performance. To address these issues, we introduce KEMed, a knowledge-augmented model for medical time series classification. The model incorporates medical textual prior knowledge by generating natural language descriptions for each channel and leveraging Pre-trained Language Model (PLM) for semantic representation, enabling precise identification of physiological and pathological similarities and differences between channels. Specifically, KEMed optimizes channel relationships through knowledge-guided clustering and weighting mechanisms and leverages Large Language Model (LLM) to capture spatiotemporal dependencies, thereby enhancing classification performance. Experimental results on five medical time series datasets demonstrate that KEMed consistently outperforms state-of-the-art methods, validating the effectiveness and superiority of knowledge augmentation in medical time series classification.
Electroencephalogram (EEG) signal classification faces significant challenges due to data distribution shifts caused by heterogeneous electrode configurations, acquisition protocols, and hardware discrepancies across domains. This paper introduces IMAC, a novel channel-dependent mask and imputation self-supervised framework that formulates the alignment of cross-domain EEG data shifts as a spatial time series imputation task. To address heterogeneous electrode configurations in cross-domain scenarios, IMAC first standardizes different electrode layouts using a 3D-to-2D positional unification mapping strategy, establishing unified spatial representations. Unlike previous mask-based self-supervised representation learning methods, IMAC introduces spatio-temporal signal alignment. This involves constructing a channel-dependent mask and reconstruction task framed as a low-to-high resolution EEG spatial imputation problem. Consequently, this approach simulates cross-domain variations such as channel omissions and temporal instabilities, thus enabling the model to leverage the proposed imputer for robust signal alignment during inference. Furthermore, IMAC incorporates a disentangled structure that separately models the temporal and spatial information of the EEG signals separately, reducing computational complexity while enhancing flexibility and adaptability. Comprehensive evaluations across 10 publicly available EEG datasets demonstrate IMAC's superior performance, achieving state-of-the-art classification accuracy in both cross-subject and cross-center validation scenarios. Notably, IMAC shows strong robustness under both simulated and real-world distribution shifts, surpassing baseline methods by up to 35% in integrity scores while maintaining consistent classification accuracy.
Volumetric video enables immersive experiences by capturing dynamic 3D scenes, enabling diverse applications for virtual reality, education, and telepresence. However, traditional methods struggle with fixed lighting conditions, while neural approaches face trade-offs in efficiency, quality, or adaptability for relightable scenarios. To address these limitations, we present BEAM, a novel pipeline that bridges 4D Gaussian representations with physically-based rendering (PBR) to produce high-quality, relightable volumetric videos from multi-view RGB footage. BEAM recovers detailed geometry and PBR properties via a series of available Gaussian-based techniques. It first combines Gaussian-based human performance tracking with geometry-aware rasterization in a coarse-to-fine optimization framework to recover spatially and temporally consistent geometries. We further enhance Gaussian attributes by incorporating PBR properties step by step. We generate roughness via a multi-view-conditioned diffusion model, and then derive AO and base color using a 2D-to-3D strategy, incorporating a tailored Gaussian-based ray tracer for efficient visibility computation. Once recovered, these dynamic, relightable assets integrate seamlessly into traditional CG pipelines, supporting real-time rendering with deferred shading and offline rendering with ray tracing. By offering realistic, lifelike visualizations under diverse lighting conditions, BEAM opens new possibilities for interactive entertainment, storytelling, and creative visualization.
The domain of time series forecasting has gained significant attention due to its critical applications in multimedia-rich web traffic (including video streaming workloads and dynamic content delivery) and cross-platform advertisement click predictions, which are essential for web operations planning. While models like TimeSieve have demonstrated strong capabilities in predicting web visitation metrics, they suffer from critical unfaithfulness issues, including sensitivity to random seeds, input noise, layer noise, and parametric perturbations. To address these limitations, we propose Faithful TimeSieve (FTS), an enhanced framework designed to improve prediction reliability and robustness. Our approach systematically detects and mitigates unfaithfulness in TimeSieve, significantly enhancing its stability and consistency. Experimental results demonstrate that FTS substantially improves the model's faithfulness, setting a new standard for temporal forecasting methods. This advancement not only increases TimeSieve's reliability but also contributes to more robust temporal modeling, particularly crucial for web traffic forecasting where prediction accuracy directly impacts operational decisions. Our work thus represents a significant step toward more dependable time series predictions in web-related applications.
Accessibility of multimedia content for all users, particularly blind and low-vision individuals (BLVIs), remains a significant challenge. While screen readers assist BLVIs by converting text to speech via Alt-Text and image descriptions, these methods are inherently text-based and struggle to convey spatial and graphical information effectively. To help this, we propose a framework that converts graphical components into tactile graphics rendered on a refreshable pin array. Our framework leverages on-device AI models to generate tactile representations without transmitting personal data. It thereby minimizes processing time and mitigates privacy concerns. The benchmark test showed that our on-device AI outperformed GPU servers (RTX 4090) operating in an intranet environment. To optimize the tactile output and evaluate the system's effectiveness on media accessibility, we conducted a series of user studies with three different use case scenarios. First, we derived the optimal threshold values for edge detection in tactile graphics, which resulted in 70 on a 0-255 scale. Then, we compared the proposed system to a vision language model (VLM; GPT-4o). The results indicated that our proposed framework is more effective regarding both information delivery and subjective satisfaction. The proposed framework can be directly applied to several visual media accessibility scenarios, with the benefits of using local AI, such as privacy protection, personalization, and cost-effectiveness.
Electrocardiogram (ECG), a key bioelectrical time-series signal, is crucial for assessing cardiac health and diagnosing various diseases. Given its time-series format, ECG data is often incorporated into pre-training datasets for large-scale time-series model training. However, existing studies often overlook its unique characteristics and specialized downstream applications, which differ significantly from other time-series data, leading to an incomplete understanding of its properties. In this paper, we present an in-depth investigation of ECG signals and establish a comprehensive benchmark, which includes (1) categorizing its downstream applications into four distinct evaluation tasks, (2) identifying limitations in traditional evaluation metrics for ECG analysis, and introducing a novel metric; (3) benchmarking state-of-the-art time-series models and proposing a new architecture. Extensive experiments demonstrate that our proposed benchmark is comprehensive and robust. The results validate the effectiveness of the proposed metric and model architecture, which establish a solid foundation for advancing research in ECG signal analysis.
Finance decision-making often relies on in-depth data analysis across various data sources, including financial tables, news articles, stock prices, etc. In this work, we introduce FINTMMBench, the first comprehensive benchmark for evaluating temporal-aware multi-modal Retrieval-Augmented Generation (RAG) systems in finance. Built from heterologous data of NASDAQ 100 companies, FINTMMBench offers three significant advantages. 1) Multi-modal Corpus: It encompasses a hybrid of financial tables, news articles, daily stock prices, and visual technical charts as the corpus. 2) Temporal-aware Questions: Each question requires the retrieval and interpretation of its relevant data over a specific time period, including daily, weekly, monthly, quarterly, and annual periods. 3) Diverse Financial Analysis Tasks: The questions involve 10 different financial analysis tasks designed by domain experts, including information extraction, trend analysis, sentiment analysis and event detection, etc. We further propose a novel TMMHybridRAG method, which first leverages a multi-modal LLM to convert data from other modalities (e.g., tabular, visual and time-series data) into textual format and then incorporates temporal information in each node when constructing graphs and dense indexes. Its effectiveness has been validated in extensive experiments, but notable gaps remain, highlighting the challenges presented by our FINTMMBench. The benchmark and source code will be made publicly available.
Dynamic Facial Expression Recognition(DFER) is a rapidly evolving field of research that focuses on the recognition of time-series facial expressions. While previous research on DFER has concentrated on feature learning from a deep learning perspective, we put forward an AU-enhanced Dynamic Facial Expression Recognition architecture, namely AU-DFER, that incorporates AU-expression knowledge to enhance the effectiveness of deep learning modeling. In particular, the contribution of the Action Units(AUs) to different expressions is quantified, and a weight matrix is designed to incorporate a priori knowledge. Subsequently, the knowledge is integrated with the learning outcomes of a conventional deep learning network through the introduction of AU loss. The design is incorporated into the existing optimal model for dynamic expression recognition for the purpose of validation. Experiments are conducted on three recent mainstream open-source approaches to DFER on the principal datasets in this field. The results demonstrate that the proposed architecture outperforms the State-Of-The-Art(SOTA) methods without the need for additional arithmetic and generally produces improved results. Furthermore, we investigate the potential of AU loss function redesign to address data label imbalance issues in established dynamic expression datasets. To the best of our knowledge, this is the first attempt to integrate quantified AU-expression knowledge into various DFER models. We also devise strategies to tackle label imbalance, or minor class problems. Our findings suggest that employing a diverse strategy of loss function design can enhance the effectiveness of DFER. This underscores the criticality of addressing data imbalance challenges in mainstream datasets within this domain. The source code is available at https://github.com/Cross-Innovation-Lab/AU-DFER.
The continuous advancement of multimedia technology has led to the exponential accumulation of massive time-stamped data. However, accurately identifying anomalies in such data remains a major challenge. Current anomaly detection methods still face serious limitations, including the difficulty in handling complex time series data and the anomaly masking phenomenon caused by overlapping temporal patterns. Existing methods cannot effectively address these challenges. To overcome these limitations, we propose an unsupervised time series anomaly detection algorithm DMemAD based on a dual-domain memory module. Specifically, we design an STD Mamba structure that can effectively extract trend and seasonal components in the series and enhance the connection between elements in each component through bidirectional learning. Second, we design a dual-domain memory module to avoid anomaly masking by independently storing trend and seasonal patterns. Additionally, we propose a residual-based memory update mechanism to enhance the accuracy of memory updates, ensuring that prototype patterns are stored precisely. Extensive experiments on four datasets from different domains show that DMemAD achieves an average F1 score of 96.81%, outperforming 17 baseline methods and establishing state-of-the-art performance.
Commanding robots to do chores using natural language instructions has been a dream of us for a long time. The navigation capability, as one of the key foundational abilities to achieve this goal, has garnered significant attention in this regard. When human users instruct intelligent agent, the instructions they given sometimes exhibit slight discrepancies from navigable ones, as user's understanding of scene may not be up-to-date due to instant change of environments. This paper investigates 3 common scenarios where instructions and navigation scenes are imperfectly aligned: change of navigability, incorrect landmark references, and incorrect direction descriptions. We then propose an ImperfectVLN task and dataset for evaluating an agent's navigation performance under instruction and environment imperfectly matched conditions. Evaluation results indicate significant performance fluctuations in existing state-of-the-art models under modification scenarios including referred landmark removal and original path blockages. We also provide a series of result analyses and further insights. We aim for this new dataset to become a valuable benchmark, enhancing practical VLN tasks. We further design a reflection module based on our insights, allowing an agent to review its history and identify potential errors. Experiments show that this module improves the performance on ImperfectVLN by 4.4%.
Embodied artificial intelligence has rapidly developed under the impetus of multimodal learning, robotics, and cognitive science, demonstrating great potential in fields such as navigation and manipulation. However, building embodied agents that can robustly operate in diverse and dynamic environments still faces challenges, such as handling partial observability and environmental adaptability. Multimodal large language models (MLLMs) are vital for embodied intelligence due to their ability to process multimodal information, but they encounter difficulties in understanding spatial environments and performing dynamic decisions and evolution. Inspired by the functional specialization of the left and right hemispheres of the human brain, this paper proposes a brain-inspired learning and evolution paradigm for embodied agents. The method designs an embodied context-augmented MLLM to simulate the language processing and logical analysis capabilities of the left hemisphere, responsible for understanding instructions and visual scenes. At the same time, it constructs a perceptual context-guided world model based on the recurrent state space model to simulate the spatial perception and holistic thinking functions of the right hemisphere, capturing environmental dynamics and predicting future states. By simulating the communication function of the corpus callosum, we propose dynamic communication slots for efficient information exchange between MLLMs and the world model, which also allows the agent to quickly adapt to dynamic environments without requiring extensive computational resources. Experiments show that the proposed paradigm significantly improves the performance of embodied agents in a series of tasks and enhances their generalization ability in zero-shot tasks through embodied exploration experience and online evolution. Our project page is available at https://feliciaxyao.github.io/EvoAgent/.