Human activity recognition (HAR) is an evolving technique that offers innovative solutions across various domains, such as healthcare, sports training, and human-computer interactions. This paper addresses the novel challenge of video-based activity recognition, focusing on detecting and classifying athletes' actions to enable precision sports training. Conventional HAR methods based on direct video analysis incur excessive computational overhead and constrained applicability. In contrast, our novel transformer-based framework, namely RSFomer, converts videos into multivariate time series, and then detects and classifies the athletes' actions. However, sports videos often suffer from severe occlusion, which introduces significant noise to the converted time series and thus deteriorates recognition performance. To address this challenge, we implement several innovative strategies to improve the robustness of our framework. First, we propose a dual-scale filtering mechanism that leverages the unscented Kalman filter and kinematic constraints to reduce noise and outliers in the converted time series. Second, we incorporate the masking mechanism and temporal slicing mechanism to enhance the transformer's ability to handle anomalies and extract multi-scale features for accurate action recognition. We perform extensive evaluations on our Boxing dataset as well as the UEA and FineGym datasets. The results demonstrate that our RSFomer is effective, outperforming existing state-of-the-art methods with significant advantages.
论文检索
输入标题、作者或关键词，从 1,620 篇学术成果中精准定位
Immersive telepresence aims to authentically reproduce remote physical scenes, enabling the experience of real-world places, objects and people over large geographic distances. This requires the ability to generate realistic novel views of the scene with low latency. Existing methods either depend on depth data from specialized hardware setups or precomputed templates such as human models, which severely restrict their practicality and generalization to diverse scenes. To address these challenges, we introduce RIFTCast, a real-time template-free volumetric reconstruction framework that synthesizes high-fidelity dynamic scenes from a multi-view RGB-only capture setup. The framework is specifically targeted at the efficient reconstruction, transmission and visualization of complex scenes, including extensive human-human and human-object interactions. For this purpose, our method leverages a GPU-accelerated client-server pipeline that computes a visual hull representation to select a suitable subset of images for novel view synthesis, substantially reducing bandwidth and computation demands. This lightweight architecture enables deployment from small-scale configurations to sophisticated multi-camera capture stages, achieving low-latency telepresence even on resource-constrained devices. For evaluation, we provide a comprehensive high-quality multi-view video data benchmark as well as our reconstruction and rendering code, including tools for loading and processing a variety of data input formats, to facilitate future telepresence research.
Road Surface Reconstruction (RSR) is crucial for autonomous driving, enabling the understanding of road surface conditions. Recently, RSR from the Bird's Eye View (BEV) has gained attention for its potential to enhance performance. However, existing methods for transforming perspective views to BEV face challenges such as information loss and representation sparsity. Moreover, stereo matching in BEV is limited by the need to balance accuracy with inference speed. To address these challenges, we propose two efficient and accurate BEV-based RSR models: FastRSR-mono and FastRSR-stereo. Specifically, we first introduce Depth-Aware Projection (DAP), an efficient view transformation strategy designed to mitigate information loss and sparsity by querying depth and image features to aggregate BEV data within specific road surface regions using a pre-computed look-up table. To optimize accuracy and speed in stereo matching, we design the Spatial Attention Enhancement (SAE) and Confidence Attention Generation (CAG) modules. SAE adaptively highlights important regions, while CAG focuses on high-confidence predictions and filters out irrelevant information. FastRSR achieves state-of-the-art performance, exceeding monocular competitors by over 6.0% in elevation absolute error and providing at least a 3.0× speedup by stereo methods on the RSRD dataset. The source code will be released.
Large Video Language Models (LVLMs) have rapidly emerged as the focus of multimedia AI research. Nonetheless, when confronted with lengthy videos, these models struggle: their temporal windows are narrow, and they fail to notice fine-grained semantic shifts that unfold over extended durations. Moreover, mainstream text-based retrieval pipelines, which rely chiefly on surface-level lexical overlap, ignore the rich temporal interdependence among visual, audio, and subtitle channels. To mitigate these limitations, we propose TV-RAG, a training-free architecture that couples temporal alignment with entropy-guided semantics to improve long-video reasoning. The framework contributes two main mechanisms: (i) a time-decay retrieval module that injects explicit temporal offsets into the similarity computation, thereby ranking text queries according to their true multimedia context; and (ii) an entropy-weighted key-frame sampler that selects evenly spaced, information-dense frames, reducing redundancy while preserving representativeness. By weaving these temporal and semantic signals together, TV-RAG realises a dual-level reasoning routine that can be grafted onto any LVLM without re-training or fine-tuning. The resulting system offers a lightweight, budget-friendly upgrade path and consistently surpasses most leading baselines across established long-video benchmarks such as Video-MME, MLVU, and LongVideoBench, confirming the effectiveness of our model.
We study how large vision models (LVMs) can predict food nutrition through lightweight and interpretable adapters---the machine learning modules the predictions of which could be understood by humans. We introduce novel nutrition adapters that use features extracted by pre-trained LVMs and output the so-called nutrition maps. Nutrition maps indicate the concentration of nutrition values per each image location. We use such an interpretable representation to obtain the nutrition targets as a sum of all nutrition concentrations on the maps. To understand our approach's generalization capability, we systematically analyze the behavior of our novel interpretable adapters leveraging different LVMs with different food image-nutrition datasets. Our lightweight approach delivers better or on-par performance than the state-of-the-art models on the Nutrition5k and the Nutritionverse-Real benchmarks. The code is provided at https://github.com/vitaly-emelianov/nutrition-adapters.
Text-to-Visualization (Text2Vis) generates data visualizations directly from natural language queries, democratizing access to data insights. Early Text2Vis efforts, primarily relying on rule-based systems and machine learning models, struggled to handle semantically intricate queries. The advent of large language models (LLMs) allows for better generalization in generating visualization code. However, LLM-based approaches have mainly focused on textual or code-level optimizations, neglecting the potential benefits of assessing and improving visualized charts. Hence, we propose Visualization Refinement (VisRef), a novel framework based on vision-language models (VLMs) to enhance Text2Vis outputs. (1) Knowledge Extraction -- VisRef extracts visualization assessment knowledge through a hierarchical contrastive prompt and multi-granularity quality assessment framework by comparing superior ground-truth charts with inferior Text2Vis outputs; and (2) VLM Fine-Tuning -- This knowledge is used to fine-tune a VLM through a two-stage approach, including warm-up and iterative preference alignment phases, to judge visualization quality and provide code-level refinement suggestions. Experimental results demonstrate that VisRef significantly outperforms state-of-the-art approaches, including LLM-based and VLM-prompted, and exhibits strong orthogonal compatibility with existing approaches.
End-to-end automated fact-checking (AFC) aims to assess the truthfulness of claims using retrieved evidence. Some researchers use crawlers or search APIs to retrieve evidence from the web for veracity classification. However, existing methods indiscriminately rely on the retrieved evidence and overlook that the retrieved results are not always reliable. This unilateral reliance on evidence significantly hampers the performance of fact-checking. In this paper, we account for the diverse reliability levels of retrieved evidence and eliminate the negative impact from the causal perspective. To achieve our goal, we propose a novel Causal intervention and Counterfactual reasoning based Multi-Checker framework (CCMC), which introduces two additional counterfactual fact-checkers to verify claims from the counterfactual perspective. Specifically, we construct two distinct types of counterfactual instances via causal intervention to imitate the situation where the evidence is partially reliable or totally unreliable. Correspondingly, two counterfactual fact-checkers are trained with tailored counterfactual instances by counterfactual reasoning. During inference, the two counterfactual fact-checkers are employed to estimate and eliminate the potential impact of unreliable evidence. Extensive experiments on two real-world datasets demonstrate the superiority of our approach for improving end-to-end AFC. Especially, we surpass existing methods by 3.70% and 5.55% under gold and system evidence on the MOCHEG benchmark, respectively. Our code is available at https://github.com/BeiyuXuboL/CCMC.
Flexible object recognition remains challenging in multimedia scenarios due to inherently diverse shapes and sizes, and subtle inter-class differences. Graph-based vision models show promise in flexible objects recognition by capturing variable relationships. However, they suffer from two problems: (1) inter-class ambiguity hinders model discrimination and (2) frequent scale changes degrade model generalization. To address these limitations, we propose a unified graph distillation framework that enhances inter-class discrimination and spatial generalization while maintaining computational efficiency. For inter-class ambiguity problem, we introduce a virtual prototype module that dynamically generates learnable class prototypes via clustering intermediate features. These prototypes are incorporated into the distillation loss to sharpen decision boundaries. A global-local distillation mechanism further capture both image-level global semantics and patch-level local details, enhancing inter-class discrimination. For frequent scale changes problem, we design a patch-aware distillation strategy that transfers knowledge across multiple patch scales, strengthening the student model's spatial generalization to match various shapes and sizes of flexible objects, thus alleviate generalization degradation. Extensive experiments on flexible-object datasets (FDA, FSCW, CCSN) and challenging benchmarks (CIFAR-100, Mini-ImageNet) confirm effectiveness and efficiency of our method.
Existing action counting methods typically rely on pixel-based changes within videos, leading to high computational redundancy and low accuracy due to the limited spatial sensitivity. To address these challenges, we introduce MoCount, the first framework that leverages 3D motion representations for counting tasks. MoCount significantly reduces computational overhead and improves counting accuracy, benefiting from the simplicity of motion representation and strong spatial sensitivity. Specifically, we utilize a motion estimator to convert video subjects into 3D motion data. A motion encoder, combined with a Sparse Spatial-Temporal module, is then applied to extract robust human body representations, yielding precise counting results. Extensive experiments on the RepCount and UCFRep datasets show that MoCount achieves state-of-the-art performance, reducing inference latency by approximately 2-3 times compared to existing video counting models. These advantages position MoCount as a leading solution for real-world action counting applications.
Diffusion-based super-resolution methods have achieved impressive results under normal lighting conditions. However, their performance in low-light scenarios faces fundamental limitations due to two inherent challenges. First, the characteristic noise patterns and complex degradation features in severely underexposed images create significant obstacles for diffusion models to establish reliable noise prediction mechanisms. Second, these methods often fail to establish effective coupling between the degradation priors of low-light observations and the reconstruction process, resulting in compromised detail recovery and unrealistic texture synthesis.To address these limitations, we propose Degradation-aware Adaptation with Representation Embedding (DARE) method, a novel one-step diffusion framework specifically designed for super-resolution in dark environments. DARE employs a degradation-aware low-rank adaptation strategy that dynamically adjusts model parameters conditioned on degradation-specific features, effectively addressing compound degradations such as low-light, blur, and noise. Furthermore, we introduce a content-sensitive representation embedding mechanism, integrating complementary spatial and frequency domain priors through a bilinear cross-attention module. This module explicitly captures second-order statistical correlations, enriching semantic understanding and detail recovery during the denoising process. Extensive experiments across diverse low-light scenarios demonstrate that DARE outperforms state-of-the-art methods in terms of both visual quality and perceptual accuracy. The code is available at https://github.com/csmty/DARE.
Conventional RGB cameras struggle in high-speed vision due to motion blur (above 60Hz sampling) and limited dynamic range (<60dB). To address these limitations, we propose a multimodal framework integrating event cameras, leveraging their microsecond temporal resolution (1μs) and 140dB dynamic range. Our key innovations include: (1) DSF-Net: An innovative spike-triggered dynamic sparse fusion network that effectively and efficiently fuses discriminative features from Event-RGB, enabling high-speed object detection; (2) HS-Multi: The first large-scale Event-RGB dataset specifically designed for high-speed objects, featuring 73k annotated samples across 11 object categories, with dedicated high-speed settings (HS-CAR and HS-FAN). Extensive evaluations on three benchmarks (HS-CAR, HS-FAN, PKU-DDD17-Car) demonstrate consistent advantages: (a) superiority on high-speed detection, DSF-Net significantly surpasses both unimodal (RGB/Event) and existing multimodal fusion methods, with particular superiority on HS-CAR where it achieves 87.3% mAP (9.5%↑ vs. RGB-only); (b) Generalization capability: DSF-Net achieves 50.1% mAP on PKU-DDD17-Car, surpassing prior multimodal framework in both accuracy (+4.1%) and speed (+8.3fps).
Remote sensing image classification with noisy labels is receiving increasing attention. However, the existing methods ignore the context information of the training sample and judge whether the label is a noise label only by monitoring the loss value of a single sample, which may lead to misjudgment of the sample label. Additionally, these algorithms do not consider constructing pairs of confidence instances to obtain robust potential representations after identifying confidence instances. In this paper, a Multi-view Collaborative Representation Learning (MCRL) approach from noisy labels is proposed to improve the classification performance of very high resolution (VHR) remote sensing images. Specifically, we design a correction strategy based on spatial consistency and confidence-aware mechanisms. This strategy quantitatively measures label reliability by mining the contextual information of labelled samples within the adaptive region. Leveraging the spatial consistency principle and the confidence-aware mechanism to correct and smooth the noisy labels progressively. Moreover, we construct confidence sample pairs by establishing relationships between samples within and between views to obtain robust latent representations, which improves the model's tolerance to noisy labels. Experiments show that the MCRL can significantly reduce the impact of noisy labels on the model and is more competitive than homologous algorithms.
This paper presents a novel approach to lifelogging by transforming it into a narrative-driven Virtual Reality (VR) experience that we call SLIVeR (Someone else's Lifelog in Virtual Reality). The system situates users in an immersive, emotionally charged environment where they explore lifelog video fragments as part of a memory recovery narrative. Users begin disoriented, responding to existential prompts such as 'Who am I?' and 'What was the last thing I did?' - each unlocking cinematic scenes that reconstruct a car accident central to the character's identity loss. The experience transitions into a metaphorical space representing the fragmented mind, where users interact with floating lifelog questions organised by theme (e.g., relationships, movement, work). These interactions simulate a lifelog search interface embedded within a story arc, encouraging reflection and engagement. To evaluate SLIVeR, we conducted a mixed-methods study with 30 participants, analysing engagement using the User Engagement Scale (UES) and thematic reflection questions. Results showed that narrative coherence, visual interactivity, and identity-driven content enhanced user engagement. Social and emotionally resonant question types were rated as more meaningful than routine-based ones. This work demonstrates how VR, narrative framing, and lifelog data can be fused into a reflective, game-like experience that deepens both interaction and emotional connection.
Medical time series, such as Electroencephalogram (EEG) and Electrocardiogram (ECG), are widely used for disease detection, with multiple electrodes or sensors recording simultaneously. Accurately modeling inter-channel relationships is crucial for improving detection performance. Current methods mainly rely on data-driven approaches to model channel relationships, facing two challenges: (1) insufficient integration of medical prior knowledge, hindering the accurate representation of physiological correlations between channels, and (2) high temporal pattern similarity across channels, leading to feature redundancy and degraded classification performance. To address these issues, we introduce KEMed, a knowledge-augmented model for medical time series classification. The model incorporates medical textual prior knowledge by generating natural language descriptions for each channel and leveraging Pre-trained Language Model (PLM) for semantic representation, enabling precise identification of physiological and pathological similarities and differences between channels. Specifically, KEMed optimizes channel relationships through knowledge-guided clustering and weighting mechanisms and leverages Large Language Model (LLM) to capture spatiotemporal dependencies, thereby enhancing classification performance. Experimental results on five medical time series datasets demonstrate that KEMed consistently outperforms state-of-the-art methods, validating the effectiveness and superiority of knowledge augmentation in medical time series classification.
Unsupervised remote sensing dehazing remains a challenging and ill-posed task due to the absence of reliable supervision signals. Existing dehazing methods with unpaired data often oversimplify haze removal as style transfer, limiting generalization in complex scenarios. Moreover, current unimodal frameworks neglect cross-modal cues that could improve contextual reasoning. To address these issues, we propose a novel cross-modal guided self-supervised dehazing framework called CLIP-HNet, which achieves multi-model feature extraction, boundary-focused reconstruction and adaptive sample filtering. Specifically, to capture global-local contextual features, a hybrid feature interaction network is designed, which bridges the feature representations of multi models with global context-aware module (GCAM) and hybrid feature fusion module (HF2 M). Then, based on the hybrid features, a boundary-aware feature reconstruction (BFRec) is proposed to further refine edge details. Furthermore, a CLIP-guided progressive information distillation scheme is presented to dynamically prioritize training samples and distill useful signals, which predicts haze concentration by CLIP and progressively increases sample difficulty during the training stage. Finally, a frequency-domain texture matching (FTM) strategy refines texture and spectral details, enhancing the model's ability to recover fine details. Experiments on synthetic and real RSIs demonstrate that the proposed CLIP-HNet surpasses state-of-the-art approaches, achieving superior visual quality and quantitative performance.
In this paper, we propose a new method called Multi-Modal Gradual Domain Osmosis, which aims to solve the problem of smooth knowledge migration from the source domain to the target domain in Gradual Domain Adaptation (GDA). Traditional Gradual Domain Adaptation methods mitigate domain bias by introducing intermediate domains and self-training strategies but often face the challenges of inefficient knowledge migration or missing data in intermediate domains. In this paper, we design an optimization framework based on the hyperparameter łambda by dynamically balancing the loss weights of the source and target domains, which enables the model to progressively adjust the strength of knowledge migration (łambda incrementing from 0 to 1) during the training process, thus achieving cross-domain generalization more efficiently. Specifically, the method incorporates self-training to generate pseudo-labels and iteratively updates the model by minimizing a weighted loss function to ensure stability and robustness during progressive adaptation in the intermediate domain. The experimental part validates the effectiveness of the method on rotated MNIST, color-shifted MNIST, portrait dataset, and forest cover type dataset, and the results show that it outperforms existing baseline methods. The paper further analyses the impact of the dynamic tuning strategy of the hyperparameter łambda on the performance through ablation experiments, confirming the advantages of progressive domain penetration in mitigating domain bias and enhancing the model generalization capability. The study provides theoretical support and a practical framework for asymptotic domain adaptation and expands its application potential in dynamic environments.
Graph classification is a fundamental machine learning problem with extensive applications in multimedia and biochemical analysis. Contemporary graph classification models usually require precise graph labels for supervision, even after self-supervised pre-training. However, in practical applications, the extensive precise annotation of graphs could be expensive or impractical. To exploit data efficiently, this work studies partial label graph learning, in which each graph is linked to a set of candidate labels but only one of them is accurate. Label ambiguity would bring difficulties in extracting graph semantics and the risk of overfitting noisy partial labels. Here, we present a novel approach called Coupled Dual Separation (CODE). To improve graph semantics mining under label ambiguity, our CODE contains a message passing branch and a graph kernel branch, which explore graph semantics implicitly and explicitly, respectively. To facilitate information exchange, we utilize one branch to separate partially labeled graphs into an informative set and an uninformative set, which provides guidance for the optimization of the other branch. Furthermore, to mitigate the risk of overfitting, parameters in coupled branches are partitioned into critical and non-critical ones for separated optimization procedures. Extensive experiments on several benchmark datasets validate the effectiveness of the proposed CODE.
Traditional ship detection methods primarily rely on single-modal approaches, such as visible or infrared images, which limit their application in complex scenarios involving varying lighting conditions and heavy fog. To address this issue, we explore the advantages of short-wave infrared (SWIR) and long-wave infrared (LWIR) in ship detection and propose a novel single-stage image fusion detection algorithm called LSFDNet. This algorithm leverages feature interaction between the image fusion and object detection subtask networks, achieving remarkable detection performance and generating visually impressive fused images. To further improve the saliency of objects in the fused images and improve the performance of the downstream detection task, we introduce the Multi-Level Cross-Fusion (MLCF) module. This module combines object-sensitive fused features from the detection task and aggregates features across multiple modalities, scales, and tasks to obtain more semantically rich fused features. Moreover, we utilize the position prior from the detection task in the Object Enhancement (OE) loss function, further increasing the retention of object semantics in the fused images. The detection task also utilizes preliminary fused features from the fusion task to complement SWIR and LWIR features, thereby enhancing detection performance. Additionally, we have established a Nearshore Ship Long-Short Wave Registration (NSLSR) dataset to train effective SWIR and LWIR image fusion and detection networks, bridging a gap in this field. We validated the superiority of our proposed single-stage fusion detection algorithm on two datasets. The source code and dataset are available at https://github.com/Yanyin-Guo/LSFDNet.
Phase distribution plays a critical role in various holographic applications; it affects the randomness within phase holograms, which, in turn, influences the viewing angle, speckle noise, and compression efficiency of the hologram. It also impacts the accuracy of measuring distortion between two phase holograms. Unlike ordinary image signals, uniquely determining the probability distribution of a phase hologram can be challenging due to the inherent periodicity of phase values. Although this ambiguity could impair the performance of many holographic applications, it has received little attention in prior works. In this paper, we introduce the Phase Distribution Alignment (PDA) method, which can transform a phase hologram so that its phase distribution is centered at a predefined reference point, resolving the ambiguity without affecting the numerical reconstruction result. Furthermore, when provided with a source and target phase hologram, PDA can align the source to maximize the overlap with the target distribution. We demonstrate the practical benefits of PDA in a range of holographic applications, including neural phase hologram generation, compression, and phase-domain distortion measurement.
This paper studies the problem of graph out-of-distribution generalization, which aims to enhance the performance of graph neural networks (GNNs) under distribution shifts. Existing approaches usually learn graph representations from a casual graph, which may not explicitly utilize environment information explicitly. Furthermore, they could suffer from performance degradation when confusing semantics related to target labels and environments. In this paper, we propose a novel approach named Dual Prompt Learning with Information Bottleneck (DATE) for graph out-of-distribution generalization. The core of our DATE is to utilize dual prompts to extract task-oriented semantics and model distribution shifts, respectively. In particular, we first pre-train a GNN using contrastive learning with pretext tokens introduced. More importantly, we not only introduce a task-oriented prompt based on LLMs to generate environment-invariant representations, but also learn the environment-oriented prompts to simulate subgraphs in different environments. To optimize our prompts, we introduce a graph information bottleneck framework, which minimizes the mutual information between environment-invariant representations and environment semantics with the most semantics preserved. Extensive experiments on various benchmark datasets validate the effectiveness of our DATE against various state-of-the-art approaches.