In the literature, prior studies on Video Anomaly Detection (VAD) primarily focus on anomalies that have already occurred (i.e., consequential anomaly), but cannot identify the causative anomalies (i.e., the cause of final anomaly), while this type of causative anomaly could be powerfully beneficial to early warning against the anomalies. Meanwhile existing work mainly focuses on classifying whether each video clip is abnormal, and couldn't extract structured video information, such as what is the abnormal type, which people or things are involved, whereas such structured information can potentially contribute to building an efficient system to monitor the above causative and consequential anomalies. To this end, this paper proposes a new chat-paradigm Video Abnormal Events' Early Warning (VAE-EW) task, aiming to localize and extract not only the consequential abnormal event quadruples but also the causative abnormal event quadruples (i.e., subject, predicate, object, and event type). Further, this paper believes that this new task faces two key challenges, i.e., Spatial-temporal modeling challenge and temporal highlighting challenge. On this basis, this paper proposes a new Skynet-V1 Model with a spatial-temporal causal-enhanced Mixture-of-Expert (MoE) Framework, i.e., acting like Skynet in movie 'The Terminator' to track and early warn against abnormal events, for VAE-EW task. Specifically, this model designs a Spatial-temporal Aware MoE Block (SAMB) and a Causal-guided Temporal Enhancing Block (CTEB) to address the two challenges respectively. Extensive experiments on our VAE-EW dataset show the superiority of our model in localizing and extracting abnormal events, especially the causative events, compared to other advanced baseline models, highlighting the importance of the new VAE-EW task and the effectiveness of Skynet-V1 in addressing such task.
论文检索
输入标题、作者或关键词，从 1,620 篇学术成果中精准定位
Experiment commentary is crucial in describing the experimental procedures, delving into underlying scientific principles, and incorporating content-related safety guidelines. In practice, human teachers rely heavily on subject-specific expertise and invest significant time preparing such commentary. To address this challenge, we introduce the task of automatic commentary generation across multi-discipline scientific experiments. Current LMMs' ability to generate fine-grained and insightful experiment commentary remains largely under-explored. In this paper, we make the following contributions: (i) We construct ExpInstruct, the first dataset tailored for experiment commentary generation, featuring over 7 K step-level commentaries across 21 scientific subjects from 3 core disciplines. (ii) We propose ExpStar, an automatic experiment commentary generation model that leverages a retrieval-augmented mechanism to adaptively access, evaluate, and utilize external knowledge. (iii) Extensive experiments show that our ExpStar substantially outperforms 14 leading LMMs, which highlights the superiority of our dataset and model. We believe that ExpStar holds great potential for advancing AI-assisted scientific experiment instruction.
Current medical multimodal large language models (MLLMs) have demonstrated high accuracy and effectiveness on specific medical visual question answering (Medical VQA) tasks. However, they largely fail to tackle continuously emerging unseen Medical VQA scenarios (e.g., MRI, X-ray) in real-world settings, which significantly hinders their broader adoption in practical clinical environments. Motivated by these gaps, this paper introduces a new task, namely LLM-centric Lifelong Learning for New Medical VQA (L3NMV), which enables Large Language Models (LLMs) to continually learn medical image-text knowledge across various medical VQA tasks. Furthermore, this paper reveals two critical challenges: 1) Efficient medical knowledge retention (Each-task), which aims to retain essential knowledge for each Medical VQA task efficiently with limited data. 2) Efficient medical interference mitigation (Cross-task), which focuses on efficiently mitigating information interference across various Medical VQA tasks with knowledge barriers. To address these challenges, this paper proposes the OmniDoctor model, i.e., an omniscient doctor that simulates how doctors continuously update their knowledge and skills through continuous medical education, with the goal of equipping the model with lifelong learning capabilities via an efficient incremental medical parameter constraining mechanism for L3 NMV. This model is designed with two key modules to address the above two challenges, respectively. Especially, this paper constructs an Unseen L3NMV dataset to simulate real-world incremental clinical scenarios. Extensive experiments on this dataset demonstrate that OmniDoctor outperforms several advanced lifelong learning baselines. These results justify the significance of the L3 NMV task and the effectiveness of OmniDoctor in continually adapting to new Medical VQA tasks.
In the digital age, brand meaning is increasingly shaped through user participation and content sharing on social media platforms. However, significant perceptual gaps often exist between official brand narratives and consumer interpretations. These multimodal and cognitively nuanced gaps are challenging to detect and model using traditional analytical methods. To address this, we propose a multi-agent framework that metaphorically models perception as an optical process-propagation, interference, and measurement---termed OPIM. We construct a novel dual-perspective dataset from representative social media platforms, integrating text and image content from both user-generated and official brand communications. We evaluate brand perception along six psychological dimensions. Experiments across 15 brands demonstrate that our framework effectively captures key perception gaps, particularly in sincerity, professionalism, and attractiveness. In contrast, materialism and sophistication exhibit higher alignment between brand messaging and consumer perception. Our framework enhances the cognitive alignment and multimodal interpretability of large language models, offering actionable insights for brand strategy and bridging computational modeling with human-centric understanding. The dataset will be available at https://github.com/htgan-ai/OPIM.
The exponential growth of video content necessitates efficient summarization techniques that balance local redundancy reduction and global dependency modeling. In this work, we introduce VSumMamba, an innovative video summarization approach that leverages Selective State Space Models to address the quadratic complexity limitations of Transformer based approaches meanwhile surpassing CNNs' restricted long-range modeling capabilities. The proposed framework comprises three core components: 1) a Multi-Scale Aggregator, 2) a Cascaded Temporal Modeling Module with bi-directional Mamba blocks for temporal representation enhancement, and 3) a Parallel Spatial Modeling Module employing spatial Mamba blocks, operating in concert to effectively refine spatiotemporal video representations. Through three specialized multi-scale spatial-temporal modeling schemes, VSumMamba demonstrate the ability to balance computational efficiency and summarization performance. Comprehensive evaluations on benchmarks datasets demonstrate VSumMamba's superior performance, achieving 67.5% and 56.0% F1-scores on TVSum and SumMe respectively, while maintaining lower computational cost compared to existing state-of-the-art methods.
Anomaly detection in graph-structured data is an inherently challenging problem, as it requires the identification of rare nodes that deviate from the majority in both their structural and behavioral characteristics. Existing methods, such as those based on graph convolutional networks (GCNs), often suffer from over-smoothing, which causes the learned node representations to become indistinguishable. Furthermore, graph reconstruction-based approaches are vulnerable to anomalous node interference during the reconstruction process, leading to inaccurate anomaly detection. In this work, we propose a novel and holistic anomaly evaluation framework that integrates three key components: a local-global Transformer encoder, a memory-guided reconstruction mechanism and a multi-scale representation matching strategy. These components work synergistically to enhance the model's ability to capture both local and global structural dependencies, suppress the influence of anomalous nodes, and assess anomalies from multiple levels of granularity. Anomaly scores are computed by combining reconstruction errors and memory matching signals, resulting in a more robust evaluation. Extensive experiments on seven benchmark datasets demonstrate that our method outperforms existing state-of-the-art approaches, offering a comprehensive and generalizable solution for anomaly detection across various graph domains.
Few-shot class-incremental learning (FSCIL) grapples with the dual challenge of learning new classes from minimal labeled training data while alleviating catastrophic forgetting of previous learned classes. Compared with previous methods employing static adaptation on specific parameters, current works verify that dynamic weights and sequence modeling in Selective State Space Models (SSMs) can capture distinctive feature drifts in FSCIL. However, the flattening operation in SSMs fragments the latent semantic relationship, where the resulting task isolation and representation degeneration are detrimental to FSCIL. Toward this issue, this paper presents a novel framework named Probabilistic Mixture of Hyperbolic State Space Experts (PmH-SSE) for FSCIL. First, since SSMs rely on scanning as an alternative to self-attention, the Hyperbolic state space model with multi-scale hybrid scan is built to facilitate few-shot learning by providing an extra Hyperbolic geometry that encodes hierarchical relationships. Moreover, we propose the probabilistic mixture of Mamba to increase the model's flexibility in handling non-stationary data streams in FSCIL and enhance the stability of high-parameter models in few-shot conditions. Finally, under the same experimental conditions, the proposed PmH-SSE demonstrates superior performance in comprehensive experiments. The codes are available at https://github.com/yawencui/PmH-SSE.
Due to data collection limitations and annotation reliability, the lack of multi-view data will weaken the comprehensive understanding of samples, and incomplete multi-view multi-label classification faces severe challenges. To address this problem, we propose a multi-view complementary learning framework MC-IVLC to explore the complementary information between views fully. Specifically, MC-IVLC proposes compensating for the collapse of reconstructed features and explicitly using fused features as supervisory signals to guide the completion of missing views. In addition, MC-IVLC fully utilizes the complementary relationship between views from both instance and semantic levels. Instance-level contrastive learning aims to promote the clustering of similar features in the same view to enhance the complementarity of cross-view features. Semantic-level contrastive learning utilizes pseudo-labels to infer missing labels in label embeddings. It combines pseudo-label semantic information with feature embeddings to guide the semantic relevance of cross-view features. Finally, MC-IVLC explicitly encodes view identity and introduces a view-label prediction loss term to enhance the perception of view information and align single views and multiple views, further exploring the intrinsic connection between views and labels. We conduct experiments on five widely used datasets. Experimental results show that MC-IVLC achieves excellent performance compared with state-of-the-art methods. Ablation studies further validate the effectiveness of each component.
Video anomaly detection (VAD) is vital for public safety, yet current approaches struggle with limited generalization, low interpretability, and high resource demands. To address these challenges, we propose HoloTrace, an edge-cloud collaborative VAD system that integrates large language models (LLMs) to construct and update a novel bidirectional causal knowledge graph. At the edge, HoloTrace leverages LLM-based cross-modal understanding and employs Hidden Markov Model (HMM) for bidirectional event reasoning, obtaining anomaly boundaries with low computational overhead. On the cloud side, LLMs are leveraged to dynamically update the Bi-CKG graph with key frames sent from the edge, in order to update causal relationships between events. Additionally, we introduce SVAD, a new large-scale VAD dataset comprising 632 real-world surveillance videos across 10 anomaly types and diverse scenes, with manually labeled frame-level annotations. Experimental results demonstrate that HoloTrace not only achieves the highest accuracy but also enhances interpretability and efficiency, paving the way for more generalizable and explainable video anomaly detection systems.
The significance of informative and robust point representations has been widely acknowledged for 3D scene understanding. Despite existing self-supervised pre-training counterparts demonstrating promising performance, the model collapse and structural information deficiency remain prevalent due to insufficient point discrimination difficulty, yielding unreliable expressions and suboptimal performance. In this paper, we present GaussianCross, a novel cross-modal self-supervised 3D representation learning architecture integrating feed-forward 3D Gaussian Splatting (3DGS) techniques to address current challenges. GaussianCross seamlessly converts scale-inconsistent 3D point clouds into a unified cuboid-normalized Gaussian representation without missing details, enabling stable and generalizable pre-training. Subsequently, a tri-attribute adaptive distillation splatting module is incorporated to construct a 3D feature field, facilitating synergetic feature capturing of appearance, geometry, and semantic cues to maintain cross-modal consistency. To validate GaussianCross, we perform extensive evaluations on various benchmarks, including ScanNet, ScanNet200, and S3DIS. In particular, GaussianCross shows a prominent parameter and data efficiency, achieving superior performance through linear probing (<0.1% parameters) and limited data training (1% of scenes) compared to state-of-the-art methods. Furthermore, GaussianCross demonstrates strong generalization capabilities, improving the full fine-tuning accuracy by 9.3% mIoU and 6.1% AP50 on ScanNet200 semantic and instance segmentation tasks, respectively, supporting the effectiveness of our approach. The code, weights, and visualizations are publicly available at https://rayyoh.github.io/GaussianCross/.
Electrocardiogram (ECG), a key bioelectrical time-series signal, is crucial for assessing cardiac health and diagnosing various diseases. Given its time-series format, ECG data is often incorporated into pre-training datasets for large-scale time-series model training. However, existing studies often overlook its unique characteristics and specialized downstream applications, which differ significantly from other time-series data, leading to an incomplete understanding of its properties. In this paper, we present an in-depth investigation of ECG signals and establish a comprehensive benchmark, which includes (1) categorizing its downstream applications into four distinct evaluation tasks, (2) identifying limitations in traditional evaluation metrics for ECG analysis, and introducing a novel metric; (3) benchmarking state-of-the-art time-series models and proposing a new architecture. Extensive experiments demonstrate that our proposed benchmark is comprehensive and robust. The results validate the effectiveness of the proposed metric and model architecture, which establish a solid foundation for advancing research in ECG signal analysis.
Three-dimensional (3D) mapping is vital in modern remote sensing. Satellites provide map data but are limited by cloud cover, especially during natural disasters (e.g., earthquakes, tsunamis), where rapid response is crucial and damaged infrastructure often renders digital maps unusable. Although Unmanned Aerial Vehicles (UAVs) present a viable alternative, the generation of precise 3D maps using monocular camera systems remains technically challenging. This work introduces an innovative approach for fast 3D mapping with intelligent trajectory planning. The method employs 2D Gaussian Splatting (2DGS) with block-based parallel optimization, integrating a monocular depth prior, depth filter, and a novel dense gradient strategy to reconstruct 3D maps from 2D images. To address operational reliability, we implement a multi-agent planning system which integrates artificial intelligence generated content (AIGC) models as agents. Each UAV's trajectory is managed by the agents to optimize paths dynamically. Experiments demonstrate the method's superiority in speed and effectiveness, offering a robust solution for disaster response and reconstruction.
The goal of generic multimodal summarization is to extract the most important information from different modalities to form summaries. Yet the importance of scenes and text in a video is often subjective, and users should have the option of customizing the summary by using natural language to specify what is important to them. However, existing methods for fully automatic multimodal summarization have not exploited available language models, which can serve as an effective prior for saliency. To address this issue, we introduce Query-Focused Multimodal Summ arization(QFSumm), a single framework for addressing both generic and query-focused multimodal summarization, typically approached separately in the literature. In addition, we propose a novel gate-guided mixture-of-experts that uses expert gate module to organize three experts (video expert, text expert and shared expert) to model the correlations between multimodal information. In addition, we propose two novel contrastive losses to represent consistency and diversity. Extensive experiments on a query-focused video summarization dataset (QFVS), two standard video summarization datasets (TVSum and SumMe) and three multimodal summarization datasets (CNN, Daily Mail and BLiSS) demonstrate the superiority of QFSumm, achieving state-of-the-art performances on all datasets.
Multi-view clustering (MVC) has gained extensive attention for its capacity to handle heterogeneous data. However, current autoencoder-based MVC methods suffer from a limitation: embedding space exhibits severe imbalances in the efficacy of feature direction, creating a long-tailed singular value distribution where few directions dominate. To mitigate this, we introduce a novel Activate-Then-Eliminate Strategy for Multi-View Clustering (AEMVC), inspired by the observation that balanced feature directions can facilitate enhancing discrimination of learned representations. AEMVC dynamically adjusts the contributions of different feature directions through two keys: a Feature Activation Module that narrows singular value discrepancies to prevent dominant directions from controlling clustering decisions, and an Inter-view Mutual Supervision strategy that filters redundant information by adaptively determining view-specific thresholds based on cross-view consistency. By activating more feature directions and eliminating each view's adverse factors, AEMVC achieves more balanced and discriminative embedding representations. Extensive experiments on seven multi-view benchmarks validate AEMVC's effectiveness, demonstrating substantial improvements over state-of-the-art methods.
Cross-domain recommendation (CDR) aims to address the persistent cold-start problem in Recommender Systems. Current CDR research concentrates on transferring cold-start users' information from the auxiliary domain to the target domain. However, these systems face two main issues: the underutilization of multimodal data, which hinders effective cross-domain alignment, and the neglect of side users who interact solely within the target domain, leading to inadequate learning of the target domain's vector space distribution. To address these issues, we propose a model leveraging Multimodal data and Side users for diffusion Cross-domain recommendation (MuSiC). We first employ a multimodal large language model to extract item multimodal features and leverage a large language model to uncover user features. Secondly, we propose the cross-domain diffusion module to learn the generation of feature vectors in the target domain. This approach involves learning feature distribution from side users and understanding the patterns in cross-domain transformation through overlapping users. Subsequently, the trained diffusion module is used to generate feature vectors for cold-start users in the target domain, enabling the completion of cross-domain recommendation tasks. Finally, our experimental evaluation of the Amazon dataset confirms that MuSiC achieves state-of-the-art performance, significantly outperforming all selected baselines. Our code is available: https://github.com/zhangf16/MuSiC.
Personalized news recommendation aims to deliver content aligned with user interests. However, most existing methods rely on the objective textual content of news, overlooking the subjective social review that reflects how the news is socially perceived. Inspired by social constructionism, we propose Social Review-aware Recommendation (SRec), a novel framework that integrates both objective content and the social review. The latter is constructed through group deliberation modeled by an agent-based social simulator, providing structured representations of collective understandings toward news. In addition, SRec incorporates a reasoning-guided explanation module that produces interpretable rationales by aligning user preferences with the social review of news. Experimental results on the MIND-small and MIND-large datasets demonstrate that SRec improves AUC by at least 2.45% over competitive baselines. Further analysis confirms the value of the social review generated by the simulator, and shows the flexibility of SRec as a lightweight enhancement to existing recommendation systems.
Sequential Recommendation (SR) focuses on personalizing user experiences by predicting future preferences based on historical interactions. Transformer models, with their attention mechanisms, have become the dominant architecture in SR tasks due to their ability to capture dependencies in user behavior sequences. However, traditional attention mechanisms, where attention weights are computed through query-key transformations, are inherently linear and deterministic. This fixed approach limits their ability to account for the dynamic and non-linear nature of user preferences, leading to challenges in capturing evolving interests and subtle behavioral patterns. Given that generative models excel at capturing non-linearity and probabilistic variability, we argue that generating attention distributions offers a more flexible and expressive alternative compared to traditional attention mechanisms. To support this claim, we present a theoretical proof demonstrating that generative attention mechanisms offer greater expressiveness and stochasticity than traditional deterministic approaches. Building upon this theoretical foundation, we introduce two generative attention models for SR, each grounded in the principles of Variational Autoencoders (VAE) and Diffusion Models (DMs), respectively. These models are designed specifically to generate adaptive attention distributions that better align with variable user preferences. Extensive experiments on real-world datasets show our models significantly outperform state-of-the-art in both accuracy and diversity.
To mitigate data sparsity in Sequential Recommendation, Cross-Domain Sequential Recommendation (CDSR) exploits dynamic knowledge transfer across domains. Traditional CDSR approaches merge specific-domain sequences into mixed-domain sequences to reconnect users' dispersed interests. However, most methods rely on unidirectional transfer between mixed and specific domains on each domain task, overlooking the complex interplay between mixed-domain and domain-specific dynamics. Moreover, token-level transfer between coinciding domain sequences fails to consider inherent sequential dynamics. To address these limitations, we propose Multi-Domain Enhancement via Residual Interwoven Transfer (MERIT). Specifically, MERIT enhances domain representations along multiple domain-to-domain paths, leveraging the proposed extended cross-attention fusion compatible with partially overlapping sequences. To facilitate such transfers, MERIT further employs MoE networks in encoders to generate both intra-domain and inter-domain representations. In addition, by integrating stopped-gradient mixed-domain representations into specific-domain representations, MERIT enables the model to learn the residual signal of the mixed-domain information, better aligning with downstream specific-domain tasks. Extensive experiments on three real-world datasets demonstrate that MERIT consistently outperforms state-of-the-art CDSR counterparts with statistical significance.
Learning user preferences in recommendation systems is enriched by multimodal features, such as textual and visual content, and amplified by multi-interest modeling with Variational AutoEncoders (VAEs). However, prior efforts are limited by single modality focus and cumbersome, parameter-heavy architecture designs. To address these limitations, we introduce an innovative solution that blends the semantic richness of multimodal data with the representational power of multi-representation VAEs. Drawing inspiration from Mixture of Experts (MoE), we cast each VAE as an expert tailored to a specific modality, then fuse them via a novel parameter-merging function into a lean, unified model. This approach efficiently captures diverse user preferences behind multimodal data with minimal complexity. Rigorous experiments on real-world benchmarks show our method outshines state-of-the-art baselines while slashing parameter counts. Our work sets a new, streamlined standard for multimodal, multi-interest recommendation systems.
With the rapid development of AI-generated content (AIGC), the creation of high-quality AI-generated videos has become faster and easier, resulting in the Internet being flooded with all kinds of video content. However, the impact of these videos on the content ecosystem remains largely unexplored. Video information retrieval remains a fundamental approach for accessing video content. Building on the observation that retrieval models often favor AI-generated content in ad-hoc and image retrieval tasks, we investigate whether similar biases emerge in the context of challenging video retrieval, where temporal and visual factors may further influence model behavior. To explore this, we first construct a comprehensive benchmark dataset containing both real and AI-generated videos, along with a set of fair and rigorous metrics to assess bias. This benchmark consists of 13,000 videos generated by two state-of-the-art open-source video generation models. We meticulously design a suite of rigorous metrics to accurately measure this preference, accounting for potential biases arising from the limited frame rate and suboptimal quality of AIGC videos. We then applied three off-the-shelf video retrieval models to perform retrieval tasks on this hybrid dataset. Our findings reveal a clear preference for AI-generated videos in retrieval. Further investigation shows that incorporating AI-generated videos into the training set of retrieval models exacerbates this bias. Unlike the preference observed in image modalities, we find that video retrieval bias arises from both unseen visual and temporal information, making the root causes of video bias a complex interplay of these two factors. To mitigate this bias, we fine-tune the retrieval models using a contrastive learning approach. The results of this study highlight the potential implications of AI-generated videos on retrieval systems and offer valuable insights for future research in this area. Our dataset and code are publicly available at https://github.com/Siaaaaaa1/video-source-bias.