Micro-expressions (MEs) are involuntary and quickly displayed facial expressions that reveal subtle psychological activities. Most previous research typically focused on two separate tasks: micro-expression spotting and recognition. We aim to propose a high-precision "spotting+recognition" method that can spot ME intervals from long videos and recognize their emotional categories. Due to the occurrence sparsity of MEs, there is a significant imbalance between the number of micro-expression intervals and non-micro-expression intervals in long videos. This imbalance makes it challenging for models trained using conventional strategies to distinguish true MEs from noise samples caused by head movements, blinking, and macro-expressions, resulting in a high false-positive-rate and reducing the overall performance. We reduce the number of smooth segments to alter the data distribution within the non-micro-expression (non-ME) category. This adjustment enables the model to focus more on the subtle differences between noise samples and ME samples. To achieve this, we design an ingenious training data preparation strategy: using false positive samples from the initial spotting results as non-ME category samples, and using true positive and false negative samples from the initial spotting as emotion category samples. These are combined as the training data, creating a recognition model capable of both emotion classification and non-ME category determination. Additionally, we propose a three-stage micro-expression analysis method, including ME spotting, ME recognition and non-ME intervals removal module. Our method is validated through five-fold cross-validation experiments on the CAS(ME)² and SAMM Long Video datasets, achieving a overall STRS metric of 0.16, which significantly outperformed baseline methods and demonstrated the effectiveness of our approach.
论文检索
输入标题、作者或关键词，从 1,237 篇学术成果中精准定位
Recently, automatic micro-expression (ME) analysis has attracted increasing attention, since ME is a spontaneous facial expression that can truly reflect the emotional state an individual tries to conceal. As a crucial step in ME analysis, Micro- and Macro-expression (MaE) spotting aims to sequentially identify the occurrence intervals of MEs and MaEs within a long video sequence. However, the subtle spatiotemporal movements of MEs and the scarcity of well-labeled data pose great challenges for accurately spotting them. To this end, this paper proposes a novel spotting framework based on Multi-scale Feature Learning Network with Optical Flow Correction. Specifically, we first integrate the pre-trained VideoMAE and customized convolutional layers as a visual feature extraction module to learn the facial motion features in long video sequences. Then, to comprehensively locate and identify the existing ME and MaE segments, we introduce a multi-scale candidate segment generation method based on the ActionFormer. In particular, a multi-start points optical flow filtering method is proposed to improve the precision of expression spotting. Finally, we conduct comprehensive experiments on the MEGC2024 spotting task, and the experimental results demonstrate the effectiveness of our method, which ranks second in this task. The implemented code is also publicly available at https://github.com/zzy188zzy/megc_spotting_code.
Micro-expressions, as a type of facial expression corresponding to macro-expressions, usually have a short duration and low intensity. Due to these characteristics, micro-expression spotting holds significant value in medical care and public safety. Recent years have witnessed advancements in micro-expression spotting methods; however, spotting micro-expressions remains a challenging task due to their brief duration and low intensity. In this paper, we propose a micro-expression spotting method based on optical flow features with boundary calibration. We first perform face detection, cropping, and alignment on images containing faces. Then, regions of interest (ROIs) are defined, and optical flow features are extracted. Furthermore, candidate expression segments are identified based on the magnitude of the processed optical flows. Finally, a boundary calibration module is utilized to calibrate the boundaries. The effectiveness of the proposed method is evaluated on the MEGC2024 test set, resulting in an overall F1-score of 0.27.
Micro-Expression is subtle facial movements that reveal hidden emotions, but their fleeting and involuntary nature poses significant challenges for detection. This paper introduces a novel approach addressing two critical tasks in Micro-Expression analysis: spotting and Recognize. We integrate the VideoMAE V2 framework with a temporal information adapter and multi-scale feature fusion to enhance the performance of Micro-Expression Spotting-then-Recognize. Our method leverages the temporal information adapter to capture local temporal context within video frames, improving feature extraction efficiency. Additionally, we construct a multi-scale image pyramid to capture a range of motion features, from broad movements to subtle details. By combining these multi-scale features, our approach strengthens the model's capabilities in Micro-Expression Spotting-then-Recognize. Our method effectively addresses issues related to environmental variations, involuntary facial movements, and dataset imbalance, leading to improved accuracy in Micro-Expression Spotting-then-Recognize.
Facial micro-expressions (MEs) are involuntary spontaneous movements of the face that typically appear in high-stakes situations where a person attempts to conceal a certain emotion from being known. A decade after the inception of the widely used CASME II and SMIC datasets, research in computational analysis of MEs has now advanced toward new pathways, exploring problems crucial to model generalization and real-world practicality. It is often challenging to design robust algorithms or models for spotting micro-expressions due to the high variability across diverse cultural backgrounds. Also, treating spotting and recognition as separate tasks is undesirable when handling long-spanning videos under realistic settings. This Grand Challenge comprises two distinct tracks: the Cross-Cultural Spotting (CCS) track, and the Spot-Then-Recognize (STR) track. All participating solutions submitted their results to a leaderboard, and several submissions performed well surpassing their respective baseline results. More details are available at: https://megc2024.github.io.
This paper presents an optimized approach for the AI-generated image detection task in the 2024 ACM Multimedia Grand Challenge. Given the rapidly evolving capabilities of generative models, traditional detection methods often struggle with accuracy and generalization. The proposed solution builds upon the baseline by integrating advanced model architecture enhancements and novel detection modules, specifically designed to address the complexities of AI-generated images. Through rigorous testing, the approach demonstrates significant improvements in precision, particularly in identifying images produced by unknown generative models. This work highlights the critical need for adaptable and robust detection methods to keep pace with the advancements in AI-generated content and sets a new standard for future research and development in this domain.
In this rapidly evolving era of AI technology, image generation models are pushing the boundaries of art, design, and even information dissemination at an unprecedented pace. However, alongside this technological advancement comes the growing challenge of combating misinformation and image manipulation. One major difficulty in determining AI-generated images lies in dealing with various generation models. In this report, our core solution can be summarized as ''balancing data sources is key to enhancing the generalization capability of the detection model.'' In the competition, we achieved a score of 0.9558 on the B leaderboard, ultimately securing the second place.
With the development of AIGC, images generated by AI are difficult to identify with the human eye. This article proposes a simple training architecture and a simple model. In the training process of deep forgery detection, a variety of data enhancement techniques are used, including horizontal flipping, adding Gaussian noise and random cropping to enhance the generalization ability of the model. Differentiated processing strategies are adopted for images of different sizes: center cropping is used for images smaller than 224 pixels, random cropping is used for images between 224 and 512 pixels, and images larger than 512 pixels are first scaled and then randomly cropped. Furthermore, the concept of neighbor pixel relations (NPR) is introduced, which is an effective way to capture local structural artifacts introduced by upsampling operations. As a complement to image representation, NPR significantly improves the detector's ability to identify forged images, especially when the forgery technology is unknown. By integrating the NPR module into the ResNet-50 model, it not only enhances the classifier's extraction of forged features, but also maintains the computational efficiency and versatility of the model. Finally, AIGC generated images in the ''Malanshan Cup" International Audio-Video Algorithm Competition. Determine the first place result for the track.
ACM Multimedia 2024 Grand Challenge Report for Artificial Intelligence Generated Image Detection
PDF ↗The AI Generated Image Detection Challenge, organized by MGTV, invites participants to develop advanced algorithms capable of accurately distinguishing between real and AI-generated images. These images may be created using various cutting-edge techniques, including but not limited to GAN and Stable Diffusion algorithms. Participants are encouraged to utilize open-source datasets or develop their own datasets to train their algorithms. This challenge presents a unique opportunity to enhance the field of AI-generated image detection, particularly in improving the algorithm's generalization capabilities to identify unknown and emerging samples. For more details and resources, please visit our official website (https://challenge.ai.mgtv.com/#/track/24).
Revisiting Vision-Language Features Adaptation and Inconsistency for Social Media Popularity Prediction
PDF ↗Social media popularity (SMP) prediction is a complex task involving multi-modal data integration. While pre-trained vision-language models (VLMs) like CLIP have been widely adopted for this task, their effectiveness in capturing the unique characteristics of social media content remains unexplored. This paper critically examines the applicability of CLIP-based features in SMP prediction, focusing on the overlooked phenomenon of semantic inconsistency between images and text in social media posts. Through extensive analysis, we demonstrate that this inconsistency increases with post popularity, challenging the conventional use of VLM features. We provide a comprehensive investigation of semantic inconsistency across different popularity intervals and analyze the impact of VLM feature adaptation on SMP tasks. Our experiments reveal that incorporating inconsistency measures and adapted text features significantly improves model performance, achieving an SRC of 0.729 and an MAE of 1.227. These findings not only enhance SMP prediction accuracy but also provide crucial insights for developing more targeted approaches in social media analysis.
The prediction task of social media popularity aims to automatically forecast the future popularity of the posts by leveraging vast amounts of social media data. This data encompasses diverse visual and textual content, including photos, categories, custom tags, temporal information, and geographical data. Existing methods have explored multiple feature types to enhance popularity prediction. Despite their success, visual and textual features-both crucial pieces of information-are often simply concatenated after extraction, ignoring the divergence between these two feature spaces. In this paper, we propose a method to project visual and language information into an aligned semantic representation, thereby uncovering intricate associations between these two modalities. Specifically, we leverage the BLIP-2 model to understand and generate visual description text that encapsulates the content of photos. Semantic embeddings are then extracted from all available visual and textual information. Additionally, we deeply exploit user-related behavior and characteristic information to extract features, uncovering hidden clues for post popularity prediction. Leveraging these improvements, we conduct extensive experiments to demonstrate the effectiveness of our proposed method.
Social media has emerged as a vital platform for communication, information sharing, and acquisition. Predictive analysis of social media data has wide applications, such as sentiment examination and social network analysis. However, existing work often directly utilizes social media data for training, neglecting the issue of mismatched text and images. This neglect can lead to confusion about the contents, thereby affecting the identification of trending topics and the accuracy of social media predictions. In this paper, an approach named Dual-Stream Pre-training Transformer (DSPT) is introduced to address this gap. In DSPT, we use a Visual-Language Model (VLM) and a Language Model (LM) to separately learn from image and text data, mitigating the impact of text-image mismatches. Moreover, to enhance the understanding of the model to social media data, we conduct incremental pre-training for both models. To achieve better feature interaction, we construct an integrated regression module combining LightGBM and CatBoost, jointly predicting the extracted feature embeddings. This dual-stream multimodal feature extraction method improves the performance of predictive tasks. Experimental results validate the effectiveness of our approach, demonstrating its potential and providing deeper insights into multimodal data mining in social media.
The Social Media Prediction (SMP) challenge focuses on predicting the popularity of an online post in social media. Social media contains multimodal information, such as texts, images, user IDs, timestamps, locations, etc. Many studies have presented various feature extraction methods to retrieve the multimodal features and used the retrieved features to predict the popularity of posts. However, they often overlook the phenomenon that the posts made by the same user tend to have similar popularity. In this paper, we propose the MultiModal Framework, called MMF, our winning solution to the Social Media Prediction (SMP) Challenge 2024. MMF derives the post representations by considering the interaction relationships and inter-relationships among the extracted multimodal features. Also, it introduces the pseudo labels to consider the phenomenon that the posts made by the same user tend to have similar popularity and learn the popularity distributions of users in the test dataset. Therefore, MMF can simultaneously learn the relationships between post representations and true labels from the data of the training dataset, and users' popularity distributions from the data of the test dataset. The extensive experiments on the Social Media Prediction Dataset show that our proposed framework outperforms the compared models in terms of Spearman's rank correlation and mean absolute error.
SMP Challenge is an annual challenge that seeks top research teams to develop innovative forecasting methods that can enhance social and business applications. We define and introduce the Social Media Popularity Prediction (SMPP) task that predicting the future popularity of a post made by a specific user at a given time on social media. This task is pivotal in various applications and scenarios, such as online advertising, social recommendations, post ranking, and demand forecasting, etc. To motivate diverse perspectives of social media prediction researches, we built a large-scale benchmark Social Media Prediction Dataset (SMPD) that includes approximately 500K posts, along with associated 756 tags, visual-language data, and spatial-temporal information, and sourced from around 70K users and their profiles. With participation and contribution from top teams worldwide, the challenge has seen continuous performance improvements in recent years, driven by technological advancements. For the latest information, leaderboard or online evaluation, please visit the SMP Challenge Homepage: www.smp-challenge.com.
Multimodal Large Language Models (MLLMs), by expanding the model's capabilities to perceive and interact through multi-modalities, have significantly enhanced performance across various tasks. The perception of vision is an important modality developed into LLM, enabling research in vision-language to continuously lead the cutting-edge advancements in the MLLM community. However, the standard pre-training pipeline on image-text pairs results in a limited model understanding of relationships between multiple images and texts, as well as visual details. Additionally, the setting of fine-tuning with a frozen visual backbone hinders the enhancement of visual representations on new data. These two issues lead to suboptimal performance in models for demonstrative instruction following about multiple images. This work introduces a novel framework called MLoEM, which first converts long multimodal data into an interleaved image-instruction format, and then adopts a fully autoregressive architecture model, allowing for more robust and coherent learning from naturally occurring multimodal documents than pair-based pipeline. Additionally, we incorporate the Low-Rank Adaptation (LoRA) fine-tuning method, enhancing visual representations while maintaining the stability of previously learned knowledge. Finally, we utilize ensemble methods to enhance model performance on tasks. To alleviate the storage overhead issue of parallel ensembles with large models, we design an ensemble approach that shares the MLLM while only switching the LoRA matrices. In the experiments, the proposed MLoEM shows superior performance on the testing set.
Recent advancements in Multimodal Large Language Models (MLLMs) have showcased their ability to handle tasks involving single images, such as generating detailed descriptions and answering related queries. These models have significantly pushed the boundaries of multimodal content comprehension and generation. However, when confronted with more complex multimodal contexts, particularly those involving interleaved text and multiple images, MLLMs often face difficulties in effectively processing and understanding these complex contexts. Furthermore, MLLMs exhibit weaknesses in instruction following, especially when dealing with tasks that require reasoning about and prioritizing specific details. In this paper, we design a novel approach, InstructFusion, which introduces an enhanced module Fusing Former to improve the model's understanding of relationships among multiple images within convoluted multimodal tasks. This ensures more accurate and contextually relevant responses. Additionally, we adopt a two-stage training process, first developing general capabilities in the Fusing Former and then fine-tuning with LoRA on instruction-focused datasets to enhance instruction following ability while minimizing costs and preventing forgetting. Empirical results, including a second-place finish in the ACM Multimedia 2024 Demonstrative Instruction Following Challenge, demonstrate the effectiveness of our proposed method.
We introduce the DEMON Challenge, defined as a benchmark for demonstrative instruction following, to ACM Multimedia 2024. The DEMON Challenge aims to assess the ability of models and systems to comprehend demonstrative instructions consisting of multiple, interleaved, and multimodal context that demonstrate the required information to complete a task. These instructions are curated from a diverse range of multi-modal datasets, spanning various fields and scenarios, to ensure comprehensive coverage and challenge diversity. The challenge details and participation information are available on the https://dcdmllm.github.io/DEMON-challenge/.
Visual Spatial Description (VSD) aims to generate texts that describe the spatial relationships between objects within images. Traditional visual spatial relationship classification (VSRC) methods typically output the spatial relationship between two objects in an image, often neglecting world knowledge and lacking general language capabilities. In this paper, we propose a Large Language-and-Vision Assistant for Visual Spatial Description, named LLaVA-VSD, which is designed for the classification, description, and open-ended description of visual spatial relationships. Specifically, the model first constructs a visual spatial instruction-following dataset using given figure-caption pairs for the three tasks. It then employs LoRA to fine-tune a Large Language and Vision Assistant for VSD, which has 13 billion parameters and supports high-resolution images. Finally, a large language model is used to refine the generated sentences, enhancing their diversity and accuracy. LLaVA-VSD demonstrates excellent multimodal conversational capabilities and can follow open-ended instructions to assist with inquiries about object relationships in images.
In recent years, the task of image-to-text generation has received considerable attention from scholars. One of its subtasks, Visual Spatial Description (VSD), focuses on a model's ability to understand spatial relationships. VSD is a novel task that emphasizes spatial semantics by generating sentences describing the spatial relationships between two objects in a given image. In this work, a VSD method based on large language model fine-tuning (LFVSD) is proposed to enhance the accuracy and robustness of visual spatial relationship descriptions. Initially, image and text features are extracted using pre-trained models, and Q-Former is employed for feature fusion. The original and fused features are then fed into FlanT5XXL. Object overlap priors are introduced, and momentum distillation is used to filter hard negative samples and generate soft labels. Finally, multiple VSD models are trained using data augmentation and long-tail data balancing techniques. Through multimodal feature fusion and fine-tuning, our approach is evaluated on the VSD2024 test set, which includes 5,855 images and their corresponding textual descriptions. The results demonstrate the effectiveness of our proposed method.
Visual Spatial Description (VSD) is an emerging image-to-text task which aims at generating descriptions of the spatial relationships between given objects in an image. In this paper, we apply Retrieval-Augmented Generation (RAG) technology in guiding Multimodal Large Language Models (MLLMs) for the task of VSD, complemented by an Adaptive Hallucination Corrector, and further fine-tuning them to bolster semantic understanding and overall model efficacy. We found that our approach demonstrated higher accuracy and fewer hallucination errors in both spatial relationship classification and visual language description tasks within the VSD task, achieving state-of-the-art results.