Deep learning models have emerged as a promising alternative to conventional approaches for plant disease identification, a critical challenge in agricultural production. However, the existing plant disease datasets are insufficient to address the complexities of real-world agricultural scenarios, such as multi crop disease, unseen, few-shot, and domain shift adaptation. Additionally, the lack of standardized evaluation protocols and benchmark datasets hinders the fair evaluation of models against these challenges. To bridge this gap, we introduce Deep-Plant-Disease, the largest and most diverse dataset with novel text data designed to enhance model generalization in multi crop disease identification. We revisit and reformulate the task by establishing a standardized evaluation framework that supports consistent benchmarking and guides future research. Through experiments, we further validate the robustness and adaptability of models trained on our dataset, highlighting their effective transferability to real-world agricultural challenges.
论文检索
输入标题、作者或关键词，从 1,620 篇学术成果中精准定位
The proliferation of online misinformation videos poses serious societal risks. Current datasets and detection methods primarily target binary classification or single-modality localization based on post-processed data, lacking the interpretability needed to counter persuasive misinformation. In this paper, we introduce the task of Grounding Multimodal Misinformation (GroundMM), which verifies multimodal content and localizes misleading segments across modalities. We present the first real-world dataset for this task, GroundLie360, featuring a taxonomy of misinformation types, fine-grained annotations across text, speech, and visuals, and validation with Snopes evidence and annotator reasoning. We also propose a VLM-based, QA-driven baseline, FakeMark, using single and cross-modal cues for effective detection and grounding. Our experiments highlight the challenges of this task and lay a foundation for explainable multimodal misinformation detection. Dataset will be released at https://github.com/yangbingjian/GroundLie360.
In natural language communication, emotions are often conveyed through non-verbal sounds (NVs), such as laughter, crying, cough and so on. However, most existing text-to-speech (TTS) corpora lack annotations for these non-verbal sounds, leading to a scarcity of systems capable of generating them. To address this gap, we introduce SMIIP-NV, a non-verbal speech synthesis corpus annotated with both emotions and non-verbal sounds, including laughter, crying, and cough. To the best of our knowledge, SMIIP-NV is the largest publicly available open-source expressive speech corpus that includes non-verbal speech and rich annotations. It comprises 33 hours of speech data, covering five distinct emotions and three types of non-verbal sounds, with detailed transcriptions and precise timestamps for each occurrence of non-verbal sounds. Additionally, the corpus provides annotations for speech segments that contain laughter or crying. To demonstrate the utility of this dataset, we establish a baseline for non-verbal speech synthesis by employing a lightweight large language model (LLM). The SMIIP-NV dataset and static audio demonstrations are publicly available at https://axunyii.github.io/SMIIP-NV. The interactive real-time demonstrations can be accessed at https://huggingface.co/spaces/xunyi/SMIIP-NV_Finetuned_CosyVoice2.
Long-term multi-animal tracking in densely group-housed agricultural settings is critical for automated behavior monitoring and early anomaly detection in precision livestock farming. However, it poses significant challenges due to persistent occlusions from feeders and water dispensers, high inter-individual appearance similarity, and drastic visual changes across day and night cycles. Existing multi-object tracking datasets rarely capture the combined difficulty of these real-world conditions. To address this, we introduce OinkTrack, a large-scale benchmark for continuous multi-pig tracking in commercial farm environments. The dataset comprises over five hours of annotated video across sixteen sequences, covering day, night, night-to-day, and day-to-night transitions. Each sequence ranges from one minute to one hour, featuring an average of thirty-six pigs per frame. In total, OinkTrack provides 573,700 bounding boxes linked to 574 consistent pig identities. It enables detailed behavior analysis under varying lighting and crowding conditions. We describe the data collection and annotation process, present statistical insights into tracking difficulty, and benchmark 11 state-of-the-art tracking methods. OinkTrack provides a robust foundation for developing long-term tracking models and supports downstream applications such as individual activity profiling and early detection of abnormal behavior in real-world, high-density animal populations. The complete dataset and supplementary materials are publicly accessible at https://leohuang0511.github.io/oinktrack-page.
Moving target selection in multimedia interactive systems faces unprecedented challenges as users increasingly interact across diverse, dynamic contexts-from live streaming in moving vehicles to VR gaming in varying environments. Existing approaches rely on probabilistic models that relate endpoint distribution to target properties (size, speed). However, these methods require substantial training data for each new context and lack transferability across scenarios, limiting their practical deployment in diverse multimedia environments where rich multimodal contextual information is readily available. This paper introduces MAGNeT (Multimodal Adaptive Gaussian Networks), which addresses these problems by combining classical statistical modeling with context-aware multimodal method. MAGNeT dynamically fuses pre-fitted Ternary-Gaussian models from various scenarios based on real-time contextual cues, enabling effective adaptation with minimal training data while preserving model interpretability. We take experiments on self-constructed 2D and 3D moving target selection datasets under in-vehicle vibration conditions. Extensive experiments demonstrate that MAGNeT achieves lower error rates with few-shot samples, by applying context-aware fusion of Gaussian experts from multi-factor conditions.
Image Quality Assessment (IQA) models are increasingly relied upon to evaluate image quality in real-world systems --- from compression and enhancement to generation and streaming. Yet their adoption brings a fundamental risk: these models are inherently unstable. Adversarial manipulations can easily fool them, inflating scores and undermining trust. Traditionally, such vulnerabilities are addressed through data-driven defenses --- adversarial retraining, regularization, or input purification. But what if this is the wrong lens? What if robustness in perceptual models is not something to learn but something to design? In this work, we propose a provocative idea: robustness as an architectural prior. Rather than training models to resist perturbations, we reshape their internal structure to suppress sensitivity from the ground up. We achieve this by enforcing orthogonal information flow, constraining the network to norm-preserving operations --- and further stabilizing the system through pruning and fine-tuning. The result is a robust IQA architecture that withstands adversarial attacks without requiring adversarial training or significant changes to the original model. This approach suggests a shift in perspective: from optimizing robustness through data to engineering it through design.
Serving large language models (LLMs) efficiently remains challenging due to the high memory and latency overhead of key-value (KV) cache access during autoregressive decoding. We present TinyServe, a lightweight and extensible serving system for deploying tiny LLMs (e.g., TinyLLaMA, GPT2-345M) with support for structured KV sparsity, plugin-based token selection, and hardware-efficient attention kernels. Unlike prior simulation frameworks, TinyServe executes real-time decoding with configurable sparsity strategies and fine-grained instrumentation. To reduce decoding cost, we introduce a query-aware page selection mechanism that leverages bounding-box metadata to estimate attention relevance between the query and KV cache blocks. This enables selective KV loading with minimal overhead and no model modifications. Our fused CUDA kernel integrates page scoring, sparse memory access, and masked attention in a single pass. Experiments show that TinyServe achieves up to 3.4× speedup and over 2× memory savings with negligible accuracy drop. Additional analysis of cache reuse, page hit rate, and multi-GPU scaling confirms its practicality as an efficient system-level design for LLM training and inference research on resource-constrained hardware.
How to enable agents to predict the outcomes of their own motion intentions in three-dimensional space has been a fundamental problem in embodied intelligence. To explore general spatial imagination capability, we present AirScape, the first world model designed for six-degree-of-freedom aerial agents. AirScape predicts future observation sequences based on current visual inputs and motion intentions. Specifically, we construct a dataset for aerial world model training and testing, which consists of 11k video-intention pairs. This dataset includes first-person-view videos capturing diverse drone actions across a wide range of scenarios, with over 1,000 hours spent annotating the corresponding motion intentions. Then we develop a two-phase schedule to train a foundation model-initially devoid of embodied spatial knowledge-into a world model that is controllable by motion intentions and adheres to physical spatio-temporal constraints. Experimental results demonstrate that AirScape significantly outperforms existing foundation models in 3D spatial imagination capabilities, especially with over a 50% improvement in metrics reflecting motion alignment. The project is available at: https://embodiedcity.github.io/AirScape/.
Turing Patterns for Multimedia: Reaction-Diffusion Multi-Modal Fusion for Language-Guided Video Moment Retrieval
PDF ↗Video-language models are pivotal for tasks such as moment retrieval and highlight detection, yet they often struggle to capture the dynamic, non-linear interactions between temporal video sequences and textual semantics. Existing approaches, relying on static cross-attention or prompt-tuning mechanisms, fail to adaptively model the evolving relationships between modalities, leading to suboptimal alignment and limited generalization. Inspired by systems biology, we propose Reaction-Diffusion Multimodal Fusion (RDMF), a novel framework that reimagines video-language alignment as a reaction-diffusion (RD) process, drawing on the principles of pattern formation introduced by Alan Turing. In RDMF, video features diffuse across time to capture temporal context, while text-video interactions are modeled as non-linear reactions that amplify relevant features and suppress noise, forming emergent patterns akin to biological systems. Leveraging the Gray-Scott RD model, we design a computationally efficient fusion module that integrates video and text representations, supported by rigorous mathematical analysis of stability and convergence using Turing instability criteria. Our framework is theoretically grounded, employing advanced mathematical tools to ensure stable pattern formation, and is practically viable, incorporating standard components like pretrained encoders and DETR-style heads for moment retrieval and saliency prediction. RDMF represents a pioneering interdisciplinary approach, bridging systems biology and multimedia research to address the limitations of conventional multimodal fusion. Preliminary experiments demonstrate its potential to outperform existing methods in identifying salient video moments, offering a new paradigm for video-language tasks. Beyond its immediate applications, RDMF opens avenues for exploring biologically inspired architectures in multimedia, with implications for real-time video analysis, interactive media systems, and cross-disciplinary collaboration between multimedia and systems biology communities.
The integration of multimodal information, particularly visual content, into dialogue systems has primarily focused on interpreting user-provided inputs, while comparatively little attention has been given to the proactive use of such content to enhance responses. In this paper, we explore a new research direction that addresses this gap by enabling dialogue systems to autonomously determine when and how to supplement textual responses with relevant images, based on conversational context and user intent. To support this goal, we propose AMI (Automated Multimodal Insertion), a novel framework for dynamic, context-aware multimodal supplementation in dialogue. We also introduce RID (Response with Appropriate Image Dataset), a bilingual (Chinese-English) multimodal multi-turn dialogue dataset designed to train and evaluate systems on this capability. RID features fine-grained annotations on image insertion timing and rationale, along with carefully aligned image-text pairs to ensure semantic coherence. Our experiments demonstrate that models trained with RID not only generate more informative and engaging responses, but also exhibit a stronger ability to leverage visual content when it is truly beneficial. These findings highlight the potential of proactive multimodal supplementation and offer new insights for advancing the development of intelligent, human-like dialogue systems. Code and data are available at: https://github.com/Tanthen/SCIR-AMI.
To enable robots to comprehend high-level human instructions and perform complex tasks, a key challenge lies in achieving comprehensive scene understanding: interpreting and interacting with the 3D environment in a meaningful way. This requires a smart map that fuses accurate geometric structure with rich, human-understandable semantics. To address this, we introduce the 3D Queryable Scene Representation (3D QSR), a novel framework built on multimedia data that unifies three complementary 3D representations: (1) 3D-consistent novel view rendering and segmentation from panoptic reconstruction, (2) precise geometry from 3D point clouds, and (3) structured, scalable organization via 3D scene graphs. Built on an object-centric design, the framework integrates with large vision-language models to enable semantic queryability by linking multimodal object embeddings, and supporting object-level retrieval of geometric, visual, and semantic information. The retrieved data are then loaded into a robotic task planner for downstream execution. We evaluate our approach through simulated robotic task planning scenarios in Unity, guided by abstract language instructions and using the indoor public dataset Replica. Furthermore, we apply it in a digital duplicate of a real wet lab environment to test QSR-supported robotic task planning for emergency response. The results demonstrate the framework's ability to facilitate scene understanding and integrate spatial and semantic reasoning, effectively translating high-level human instructions into precise robotic task planning in complex 3D environments.
AI-based fitness coaching systems are typically monolithic and opaque, limiting adaptability, transparency, and embodied interaction. We propose a protocol-integrated Digital Twin (DT) architecture that reimagines fitness coaching as a distributed, explainable, and emotionally adaptive ecosystem. The framework adopts a CrewAI-inspired multi-agent design, where specialized agents for posture analysis, speech, physiological sensing, and personalized recommendation collaborate through the Agent-to-Agent (A2A) protocol to enable secure and interoperable task delegation. Context is maintained through short- and long-term memory modules, while the Model Context Protocol (MCP) supports flexible tool and model invocation across heterogeneous AI resources. Transparency and efficiency are ensured with NVIDIA AgentIQ and LangSmith, which provide token-level observability, workflow profiling, and trajectory evaluation. Real-time coaching feedback is synthesized into multimodal outputs, text, speech, and embodied avatars, using Audio2Face and Omniverse, creating expressive and emotionally engaging interactions. This paper presents a blueprint for protocol-driven, multimodal DTs in health and well-being. By combining interoperability, observability, and embodied feedback, it advances beyond centralized assistants toward distributed, memory-aware, and emotionally intelligent digital coaches, laying the foundation for next-generation human-AI collaboration in multimedia health applications.
Longitudinal studies of animal vocalizations provide crucial insights into developmental patterns and communicative evolution. To aid such investigations in canines, this paper introduces the Canine Age Transition Vocalization Dataset, a large-scale collection of dog vocalizations featuring meticulously verified metadata (including precise birthdate, breed, and individual dog ID) for 125 dogs across 6 common breeds. Our in-depth longitudinal analysis of this dataset then reveals novel findings on how key vocal parameters, encompassing defined bark types and finer-grained acoustic components (Elemental Dog Bark Units, or EDBUs), change as dogs mature. This work, therefore, offers both a significant new resource and foundational data that enable deeper, more nuanced investigations into the lifelong vocal development of dogs and other animal communication.
Recent advancements in Audio Language Models (ALMs) have led to significant improvements in speech-related tasks. However, their capacity for profound metaphorical reasoning, especially when derived from audio-specific cues, has yet to be thoroughly investigated. To address this gap, we introduce Unspoken, a bilingual (Chinese-English) question answering benchmark designed to assess ALMs' comprehension of non-literal, metaphor-rich audio. Unlike prior text-centric evaluations, Unspoken emphasizes prosody, phonetic ambiguity, emotional inflection, and other nuanced acoustic features critical to metaphor understanding but often lost in transcription. We construct a high-quality dataset of 2,764 manually curated and validated QA pairs, spanning three reasoning dimensions: semantic, acoustic, and contextual, and covering six common types of metaphors. Evaluation across 23 mainstream ALMs reveals a substantial performance gap: the best model achieves only 69.5% accuracy, significantly below the human average of 81.1%. By analyzing the error patterns, we identify five key failure modes that reveal fundamental limitations in current models' reasoning capabilities. Unspoken not only sets a new standard for evaluating metaphorical reasoning in audio but also pioneers a novel research direction that moves beyond transcription-based assessments. Grounding metaphor understanding in authentic human communication scenarios offers deep insight for developing more cognitively capable ALMs. The data and codes are available at https://github.com/Hongru0306/UNSPOKEN.
Cold-start recommendation, which addresses the challenge of recommending items to new users without sufficient historical interaction data, remains difficult in personalized recommender systems. Cross-domain recommendation methods have gained attention in addressing this issue by transferring user preferences from a source domain to alleviate the cold-start problem in a target domain. Traditional embedding-based approaches typically rely on a one-to-one alignment over continuous user embeddings using overlapping user IDs, which often leads to severe overfitting and limited generalization, particularly when overlapping users between domains are sparse. In this paper, we propose a novel hierarchical recommendation framework specifically targeting the cold-start problem by modeling user preferences as hierarchical interests derived from textual embeddings and Residual Quantized Variational Autoencoders (RQ-VAE). Unlike traditional methods that directly align embeddings, our approach builds a mapping function transferring hierarchical user interest structures from the source to the target domain. This hierarchical mapping significantly enhances generalization, providing a more comprehensive and robust representation of user preferences. Additionally, we employ a generative rewriting mechanism utilizing Large Language Models to refine user-generated reviews into concise, semantically enriched summaries that explicitly highlight user interests. Extensive evaluations on Amazon review datasets demonstrate the effectiveness of our hierarchical preference modeling and generative rewriting approach, outperforming existing embedding-based methods consistently, especially in cold-start and sparsely populated scenarios. Our proposed method thus provides a robust, flexible solution for personalized recommendation tasks in cold-start conditions.
Foundation models have transformed multimedia analysis by enabling robust and transferable representations across diverse modalities and tasks. However, their static deployment conflicts with growing societal and regulatory demands-particularly the need to unlearn specific data upon request, as mandated by privacy frameworks such as the GDPR. Traditional unlearning approaches, including retraining, activation editing, or distillation, are often computationally expensive, fragile, and ill-suited for real-time or continuously evolving systems. In this paper, we propose a paradigm shift: rethinking unlearning not as a retroactive intervention but as a built-in capability. We introduce a prompt-based learning framework that unifies knowledge acquisition and removal within a single training phase. Rather than encoding information in model weights, our approach binds class-level semantics to dedicated prompt tokens. This design enables instant unlearning simply by removing the corresponding prompt-without retraining, model modification, or access to original data. Experiments demonstrate that our framework preserves predictive performance on retained classes while effectively erasing forgotten ones. Beyond utility, our method exhibits strong privacy and security guarantees: it is resistant to membership inference attacks, and prompt removal prevents any residual knowledge extraction, even under adversarial conditions. This ensures compliance with data protection principles and safeguards against unauthorized access to forgotten information, making the framework suitable for deployment in sensitive and regulated environments. Overall, by embedding removability into the architecture itself, this work establishes a new foundation for designing modular, scalable and ethically responsive AI models.
Concept-based models have been proposed as a new line of research for explainable by-design deep learning models. However, those models show their whole power when applied to benchmarks where the concepts are well defined and the concepts' attributes easily extractable from the raw data. In this paper, we challenge the most recent concept-based model initially developed for image classification, on more complex interpretative tasks from a recently proposed video benchmark where they perform poorly. We conduct a root cause analysis of the poor performances of state-of-the-art explainable concept-based models for these multimodal interpretative tasks, and propose adaptations to design robust explainable models for detecting character objectification in this novel challenging video benchmark. We show that the optimal architectural choice may vary depending on the modality setting, thereby showing that designing multimodal concept-based approaches remains an open challenge and calls for further investigation.
Understanding and measuring Quality of Experience (QoE) is crucial for optimized but still user-centered multimedia systems. However, current assessment methods rely largely on one-dimensional subjective ratings collected post hoc and therefore fail to capture how users actually experience quality in real time. Inspired by advances in neuroimaging, we investigate whether QoE can be assessed directly from brain activity. We propose a novel approach using functional near-infrared spectroscopy (fNIRS) to objectively measure perceptual quality during multimedia service interaction. In a preliminary study with 8 participants, we recorded fNIRS signals while viewers watched videos of varying quality. Results show a statistically significant increase in oxygenated-hemoglobin in the prefrontal cortex in low quality conditions, suggesting elevated cognitive effort or reduced perceptual fluency. These findings establish a neural signature of degraded quality perception and demonstrate the usefullness of fNIRS for neuro-based objective QoE estimation. Unlike traditional techniques, our method provides continuous, real-time, implicit quality measurement without interrupting the user. This work calls for a rethinking of QoE as a neuroperceptual phenomenon rather than a subjective judgment and propose a neuro-based QoE framework.
Musical structure spans nested timescales, from fine-grained fluctuations to long-range organization. To capture this, we apply Detrended Fluctuation Analysis (DFA) and Multifractal Detrended Fluctuation Analysis (MFDFA) to compare human-composed music (Billboard Top 5 hits, 1950-2024) with AI-generated outputs from Suno, DiffRhythm, and YuE. While all models capture fractal properties of music, differences persist. Suno aligns most closely with human music but shows reduced fine-scale variability. DiffRhythm yields narrower spectra and lower complexity, while YuE matches large-scale structure yet exhibits greater small-scale variability. Decade-level analysis shows divergence is smallest for earlier, more homogeneous eras (1950s-1960s) and greatest during periods of stylistic diversity and production complexity (2000s-2020s). We propose integrating fractal descriptors into training objectives and refining architectures to improve temporal sensitivity, advancing AI systems toward more structurally representative and authentic music generation. The code and results can be found at https://github.com/zhangkkevin/billboard-ai-fractal-comparison.
The future of digital marketing lies in the convergence of human creativity and generative AI, where insight, strategy, and storytelling are co-authored by intelligent systems. We present MindFuse, a brave new explainable generative AI framework designed to act as a strategic partner in the marketing process. Unlike conventional LLM applications that stop at content generation, MindFuse fuses CTR-based content AI-guided co-creation with large language models to extract, interpret, and iterate on communication narratives grounded in real advertising data. MindFuse operates across the full marketing lifecycle: from distilling content pillars and customer personas from competitor campaigns to recommending in-flight optimizations based on live performance telemetry. It uses attention-based explainability to diagnose ad effectiveness and guide content iteration, while aligning messaging with strategic goals through dynamic narrative construction and storytelling. We introduce a new paradigm in GenAI for marketing, where LLMs not only generate content but reason through it, adapt campaigns in real time, and learn from audience engagement patterns. Our results, validated in agency deployments, demonstrate up to 12× efficiency gains, setting the stage for future integration with empirical audience data (e.g., GWI, Nielsen) and full-funnel attribution modeling. MindFuse redefines AI not just as a tool, but as a collaborative agent in the creative and strategic fabric of modern marketing. In the end of the paper, we also provide a forward-looking forecast on how platforms like MindFuse and Google DeepMind's ACAI are likely to shape the future of marketing agencies. These systems will not only reset client expectations toward greater transparency, speed, and personalization, but will also redefine the skillsets demanded from the new generation of marketers. As hybrid agencies emerge-blending creative storytelling with data science and AI engineering-the competitive landscape will increasingly hinge on talent. In this new environment, professionals will be expected to pair human imagination with technical fluency, while agencies will need to reinvent themselves as AI facilitators and curators of brand authenticity amidst the ongoing talent war led by platforms such as Meta.