Live streaming platforms require real-time monitoring and reaction to social signals, utilizing partial and asynchronous evidence from video, text, and audio. We propose StreamSense, a streaming detector that couples a lightweight streaming encoder with selective routing to a Vision–Language Model (VLM) expert. StreamSense handles most timestamps with the lightweight streaming encoder, escalates hard/ambiguous cases to the VLM, and defers decisions when context is insufficient. The encoder is trained using (i) a cross-modal contrastive term to align visual/audio cues with textual signals, and (ii) an IoU-weighted loss that down-weights poorly overlapping target segments, mitigating label interference across segment boundaries. We evaluate StreamSense on multiple social streaming detection tasks (e.g., sentiment classification and hate content moderation), and the results show that StreamSense achieves higher accuracy than VLM-only streaming while only occasionally invoking the VLM, thereby reducing average latency and compute. Our results indicate that selective escalation and deferral are effective primitives for understanding streaming social tasks. Code is publicly available on GitHub.
论文检索
输入标题、作者或关键词，从 8,659 篇学术成果中精准定位
High-quality annotated datasets are crucial for advancing machine learning in medical image analysis. However, a critical gap exists: most datasets either offer a single, clean ground truth, which hides real-world expert disagreement, or they provide multiple annotations without a separate gold standard for objective evaluation. To bridge this gap, we introduce CytoCrowd, a new public benchmark for cytology analysis. The dataset features 446 high-resolution images, each with two key components: (1) raw, conflicting annotations from four independent pathologists, and (2) a separate, high-quality gold-standard ground truth established by a senior expert. This dual structure makes CytoCrowd a versatile resource. It serves as a benchmark for standard computer vision tasks, such as object detection and classification, using the ground truth. Simultaneously, it provides a realistic testbed for evaluating annotation aggregation algorithms that must resolve expert disagreements. We provide comprehensive baseline results for both tasks. Our experiments demonstrate the challenges presented by CytoCrowd and establish its value as a resource for developing the next generation of models for medical image analysis.
With the increasing relevance of web mining and content analysis in uncovering mobility patterns from large-scale online data, traffic flow prediction plays a crucial role in proactive urban planning and enhancing the responsiveness of intelligent transportation systems. However, existing traffic flow prediction methods often fail to explicitly capture correlations in continuous multivariate sequences that are naturally suited for trend and periodic pattern extraction by vision models and rely on fixed spatial graphs that neglect the cognitive advantages of granular-ball structures, which limits their ability to model interactions and strengthen spatiotemporal dependencies. To address these challenges, this paper proposes a Cross-domain Mamba framework that integrates Multi-scale Imaging and Granular-ball Computing for traffic flow prediction (MIGC-CMamba). First, a multi-scale sequence imaging method is presented, which converts the original time series into image modality and leverages MambaVision to capture both local and global dependencies. Second, a multi-granularity spatial graph is constructed via granular-ball clustering, which balances global trend representation and local detail preservation. Third, a cross-domain enhancement mechanism adaptively integrates temporal and spatial domains, strengthening spatiotemporal dependencies. Lastly, extensive experiments demonstrate superior performance over state-of-the-art baselines, highlighting how vision-based imaging, cognition-inspired granular-ball modeling, and content-aware mining jointly advance the modeling of spatiotemporal dependencies in traffic flow prediction.
The advent of machine learning as a service (MLaaS) has necessitated secure multi-party computation (MPC)-based private inference (PI) to address the privacy concerns that arise when web servers offer query inference services to users. However, the formal privacy protection incurs substantial communication and latency overheads, particularly for large models such as vision transformers (ViTs). Existing methods either ignore the inherent attention dependencies or naively extend CNN optimizations to ViTs despite their structural discrepancies, resulting in sub-optimal performance. In this paper, we co-design the MPC and architectural properties of ViT and propose SecViT, an efficient and secure inference framework that automatically adapts ViTs into privacy-friendly counterparts with optimal attention configurations at different layers and tokens under MPC, balancing model capability and efficiency. SecViT features an MPC-efficient, layer-dependent and load-balancing attention representation adapter to facilitate feature reuse across multiple highly correlated layers without impacting accuracy. To further reduce the inference cost, SecViT also develops fine-grained composite attention and activation approximation algorithms to achieve superior accuracy-efficiency trade-offs. Experiments show that SecViT reduces communication by 6.0x and latency by 4.6x with iso-accuracy over MPCViT, and improves accuracy by 4.92% with 1.6x lower latency over PriViT on Tiny-ImageNet. Compared with state-of-the-art PI protocols, SecViT further achieves 10.0x communication saving and 7.9x latency reduction over BumbleBee.
High-resolution spatial fields are critical for local decision-making, yet many operational and scientific workflows produce coarse outputs due to computational limits. We present CrossRefine, a deployable microservice for cross-domain spatial super-resolution that enhances multi-channel spatial tiles without modifying upstream models. It is built around a unified, topography-conditioned adversarial UNet trained across geographically diverse regions to ensure robustness to heterogeneous terrains and domain shifts. Unlike region-specific enhancement models, the system generalizes across domains within a single architecture, balancing numerical fidelity and structural realism through a hybrid regression–adversarial objective. The service provides REST API endpoints for batch and streaming inference, supports mixed-precision, and offers per-tile diagnostics and confidence maps to promote safe deployment. In our demo, we show interactive refinement of coarse spatial inputs, side-by-side comparison with interpolation and non-adversarial baselines, and real-time profiling of latency and throughput on commodity hardware. CrossRefine illustrates how spatial super-resolution can be delivered as a practical AI microservice, enabling scalable refinement of existing computational workflows without requiring higher-resolution upstream simulations. Demonstration video: https://shorturl.at/lz2un
We introduce a web tool that puts the full image-to-text pipeline for Ancient Egyptian hieroglyphs inside a single annotation workspace. Instead of only producing a final transcription, our system exposes editable intermediate results so users can validate and correct the pipeline step by step. User edits are stored as image-aligned annotations, which supports both text analysis and dataset creation. Quantitative results indicate improved efficiency and output quality relative to a manual baseline. A demonstration video is available at Google Drive.
We present ElderMTL, a multi-task affect monitoring system designed for elderly care settings. The system simultaneously estimates Facial Action Units (FAUs), Valence-Arousal (VA) signals, and categorical emotions (FER) from video, capturing multiple layers of affective information. To improve sensitivity to subtle affective cues common in older adults, our approach incorporates age-conditioned physiological modeling, including baseline muscle adjustments and a dynamic AU co-activation graph. This enables the system to adapt to age-related changes in facial expression patterns, providing more reliable and interpretable emotion assessments. In a live demonstration, we showcase ElderMTL processing video streams, visualizing AU activations, affective state predictions, and interpretable insights that highlight age-specific affective dynamics. This work demonstrates that physiologically grounded, multi-task affective monitoring can provide meaningful, real-world support for elderly care.
We present SparseDR, a novel differentiable rendering algorithm designed for sparse representations based on Signed Distance Fields (SDF). We leverage the Sparse Brick Set representation and propose an adaptation of redistancing for sparse SDFs. This enables SparseDR to surpass existing works in accuracy of surface reconstruction by increasing the effective resolution of the SDF representation. SparseDR is efficiently implemented in C++ and Vulkan, achieving several times shorter reconstruction time than other methods.
While Open-Set Semantic Mapping and 3D Semantic Scene Graphs (3DSSGs) have become established paradigms in robotic perception in recent years, most existing works are limited to small environments or sacrifice geometric detail and instance granularity for scalability. Deploying these systems at scale for large multi-room environments remains a major challenge due to the computational overhead of high-dimensional feature integration and the maintenance of the 3DSSG structure. In this paper, we demonstrate a modular mapping architecture that establishes 3D Semantic Scene Graphs (3DSSGs) as its foundational backend. Unlike approaches that generate scene graphs as a post-processing step, our system maintains the graph as the primary, incrementally updated knowledge representation. Our architecture is optimized for GPU-accelerated operations, enabling the dense representation of extensive environments containing thousands of unique object instances, supporting open-vocabulary queries via CLIP features without requiring any additional post-processing steps. In this live demonstration, we showcase our pipeline processing large-scale data from the Habitat Matterport 3D (HM3D) dataset as well as live data collected from a handheld device. Attendees will interact with the generated maps by performing real-time, open-set queries (e.g., “find the vintage wooden chair”) across complex, multi-story environments, highlights the system's capability to represent dynamic, human-aligned environmental understanding suitable for downstream robotic tasks.
Large-scale video repositories are increasingly available for modern video understanding and generation tasks. However, transforming raw videos into high-quality, task-specific datasets remains costly and inefficient. We present DataCube, an intelligent platform for automatic video processing, multi-dimensional profiling, and query-driven retrieval. DataCube constructs structured semantic representations of video clips and supports hybrid retrieval with neural re-ranking and deep semantic matching. Through an interactive web interface, users can efficiently construct customized video subsets from massive repositories for training, analysis, and evaluation, and build searchable systems over their own private video collections. The system is publicly accessible at https://datacube.baai.ac.cn/. Demo Video: https://youtu.be/L7bKfPBm2tU
Forest imagery analysis often involves multiple tightly coupled vision tasks, which must be performed under substantial variation in geographic regions, sensors, and acquisition conditions. However, practitioners often lack a unified tool that is geospatial-native, cloud-optimized, and ML-integrated for end-to-end workflows spanning annotation, prediction, visualization, and downstream analysis at scale. We present AwakeForest, an interactive end-to-end platform designed for large-scale forest imagery that integrates model-assisted inference, automatic annotation, and human-in-the-loop refinement within a single workflow. Our platform supports plug-and-play integration of pretrained models and enables scalable interaction with forest imagery ranging from standard aerial scenes to large orthomosaics that can span several gigabytes to hundreds of gigabytes. AwakeForest produces analysis-ready outputs that can be directly used for downstream analysis and to support iterative model and annotation updates on new scenes. We demonstrate the system on the PALMS dataset and illustrate how AwakeForest supports an end-to-end workflow for practical forest management and analysis.
We present an AI-powered interactive multimodal system that enriches digital image accessibility for blind and visually impaired (BVI) users. Our demonstration showcases two application domains: (1) an educational digital book, and (2) an online shopping interface. In both use-cases, users can virtually feel material textures (leather, wood, etc.) and engage in voice-driven inquiry about images. The system integrates state-of-the-art AI components – including voice-to-voice conversational agents, vision models for object segmentation, and a custom 16-actuator vibrotactile display – to provide multimodal feedback (haptic vibrations, spoken descriptions, and audio cues). The result is an inclusive technology with significant societal benefit, empowering BVI users to learn and shop more independently through natural multimodal interactions.
Precise descriptions of soil horizons are required for policy makers, agriculture and many applications in civil engineering. Up to date correct soil horizon annotations require human experts as they follow complex hierarchical taxonomies. We present the SoilNet App, a web-based demonstrator that guides experts through relevant tasks for expert-level soil horizon annotations from soil profile images. To demonstrate the reliability of the SoilNet app we present results of a user study with soil horizon annotation experts, which highlights the difficulty of image-only-based annotation and suggests that collaborating with our model not only increases expert performance but also improves inter-annotator consistency. Our app is publicly accessible (https://soilnet.demo.calgo-lab.de).
The rapid advancement of Artificial Intelligence (AI) has accelerated the development of personalised healthcare. However, the clinical adoption of deep learning remains constrained by a persistent âtrust gapâ surrounding model transparency, security, and data privacy. This talk presents a research vision for building trustworthy human-centred AI in computer audition and biosignal processing by integrating signal processing, machine learning, and healthcare. This vision is structured around three interconnected pillars: Sensing, Understanding, and Empowerment. The Sensing pillar focuses on enabling machines to perceive clinically relevant information from multimodal biosignals, transforming body sounds and physiological signals into non-invasive, accessible, and cost-effective windows into human health and well-being. The Understanding pillar addresses the black-box nature of modern AI systems, with the goal of ensuring transparency, efficiency, robustness, and security in clinical deployment. This is achieved through the development of explainable AI methods, knowledge distillation techniques, and defences against adversarial attacks, fostering AI systems that clinicians can trust and interpret. The Empowerment pillar seeks to restore natural communication for individuals with speech impairments, such as laryngectomy patients. By developing Silent Speech Interfaces (SSIs) that translate facial muscle activity (EMG) directly into audible speech, this research advances speech intelligibility, naturalness, real-time causal architectures, and multi-speaker communication scenarios. Collectively, these research directions aim to establish a foundation of trustworthy, human-centred AI that is not only accurate and secure, but also empathetic, accessible, and impactful, ultimately enhancing healthcare, communication, and quality of life.
Controllable generation guided by external knowledge is a key requirement in modern generative deep learning applications, enabling the synthesis of samples with explicit constraints on semantic content, structural properties, and variability. In 3D Computed Tomography (CT), such control is essential for clinical applications, including data augmentation, privacy-preserving data sharing, and the simulation of specific anatomical or pathological scenarios. While research on conditional 3D CT generation has expanded rapidly, the diversity of existing approaches makes systematic comparison difficult and obscures fundamental design choices. In this survey, we propose a conditioning-centric taxonomy that organizes the literature along three orthogonal dimensions: the type of external knowledge (K), the knowledge integration paradigm (I), and the generative architecture (A). This factorization defines an explicit design space (K x I x A) that provides a unified perspective on prior work. Using this framework, we systematize existing methods, identify dominant trends and recurring design patterns, and highlight underexplored regions of the design space that point toward promising directions for future research.
3D scene generation has rapidly evolved, significantly promoting the innovation of content creation. In this context, interaction techniques serve as a pivotal bridge connecting user intent with the generative models, thereby enabling precise control, real-time feedback and personalized customization of complex 3D scenes. Existing literature reviews predominantly focus on general generative paradigms, or are limited to specific subdomains such as single-object modeling, while often overlooking the systematic classification of interaction mechanisms. To bridge this gap, this work presented a comprehensive survey of interaction techniques in 3D scene generation. We proposed a unified taxonomy that categorized existing methods into three primary paradigms: Interactive Generation, Interactive Editing, and Embodied Interaction. For each category, we analyzed representative methods in terms of controllability, interaction granularity, and physical consistency, and discussed their advantages and limitations. We further summarized commonly used datasets and evaluation protocols for interactive 3D scene generation. Finally, we discussed the future directions toward more physically grounded, multi-modal, and user-centered interactive 3D scene generation systems. A curated list of the related papers mentioned in this work can be found at Awesome-Interactive-Techniques-in-3D-Scene-Generation-Lists.
Multimodal Emotion Recognition (MER) focuses on identifying and interpreting emotions from modality-compound inputs. Closely mirroring human cognitive processes in real-world environments, MER has drawn substantial attention from both academia and industry. Recently, a paradigm shift has been unveiled in MER, from leveraging small-scale, task-specific models to Large Language Models (LLMs). We refer to the latter as the MER-with-LLMs paradigm, which offers unprecedented generality, spurring numerous empirical attempts, even alongside speculation about their potential to achieve general emotional intelligence. However, with these new opportunities come new challenges, including the scarcity of emotionally annotated data, the affective gap both within and across modalities, and the opacity of affective interpretation. To systematically review existing research and guide future exploration, this paper categorizes prior works according to their focus on addressing these challenges into three directions: Affective Data Augmentation, Multimodal Affective Representation, and Multimodal Affective Reasoning. By thoroughly tracing the development, emerging trends, and remaining issues within each direction, this paper aims to provide a clear academic map of the MER-with-LLMs paradigm and foster its structured advancement.
Vein biometrics has emerged as a promising biometric modality for personal identity authentication, benefiting from its intrinsic properties such as high discriminative capability, resistance to forgery, and contactless acquisition. Recent advances in artificial intelligence, particularly deep learning, have significantly accelerated its development. This paper presents a comprehensive and systematic survey of AI-enhanced vein biometrics. We review fundamental principles, publicly available datasets, and evaluation protocols, and systematically analyze existing methods across the entire vein biometric pipeline, including acquisition, preprocessing, feature extraction, recognition and verification, security and privacy protection, and multimodal fusion. Furthermore, we summarize representative application scenarios, identify key challenges, and highlight promising directions for future research. To facilitate reproducible research and long-term development of the field, we release an open, evolving research resource Awesome-Vein-Biometrics that systematically summarizes and tracks recent advances in vein biometrics.
Endoscopic surgery demands continuous real-time visual decision-making under severe constraints, including a limited field of view, motion blur, and dynamically deforming anatomy. These factors impose substantial cognitive load on surgeons and motivate the integration of artificial intelligence (AI) throughout the endoscopic surgical workflow. This survey reviews recent progress in AI for endoscopic surgery and organizes the literature into four stages that span perception to action: (1) image enhancement and analysis methods that improve visual perception; (2) multimodal video understanding approaches that model and reason surgical instruments and anatomical structures over space and time; (3) 3D reconstruction techniques that enable robust tracking and interpretation of deformable anatomy; and (4) emerging paradigms of embodied surgical intelligence, where action-conditioned world models link perception to intraoperative assistance. Across these stages, we summarize current capabilities and limitations and identify key open challenges for clinical deployment. In addition, we provide an overview of 18 publicly available datasets, highlighting their scope and annotations. We hope this survey will stimulate further research toward reliable and clinically deployable AI systems for endoscopic surgery.
Graphs are a fundamental data structure underlying many problems in the natural and social sciences. Over the past decade, Graph Neural Networks (GNNs) have dominated graph machine learning, supported by solid theoretical foundations. Yet scientists often understand graph structure through vision: chemists read molecular diagrams and social scientists inspect network visualizations. Despite decades of work on graph visualization, most graph learning pipelines still treat graphs purely as symbolic structures, rarely leveraging the visual form of graphs. We argue that this gap deserves renewed attention in the era of powerful vision and vision language models. This survey provides a first systematic overview of the emerging area we term vision meets graphs, which treats visual depictions of graphs as first-class inputs for reasoning and learning. We organize existing work into three threads. Vision for Graph Reasoning studies how models can use visual depictions of graphs to understand structure and carry out multi-step reasoning. Vision for Graph Learning explores how visual features can complement or augment graph encoders beyond known limitations of message passing. Scientific Graphs examines domains where standardized depiction conventions support both reasoning and learning. Our goal is to clarify what current methods can and cannot do, and to outline a path toward foundation models that perceive and reason about graphs as scientists do.