In this companion paper, we reproduce the experiments presented in our work titled ''NIF: A Fast Implicit Image Compression with Bottleneck Layers and Modulated Sinusoidal Activations'' [2], presented at ACM Multimedia 2023. In this study, we present the architecture and the technical details of our implementation and provide instructions to reproduce the main results, the ablation study, the plots and the figures presented in the paper. All the material described in this paper is released on GitHub [3], featuring the full results, a reference software implementation and a generic environment setup that works on any system, even without a GPU.
论文检索
输入标题、作者或关键词，从 1,620 篇学术成果中精准定位
1,620篇论文
第 8 / 81 页
In this paper, we reproduce the experimental results presented in our earlier work titled ''Maskable Retentive Network for Video Moment Retrieval'' that was published in the 32nd ACM International Conference on Multimedia (MM '24). This paper aims to verify the soundness of our previous results and to help others better understand our software framework. We present artifacts that enable the reproduction of the results from our earlier work, including details on dataset preparation, source code, and the experimental environment. Using these artifacts, we demonstrate the reproducibility of our results and invite researchers to explore our software framework beyond reproducibility efforts.
Effective video retrieval in large-scale datasets presents a significant challenge, with existing tools often being too complex, lacking sufficient retrieval capabilities, or being too slow for rapid search tasks. This paper introduces diveXplore, an open-source software designed for interactive video retrieval. Due to its success in various competitions like the Video Browser Showdown (VBS) and the Interactive Video Retrieval 4 Beginners (IVR4B), as well as its continued development since 2017, diveXplore is a solid foundation for various kinds of retrieval tasks. The system is built on a three-layer architecture, comprising a backend for offline preprocessing, a middleware with a Node.js and Python server for query handling, and a MongoDB for metadata storage, as well as an Angular-based frontend for user interaction. Key functionalities include free-text search using natural language, temporal queries, similarity search, and other specialized search strategies. By open-sourcing diveXplore, we aim to establish a solid baseline for future research and development in the video retrieval community, encouraging contributions and adaptations for a wide range of use cases, even beyond competitive settings.
Over the past few decades, significant resources have been invested in developing video codecs for the storage and delivery of video data. International standards development organizations, such as MPEG (Moving Picture Experts Group) and AOM (Alliance for Open Media), have fostered large-scale competitive collaboration among industry players. While open-source software developed alongside these video codec standards has accelerated verification of the standard and enabled rapid adoption, as it is used as a starting point for implementation and future research, this practice has not been fully applied to the ecosystem of mezzanine video codecs used in high-quality video capture and post-production. To address this gap, the industry's first open source project for collaborative innovation in professional video codecs, OpenAPV, has been established. The project aims to facilitate collaborative research and development of a royalty free, open source, and open standard video codec for professional use. This paper outlines the technical aspects of the video codec, together with the open source software implementation. The codec has been successfully tested across various platforms and shows excellent R-D (Rate-Distortion) performance and coding speed, even without hardware acceleration. Furthermore, it demonstrates robust quality resilience against multiple rounds of encoding and decoding cycles, which are common in video post-production. The open source project is hosted at https://github.com/AcademySoftwareFoundation/openapv.
Motion artifacts in structural and functional magnetic resonance imaging (MRI) pose a significant challenge for both clinical use and machine learning (ML)-based image analysis. Existing ML approaches for artifact correction require paired clean and corrupted datasets, which are difficult to acquire. We present py-simpace, an open-source, pip-installable MRI motion artifact simulation toolkit with native ML integration. py-simpace supports structural MRI and functional MRI (fMRI) simulation, offering configurable k-space and image-space motion, ghosting, Gibbs ringing, and physiological noise. It provides an end-to-end pipeline with a ready-to-use PyTorch Dataset interface for ML training. We describe the design of py-simpace v2.0, compare it with existing tools, and demonstrate its utility for robust artifact correction model development.
The vitrivr multimedia retrieval stack has been shown to be versatile and effective in many areas and application domains. In this paper, we present vitrivr's new retrieval engine, the vitrivr-engine, a flexible and highly configurable information extraction and query processing solution with support for a broad range of media types.
The growth of online educational content, particularly slide-based video lectures, has created a need for tools that enhance navigation, comprehension, and accessibility. Many existing systems for video analysis are closed-source, hindering reproducibility and extension. To address this, we present the Lecture Video Analysis Toolkit, an open-source, proof-of-concept application designed for the multimodal analysis of slide video lectures. The toolkit integrates a processing pipeline that includes scene detection, visual entity extraction, transcription, optical character recognition (OCR), and semantic linking between spoken and visual content using embeddings. A key contribution is its interactive interface, motivated by early user feedback, that allows for customization of the viewing experience to suit individual preferences. The entire system is openly available and serves as a research prototype for validating the potential of multimodal analysis in creating more inclusive and improved learning experiences. A live demo is accessible at https://travis-seng.fr/svla, and the source code is openly available at https://github.com/travisseng/svla-toolkit.
Multimodal knowledge graphs often separate easily represented information (text) from that which is not (multimedia documents like images, videos, or audio). This severely limits query expressiveness, as the engines lack access to the node contents stored externally. We present MeGraS, the MediaGraph Store, a novel storage and query engine for multimodal knowledge graphs. By storing multimedia documents directly in the graph, MeGraS allows the query engine to leverage their content for enhanced capabilities, making it natively capable of performing operations such as k-NN, segmentation, or deriving non-materialized relations based on visual features. To demonstrate this, we incorporate and extend the pattern-matching query language SPARQL, resulting in a unified framework for storing and managing multimodal knowledge graphs with advanced expressiveness. MeGraS is available as open-source software: http://megras.org
Open-source foundation models are essential for advancing music audio understanding and ensuring access to general-purpose representations for music information retrieval. To this end, we present OMAR-RQ, a model trained with self-supervision via masked token prediction using a large-scale dataset with over 330,000 hours of music audio. We experiment with various input features and quantization options, outperforming existing open self-supervised models in music tagging, pitch estimation, chord recognition, beat tracking, segmentation, and difficulty estimation. Finally, we release our training and evaluation pipelines and model weights at https://github.com/mtg/omar-rq.
Recent advancements in image generative foundation models have prioritized quality improvements but often at the cost of increased computational complexity and inference latency. To address this critical trade-off, we introduce HiDream-I1, a new open-source image generative foundation model with 17B parameters that achieves state-of-the-art image generation quality within seconds. HiDream-I1 is constructed with a new sparse Diffusion Transformer (DiT) structure. Specifically, it starts with a dual-stream decoupled design of sparse DiT with dynamic Mixture-of-Experts (MoE) architecture, in which two separate encoders are first involved to independently process image and text tokens. Then, a single-stream sparse DiT structure with dynamic MoE architecture is adopted to trigger multi-model interaction for image generation in a cost-efficient manner. To support flexiable accessibility with varied model capabilities, we provide HiDream-I1 in three variants: HiDream-I1-Full, HiDream-I1-Dev, and HiDream-I1-Fast. Furthermore, we go beyond the typical text-to-image generation and remould HiDream-I1 with additional image conditions to perform precise instruction-based editing on given images, yielding a new image editing model namely HiDream-E1. We have open-sourced all the codes and model weights of HiDream-I1 and HiDream-E1: https://github.com/HiDream-ai/HiDream-I1 and https://github.com/HiDream-ai/HiDream-E1. These models quickly gained strong traction in the community, ranking among the top globally on the Hugging Face Models Trending list within just one week of launch. In under a month, it surpassed 280,000 downloads and has been officially integrated into the Diffusers library. It is now widely adopted by leading community tools and products, including ComfyUI, Recraft, WaveSpeedAI, fal.ai, and Pruna AI - reflecting the model's growing impact across the open-source AI ecosystem.
Currently, artificial intelligence is profoundly transforming the audio domain; however, numerous advanced algorithms and tools remain fragmented, lacking a unified and efficient framework to unlock their full potential. Existing audio agent frameworks often suffer from complex environment configurations and inefficient tool collaboration. To address these limitations, we introduce AudioFab, an open-source agent framework aimed at establishing an open and intelligent audio-processing ecosystem. Compared to existing solutions, AudioFab's modular design resolves dependency conflicts, simplifying tool integration and extension. It also optimizes tool learning through intelligent selection and few-shot learning, improving efficiency and accuracy in complex audio tasks. Furthermore, AudioFab provides a user-friendly natural language interface tailored for non-expert users. As a foundational framework, AudioFab's core contribution lies in offering a stable and extensible platform for future research and development in audio and multimodal AI. The code is available at https://github.com/SmileHnu/AudioFab.
Tyee: A Unified, Modular, and Fully-Integrated Configurable Toolkit for Intelligent Physiological Health Care
PDF ↗Deep learning has shown great promise in physiological signal analysis, yet its progress is hindered by heterogeneous data formats, inconsistent preprocessing strategies, fragmented model pipelines, and non-reproducible experimental setups. To address these limitations, we present Tyee, a unified, modular, and fully-integrated configurable toolkit designed for intelligent physiological healthcare. Tyee introduces three key innovations: (1) a unified data interface and configurable preprocessing pipeline for 12 kinds of signal modalities; (2) a modular and extensible architecture enabling flexible integration and rapid prototyping across tasks; and (3) end-to-end workflow configuration, promoting reproducible and scalable experimentation. Tyee demonstrates consistent practical effectiveness and generalizability, outperforming or matching baselines across all evaluated tasks (with state-of-the-art results on 12 of 13 datasets). The Tyee toolkit is released at https://github.com/SmileHnu/Tyee and actively maintained.
Recent years have brought about a surge in neuromorphic ''event'' video research, primarily targeting computer vision applications. Event video eschews video frames in favor of asynchronous, per-pixel intensity samples. While much work has focused on a handful of representations for specific event cameras, these representations have shown limitations in flexibility, speed, and compressibility. We previously proposed the unified ADΔER representation to address these concerns. This paper introduces numerous improvements to the adder-viz software for visualizing real-time event transcode processes and applications in-the-loop. The MIT-licensed software is available from a centralized repository at https://github.com/ac-freeman/adder-codec-rs.
With the rapid growth of digital content and the increasing demand for high-resolution displays, the efficient compression of screen content images characterized by text, graphics, and UI elements has become an important research field. This paper introduces a new dataset named SCID-Compress900 specially designed for image compression research. The dataset consists of 900 high-quality screen content images, including 500 4K images and 400 1080P images. All these images are mainly composed of text/graphics, reflecting typical screen content scenarios such as office documents, software interfaces, and presentation slides. The dataset covers a diverse range of content, including various font sizes, graphic styles, and color modes, providing a comprehensive testbed for compression algorithms. To demonstrate the effectiveness of SCID-Compress900, we conduct benchmark tests using several deep learning-based image compression methods commonly employed by researchers. The experimental results show that SCID-Compress900 can well differentiate the performance of different compression algorithms. Compared with existing datasets, SCID-Compress900 offers higher resolution, larger scale, and more targeted content, making it an ideal resource for developing and evaluating advanced image compression algorithms for screen content. This dataset will not only promote the research and development of screen content compression technology but also contribute to the standardization and optimization of compression algorithms in practical applications. The project is available at https://openi.pcl.ac.cn/OpenDatasets/SCID-Compress900.
Amidst the swift advancement of 3D vision technology, Multi-view Compression (MVC) has become a crucial technique, widely applied in fields such as virtual reality, augmented reality, autonomous driving, telemedicine, and security surveillance. The technology effectively handles views from multiple cameras, utilizing the inter-view correlations to compress data efficiently. It substantially decreases the data transmission and storage requirements, enabling a richer and more realistic visual experience within the same bandwidth constraints. To further enhance compression performance, new methods continue to emerge. However, the absence of a unified benchmark testing library capable of effectively evaluating existing algorithms poses significant challenges to the further development of the field and the practical deployment of algorithms. To address this issue, we introduce OpenMVC, an Open-Source Library for Learning-based Multi-view Compression. We provide a comprehensive description and analysis of the performance advantages of existing algorithms. Furthermore, we conduct extensive and comprehensive benchmark testing of nine typical algorithms in the last five years, evaluating them in a consistent environment across various metrics. The open-source library for OpenMVC is available at https://openi.pcl.ac.cn/OpenAICoding/OpenMVC.
We present Open-CD, a change detection toolbox that contains a rich set of change detection methods as well as related components and modules. The toolbox started from a series of open source general vision task tools, including OpenMMLab Toolkits, PyTorch Image Models (Timm), etc. It gradually evolves into a unified platform that covers many popular change detection methods and contemporary modules. It not only includes training and inference codes, but also provides some useful scripts for data analysis. We believe this toolbox is by far the most comprehensive change detection toolbox. In this report, we introduce the features, supported methods and applications of Open-CD. In addition, we also conduct a benchmarking study on different methods and components. We wish that the toolbox and benchmark could serve the growing research community by providing a flexible toolkit to re-implement existing methods and develop their own new change detectors. Code and models are available at https://github.com/likyoo/open-cd.
For thousands of years, Chinese culture has regarded calligraphy as both a form of painting and a deeply physical, philosophical practice. While traditionally manifesting as a two-dimensional art form, Chinese calligraphy emerges from three-dimensional, embodied movements infused with breath, emotion, and intention. This demo/video paper introduces Embodied Ink, a multimedia interactive installation that reinterprets Chinese calligraphy through motion capture and generative AI. By translating the audience movements into real-time dynamic visuals and soundscapes, the project reveals the hidden kinetics and philosophical depth underlying calligraphy. Embodied Ink reimagines the static medium of calligraphy as a dynamic interplay of forces and particles, inviting viewers to experience calligraphy as a living, ever-changing art form.
Nüshu (Jiangyong Nüshu), a script developed and practiced exclusively by women in China, holds recognition as National Intangible Cultural Heritage. This paper presents the design and implementation of an immersive multimodal interaction system centered on The Song of Nüshu, a foundational cultural artifact. By integrating Natural Interaction (NI) and Mixed Reality (MR) technologies, the system constructs a visual environment inspired by classical Chinese landscape painting aesthetics. Within this experiential space, users engage with representations of pivotal female historical figures and events interwoven with Nüshu textual elements. This framework enables participant-driven narrative construction that foregrounds Nüshu's aesthetic dimensions and socio-cultural significance. Our work advances a heritage preservation methodology that innovatively reinterprets tradition while explicitly foregrounding women's historical agency and expressive practices.
In this article, we present a multimedia system for reading a sonagram using a smartphone, the two main features of which are that it is both interactive and real-time.
RhythmGate reimagines the elevator as a spatial medium for sensing and expressing the unspoken codes of shared space. By combining pressure-sensitive flooring with overhead visual sensing, the system captures bodily presence and spatial movement in real time. These inputs are processed to classify position and trigger one of five pre-recorded human voices, selectively layered into a responsive soundscape. As riders shift their stance, they ''cut'' between temporal zones mapped across the floor, composing a living acoustic montage. The resulting auditory feedback surfaces subtle social dynamics-such as avoidance, boundary-setting, or latent tension-and gently dissolves them. In doing so, RhythmGate transforms a routine ascent into a fleeting, multisensory conversation among strangers, system, and space.