In today's era, three-dimensional point cloud data is not only voluminous but also widely applicable. Therefore, data compression has become a crucial step prior to processing. Although existing 3D point cloud compression techniques primarily focus on fidelity, in practical applications, the vast majority of compressed data serves machine perception tasks. Therefore, point cloud compression tailored for machine perception becomes particularly significant. To address this problem, we introduce an innovative point cloud compression algorithm library specifically designed for both machine and human perceptual requirements. This library represents the first collection of multi-perception point cloud compression algorithms on the PyTorch platform, integrating eleven advanced, learning-based algorithms. We category and analyze these algorithms in depth, according to different analysis tasks, to facilitate a better understanding and comparison. Moreover, we successfully replicate these algorithms and meticulously organize the pre-processing of point cloud data and the analysis networks for downstream tasks. Ultimately, we conduct experiments on multiple perceptual datasets for compression and analysis tasks, with results comprehensively summarized across various performance metrics. We will continue to update these algorithms to ease their adoption by researchers.
论文检索
输入标题、作者或关键词，从 1,237 篇学术成果中精准定位
Three-dimensional point cloud data is one of the most extensively used data representation today, favored in various fields for its realistic and lifelike visual effects. However, the substantial volume of data poses significant challenges for storage and transmission. To advance point cloud compression (PCC) technology, we develop a learning-based PCC algorithm library, namely LearningPCC. To our knowledge, this is the first comprehensive set of algorithms that is compatible with all types of point cloud data. This PyTorch library incorporates eleven learning-based algorithms that address both geometry and attribute compression of point cloud data. We categorize the existing methods into six main classes and thoroughly introduce and analyze the principles of these algorithms. Moreover, we conduct performance evaluations using point clouds with various densities, offering detailed test results on several compression metrics, such as RD curves, BD-BR gains, compression ratio improvements, and encoding times. We will provide researchers with convenient access to these methods, replicate codes, and experiment results. Our commitment includes maintaining and updating these algorithms to offer researchers the latest in compression technologies.
Versatile Video Coding (VVC/H.266) standard is the emerging successor to the widespread High Efficiency Video Coding (HEVC/H.265). This work introduces the latest version of our academic open-source VVC intra encoder called uvg266. It has been developed from our well-known Kvazaar HEVC encoder by introducing new VVC coding tools into carefully optimized and parallelized coding flow of Kvazaar. This paper outlines the design methodology and implementation aspects of all intra (AI) configuration of uvg266. The experimental results show that single-threaded uvg266 is more than twice as fast as the state-of-the-art VVenC encoder in all our test cases. In speed-optimized coding, the coding overhead of uvg266 is 21.7% but the gap narrows down to 2.6% in the rate distortion optimized coding case. uvg266 has almost linear speedup with core count up to 32 threads. The better scalability of uvg266 quadruples the speed over VVenC. Furthermore, single-threaded uvg266 is up to 380× as fast as VVC reference software VTM and the gap raises to over 11,000× with 32 threads. To the best of our knowledge, uvg266 is currently the fastest available open-source VVC intra software encoder.
We present NNVISR - an open-source filter plugin for the VapourSynth video processing framework, which facilitates the application of neural networks for various kinds of video enhancing tasks, including denoising, super resolution, interpolation, and spatio-temporal super-resolution. NNVISR fills the gap between video enhancement neural networks and video processing pipelines, by accepting any network that enhances a group of frames, and handling all other network agnostic details during video processing. NNVISR is publicly released at https://github.com/tongyuantongyu/vs-NNVISR.
3D reconstruction remains a pivotal topic in contemporary multimedia research, with multi-view stereo (MVS) methods being instrumental. Traditional MVS algorithms, have demonstrated robust performance in large-scale scenes. However, these methods often struggle with depth estimation in non-textured regions, resulting in inconsistencies across different areas of the reconstructed 3D model. This paper introduces a novel MVS framework, General Sampling Non-local Texture Features (GST-MVS), designed to enhance depth estimation by leveraging the intrinsic relationship between image texture and object depth. Our approach optimizes the 3D reconstruction process, yielding superior results, especially in low-textured regions. Comprehensive evaluations on the ETH3D dataset indicate that GST-MVS achieves dense and accurate 3D point cloud reconstructions, outperforming existing MVS algorithms. The GST-MVS framework is open-source and accessible on GitHub at https://github.com/Jasmine-tjy/GST-mvs. Additionally, Docker image facilitating the required execution environment, which is available at https://hub.docker.com/r/tangjas111/cmake_cuda_opencv/tags.
Although automatic shot transition detection approaches are already investigated for more than two decades, an effective universal human-level model was not proposed yet. Even for common shot transitions like hard cuts or simple gradual changes, the potential diversity of analyzed video contents may still lead to both false hits and false dismissals. Recently, deep learning-based approaches significantly improved the accuracy of shot transition detection using 3D convolutional architectures and artificially created training data. Nevertheless, one hundred percent accuracy is still an unreachable ideal. In this paper, we share the current version of our deep network TransNet V2 that reaches state-of-the-art performance on respected benchmarks. A trained instance of the model is provided so it can be instantly utilized by the community for a highly efficient analysis of large video archives. Furthermore, the network architecture, as well as our experience with the training process, are detailed, including simple code snippets for convenient usage of the proposed model and visualization of results.
Effective and smooth video communication has become an integral tool for interaction in our modern society. The well-known WebRTC project is optimized for server-based communication architectures with less stringent latency requirements. This paper introduces an open-source, peer-to-peer (P2P) mesh-based application called uvgComm 1.0 for low-latency multi-party video communication. Our experimental results show that uvgComm attains 25% lower latency than server-based solutions at 219 ms. In addition, it offers advanced privacy protection mechanisms with end-to-end encryption and P2P mesh topology that delivers media without any media servers. The ambitious goal of uvgComm is to take P2P mesh-based conferencing architecture to the next level and make this architecture a serious competitor to server-based communication architectures.
Social Virtual Reality is envisioned to transform how individuals communicate remotely, offering a sense of immersion and co-presence within a virtual space. Current platforms enabling remote social interactions rely on synthetic user representations. We address this limitation by enabling realistic human representation through volumetric content capture, encoding and transmission. Specifically, we present an extended version of VR2Gather, now a fully open source Unity package, available at https://github.com/cwi-dis/VR2Gather-acmmm-oss. Our platform is a customisable system to transmit volumetric content in a multi-party real-time environment, easy to integrate into existing applications.
The rising prominence of Virtual Reality (VR), Mixed Reality (MR), and Extended Reality (XR) devices is transforming various industries by offering immersive and interactive experiences. Despite this, there is a notable gap in research and development of frameworks that seamlessly integrate real-world scenes into virtual environments for collaborative use. Current methods experience extended reconstruction times and latency issues, rendering them impractical for real-time collaborative applications. This paper introduces Room2XR, a framework designed to dynamically reconstruct real-world scenes into virtual semantically similar representations using CAD models, and share them for remote collaboration. Room2XR employs advanced scene understanding technologies to create detailed virtual environments, enabling multiple users to interact with and manipulate these spaces in real time. By leveraging efficient algorithms and data transmission methods, Room2XR ensures accessibility and usability even on low-bandwidth networks. The source code and detailed documentation are provided in the following two repositories- 1. https://github.com/HenryGuo2003/Room2XR-Unity and 2. https://hub.docker.com/r/kumarhiranya/vrrec
OpenDIC: An Open-Source Library and Performance Evaluation for Deep-learning-based Image Compression
Deep learning technologies have been popular in the image compression field for some time. An increasing number of deep-learning-based models are proposed to improve Rate-Distortion (RD) performance. Previous algorithms are implemented in the specific platform and can not be applied in cross-platform environments. In this paper, we present an open-source algorithm library called OpenDIC, which integrates a variety of end-to-end image compression methods in cross-platform environments. The contribution and details of the algorithms used in the library are described. To evaluate the performance of these algorithms, we conduct a comprehensive performance test. We compare and analyze each algorithm according to RD performance, running time, and GPU memory occupancy. The algorithm library has been released at https://openi.pcl.ac.cn/OpenDIC/.
We present VLMEvalKit: an open-source toolkit for evaluating large multi-modality models based on PyTorch. The toolkit aims to provide a user-friendly and comprehensive framework for researchers and developers to evaluate existing multi-modality models and publish reproducible evaluation results. In VLMEvalKit, we implement over 70 different large multi-modality models, including both proprietary APIs and open-source models, as well as more than 20 different multi-modal benchmarks. By implementing a single interface, new models can be easily added to the toolkit, while the toolkit automatically handles the remaining workloads, including data preparation, distributed inference, prediction post-processing, and metric calculation. Although the toolkit is currently mainly used for evaluating large vision-language models, its design is compatible with future updates that incorporate additional modalities, such as audio and video. Based on the evaluation results obtained with the toolkit, we host a comprehensive leaderboard (https://huggingface.co/spaces/opencompass/open_vlm_leaderboard) to track the progress of multi-modality learning research. The toolkit is released on GitHub (https://github.com/open-compass/VLMEvalKit) and is actively maintained.
Text-driven human motion generation, which creates motion sequences based on textual descriptions, has attracted great attention in the communities of multimedia and artificial intelligence. By parsing and comprehending textual information and converting it into specific human movements, it realizes a direct transformation from human semantics to motion sequences. New text-driven human motion generators are springing up to achieve better performance. However, the absence of well-trained evaluators that can effectively estimate the consistency between the text prompts and motions generated by existing generators remains a challenge. To address the above issues, we propose an open-source library with a powerful Contrastive Language-and-Motion (CLaM) pre-training evaluator, which can be employed for evaluating a variety of text-driven human motion generation algorithms. We perform a thorough performance evaluation of the existing algorithms on various metrics, such as R-Precision. As a by-product, we build a large-scale HumanML3D-synthesis dataset, which consists of 14,616 motion sequences and 547,102 textual descriptions, which is ten times larger than the widely-used HumanML3D dataset. The source codes and models for CLaM are available at~https://github.com/SheldongChen/CLaM/.
We present Sophia-in-Audition (SiA), a new frontier in virtual production, by employing the humanoid robot Sophia within an UltraStage environment composed of a controllable lighting dome coupled with multiple cameras. We demonstrate Sophia's capability to replicate iconic film segments, follow real performers, and perform a variety of motions and expressions, showcasing her versatility as a virtual actor. Key to this process is the integration of facial motion transfer algorithms and the UltraStage's controllable lighting and multi-camera setup, enabling dynamic performances that align with the director's vision. Our comprehensive user studies indicate positive audience reception towards Sophia's performances, highlighting her potential to reduce the uncanny valley effect in virtual acting. Additionally, the immersive lighting in dynamic clips was highly rated for its naturalness and its ability to mirror professional film standards. The paper presents a first-of-its-kind multi-view robot performance video dataset with dynamic lighting, offering valuable insights for future enhancements in humanoid robotic performers and virtual production techniques. This research contributes significantly to the field by presenting a unique virtual production setup, developing tools for sophisticated performance control, and providing a comprehensive dataset and user study analysis for diverse applications.
Large Vision-Language Models (LVLMs) have demonstrated their powerful multimodal capabilities. However, they also face serious safety problems, as adversaries can induce robustness issues in LVLMs through the use of well-designed adversarial examples. Therefore, LVLMs are in urgent need of detection tools for adversarial examples to prevent incorrect responses. In this work, we first discover that LVLMs exhibit regular attention patterns for clean images when presented with probe questions. We propose an unconventional method named PIP, which utilizes the attention patterns of one randomly selected irrelevant probe question (e.g., "Is there a clock''') to distinguish adversarial examples from clean examples. Regardless of the image to be tested and its corresponding question, PIP only needs to perform one additional inference of the image to be tested and the probe question, and then achieves successful detection of adversarial examples. Even under black-box attacks and open dataset scenarios, our PIP, coupled with a simple SVM, still achieves more than 98% recall and a precision of over 90%. Our PIP is the first attempt to detect adversarial attacks on LVLMs via simple irrelevant probe questions, shedding light on deeper understanding and introspection within LVLMs. The code is available at https://github.com/btzyd/pip.
Understanding the correct input domain for black-box models is vital for tasks such as model cloning, inversion, and membership inference. However, this area remains underexplored, hindering related methods' efficacy without domain information. In this paper, we highlight the need for discovering the data domain and propose an approach that leverages existing generative models to address this challenge. With hard-label black-box access to a neural network model, our method produces a set of embeddings that, when utilized with the generative model, yield samples closely aligned with each target class's data domain, facilitating downstream tasks. Central to our method is an objective function covering both functional relevance and embedding generality. We employ an iterative search algorithm to identify the optimal set of embeddings. Starting with initial embeddings, new data points are generated and classified by the target model. Successful classifications guide embedding resampling, refining subsequent iterations' generated images closer to the target class's data domain. Consequently, the embeddings are iteratively modified to better match the data domain of the target class. Given the vast embedding space, we introduce an optional preprocessing phase. This phase leverages a comprehensive corpus like ImageNet to select a representative subset of samples, roughly aligned with the model's input domain, to serve as starting points.
This paper presents a novel model protection paradigm Model Locking that locks the performance of a finetuned model on private data to make it unusable or unextractable without the right key. Specifically, we proposed a diffusion-based framework dubbed ModelLock that explores text-guided image editing to transform the private finetuning data into unique styles or blend new objects into the background. A model finetuned on this edited dataset will be locked and can only be unlocked by the key prompt, i.e., the same text prompt used to edit the data. We conduct extensive experiments on both image classification and segmentation tasks and show that 1) ModelLock can effectively lock finetuned models without significantly reducing their unlocked performance, and more importantly, 2) the locked model cannot be easily unlocked without knowing both the key prompt and the diffusion model. Our work opens up a new direction for intellectual property protection of private models.
Visual similarity estimation plays a fundamental role both in human cognition and multimedia information processing as it is the basis for many applications ranging from image search and recommendation to visual content generation. Existing computational models to assess visual similarity often diverge from human perception as they are typically trained solely on image data without information about how humans perceive image similarity. Here, we present an approach for learning perceptual visual similarity from brain recordings obtained via electroencephalogram (EEG). Our approach establishes a mapping between similarity reflected in the human cognitive system and a latent image representation. We evaluate the approach in two tasks. First, predicting visual distance from EEG data and second, adjusting a latent representation of a generative model to generate new images at a predicted distance from a given source image. Experiments demonstrate that the predicted distances from EEG closely align with the ground truth distances, and images generated using these predicted distances closely resemble the ground truth images. These findings open new possibilities for leveraging signals measured from human cognition to infer similarity as opposed to using only content-based models.
We introduce OpenLEAF, a new benchmark designed for the emerging open-domain interleaved image-text generation task. This task aims to generate arbitrarily interleaved multimodal content from input queries. It goes beyond commonly seen single-modality image or text generation, thereby enabling various novel applications by creating content such as visual storybooks and how-to instructions. Despite the importance of this task, no established benchmark exists due to the challenges in defining evaluation scenarios and formulating effective metrics. To introduce and facilitate the new task of interleaved image-text generation, we create a new dataset covering queries with various input-output formats and 10 different application scenarios. We also propose a novel evaluation pipeline named "detection-summarization-scoring," which breaks down the evaluation into multiple reasoning steps. This pipeline leverages large multimodal models (LMMs) to thoroughly evaluate ten aspects of the generated content, which leads to the final rating. With experiments on a proposed agent system, we demonstrate that our evaluation method aligns closely with human judgments, thus together with the dataset, offering the research community a valuable benchmark for exploring interleaved image-text generation.
Quantization index modulation (QIM) based VoIP steganography can conceal secret information in VoIP streams. Malicious users could use this technology to conduct illegal activities, threatening network and public security. Hence, practical steganalysis models that could detect QIM-based VoIP steganography are urged to be developed. In recent years, deep learning (DL) models have been investigated for this task, and exciting outcomes have been achieved. However, existing models are far from practical. Two major challenges are required to be addressed. First, there is still significant room for improvement in detection accuracy. Second, studies that balance the detection accuracy and response time are still insufficient. In this context, our main research topic fits in the QIM-based VoIP steganalysis theme, which aims to detect QIM-based steganography in VoIP streams in a fast and accurate manner.
Maps are the fundamental elements of any navigation and localization system. With the fast expansion of urban areas and the increasing complexity of modern cities, traditional mapping techniques cannot meet the need for frequent map updates with enriched map details. This work investigates how geo-referenced very high-resolution (VHR) RGB satellite imagery can be used to extract geo-information to support the creation of fine-scale maps and facilitate map updates, focusing on two sub-directions. First, to alleviate the data scarcity issue of large-scale geo-information extraction from satellite imagery-based datasets, (1) GAN-assisted road segmentation proposes a new assisted training scheme to improve the model performance when the training dataset is limited and (2) a context-enhanced satellite-imagery dataset is created for large-scale parking lot detection to improve the type diversity of target objects. Second, to support rich map attribute geo-information extraction and vision-based navigation using street-view imagery, new methods are proposed to improve location and orientation extraction of the street-view imagery via cross-view matching with satellite imagery.