With the continuous progress of autonomous vehicle (AV) technologies, the resulting accidents have generated public concern, underscoring the need for comprehensive, reliable, and authoritative testing to enhance safety. X-in-the-loop (XiL) testing has emerged as a promising paradigm to bridge the gap between simulation and real-world deployment. This study addresses the gap in supporting solid-state LiDAR sensors, thereby bolstering the authority and credibility of XiL testing. Based on the operating principles of actual LiDAR, the proposed high-fidelity model simulates unique mechanisms of solid-state LiDAR to produce point clouds that closely correspond to real-world data. Additionally, the model accounts for the impact of various weather conditions on laser propagation, enhancing the coverage and reliability of testing scenarios. Extensive experiments have been conducted to validate the proposed LiDAR model from multiple levels, including scanning pattern, point cloud acquisition, and XiL testing, with real world data as the benchmark. Results confirm that the model replicates real LiDAR behavior and supports AV functions (mapping, localization, perception), offering a novel sensor model extension for controlled and repeatable XiL testing.
论文检索
输入标题、作者或关键词，从 1,620 篇学术成果中精准定位
Recent advances in 3D Gaussian Splatting (3DGS) have enabled fast, photorealistic rendering of dynamic 3D scenes, showing strong potential in immersive communication. However, in digital human encoding and transmission, the compression methods based on general 3DGS representations are limited by the lack of human priors, resulting in suboptimal bitrate efficiency and reconstruction quality at the decoder side, which hinders their application in streamable 3D avatar systems. We propose HGC-Avatar, a novel Hierarchical Gaussian Compression framework designed for efficient transmission and high-quality rendering of dynamic avatars. Our method disentangles the Gaussian representation into a structural layer, which maps poses to Gaussians via a StyleUNet-based generator, and a motion layer, which leverages the SMPL-X model to represent temporal pose variations compactly and semantically. This hierarchical design supports layer-wise compression, progressive decoding, and controllable rendering from diverse pose inputs such as video sequences or text. Since people are most concerned with facial realism, we incorporate a facial attention mechanism during StyleUNet training to preserve identity and expression details under low-bitrate constraints. Experimental results demonstrate that HGC-Avatar provides a streamable solution for rapid 3D avatar rendering, while significantly outperforming prior methods in both visual quality and compression efficiency.
Evolving multimedia systems are increasingly being adopted in virtual reality and gaming applications. Such systems emphasize immersion to engage users by bridging the gap between real and virtual content. In this context, visual and acoustic stimuli are the two key media that dictate such immersion. While visual 3D rendering is advancing rapidly, the same is not true for audio, where most research is limited to the reconstruction of the room impulse response (RIR) using omnidirectional audio or, at best, binaural. Such methods do not adequately account for the directions and orientations of the acoustic signals with respect to either the source or the listener, thereby compromising immersion quality. In this work, we explore the effect of adding such ''directionality'' to the training data to improve the estimation of the room's acoustic parameters. A more accurate set of such parameters implies in fact a more realistic predicted RIR, leading to a more immersive experience of the acoustic scene. Specifically, we propose a novel framework driven by a suitable loss function to account for directionality in ambisonic microphones, and novel variants of loss functions for both omnidirectional and ambisonic cases. We also propose to account for microphone characteristics and their contribution to the predicted RIRs. Experiments were performed using two datasets of real recordings and the results established the efficacy of the proposed methods.
The integration of prompt tuning with multimodal learning has shown significant generalization abilities for various downstream tasks. Despite advancements, existing methods heavily depend on massive modality-specific labeled data (e.g., video, audio, and image), or are customized for a single modality. In this study, we present Text as Any-Modality by Consistent Prompt Tuning (TaAM-CPT), a scalable approach for constructing a general representation model toward unlimited modalities using solely text data. TaAM-CPT comprises modality prompt pools, text construction, and modality-aligned text encoders from pre-trained models, which allows for extending new modalities by simply adding prompt pools and modality-aligned text encoders. To harmonize the learning across different modalities, TaAM-CPT designs intra- and inter-modal learning objectives, which can capture category details within modalities while maintaining semantic consistency across different modalities. Benefiting from its scalable architecture and pre-trained models, TaAM-CPT can be seamlessly extended to accommodate unlimited modalities. Remarkably, without any modality-specific labeled data, TaAM-CPT achieves leading results on diverse datasets spanning various modalities, including video classification, image classification, and audio classification. The code is available at https://github.com/Jinx630/TaAM-CPT.
The electrocardiogram (ECG) is an essential and effective tool for diagnosing heart diseases. However, its effectiveness can be compromised by noise or unavailability of one or more leads of the standard 12-lead recordings, resulting in diagnostic errors or uncertainty. To address these challenges, we propose TolerantECG, a foundation model for ECG signals that is robust to noise and capable of functioning with arbitrary subsets of the standard 12-lead ECG. TolerantECG training combines contrastive and self-supervised learning frameworks to jointly learn ECG signal representations alongside their corresponding knowledge-retrieval-based text report descriptions and corrupted or lead-missing signals. Comprehensive benchmarking results demonstrate that TolerantECG consistently ranks as the best or second-best performer across various ECG signal conditions and class levels in the PTB-XL dataset, and achieves the highest performance on the MIT-BIH Arrhythmia Database. The source is available at this link: https://github.com/Fsoft-AIC/TolerantECG
The continual learning (CL) of novel concepts from new environments represents a popular and important topic aiming to manage catastrophic forgetting. Research studies have developed dynamic expansion models to deal with network forgetting in CL. Existing CL models usually explore the full capacity of activating parameters and representations while ignoring the previously learned representations when learning new tasks. In this paper, we propose a novel dynamic expansion model that incrementally accumulates and incorporates all previously learned representations into defining new experts to add to a mixture of experts in a recursive manner, aiming to reuse previously learned parameters and features to promote future task learning. We define a graph structure having each expert as a component node. We then propose a novel expandable expert graph attention mechanism that dynamically optimizes the graph when learning new tasks, maximizing the positive knowledge transfer. In addition, we propose a novel expert cooperation mechanism to promote the cooperation between all previous experts and with the currently updated expert. Furthermore, we propose a novel memory optimization approach, which encourages each expert to capture and learn completely different information, further improving performance. We provide the results of a series of experiments demonstrating that the proposed approach outperforms the state-of-the-art performance in CL.
Deep learning-based tumor segmentation methods typically require precise pixel-level annotations, which are costly in clinical practice. While bounding box supervision offers a more efficient alternative, existing approaches assume unrealistically tight box annotations, leading to performance degradation when applied to the loose boxes commonly produced by medical annotators. To address this challenge, we propose LooBox, a novel 3D segmentation framework that utilizes loose box annotations through a self-correction and bidirectional rectification paradigm. For the self-correction part, we propose a noise cleaner that comprehensively utilizes deterministic outer box information by integrating three complementary perspectives for predictive self-rectification: entropy mapping, gradient monitoring, and foreground-background affinity measurement. For the bidirectional rectification part, we introduce an augmentation-driven comprehensive consistency constraint strategy. Specifically, the framework incorporates: an asymmetric co-teaching architecture comprising a basic UNet and an enhanced UNet variant with a noise adapter, and an augmentation-driven consistency mechanism that computes pairwise loss between self-corrected predictions after each training iteration to ensure robust tumor feature extraction. Comprehensive evaluations on LIDC-IDRI, MSD-Lung, and MSD-Pancreas datasets demonstrate that LooBox achieves superior segmentation accuracy compared to state-of-the-art box-supervised methods.
Contextual reasoning with constraints is crucial for enhancing temporal consistency in cross-frame modeling for visual tracking. However, mainstream tracking algorithms typically associate context by merely stacking historical information without explicitly supervising the association process, making it difficult to effectively model the target's evolving dynamics. To alleviate this problem, we propose RSTrack, which explicitly models and supervises context reasoning via three core mechanisms. 1) Context Reasoning Mechanism : Constructs a target state reasoning pipeline, converting unconstrained contextual associations into a temporal reasoning process that predicts the current representation based on historical target states, thereby enhancing temporal consistency. 2) Forward Supervision Strategy : Utilizes true target features as anchors to constrain the reasoning pipeline, guiding the predicted output toward the true target distribution and suppressing drift in the context reasoning process. 3) Efficient State Modeling : Employs a compression-reconstruction mechanism to extract the core features of the target, removing redundant information across frames and preventing ineffective contextual associations. These three mechanisms collaborate to effectively alleviate the issue of contextual association divergence in traditional temporal modeling. Experimental results show that RSTrack achieves state-of-the-art performance on multiple benchmark datasets while maintaining real-time running speeds. Our code is available at https://github.com/GXNU-ZhongLab/RSTrack.
The detection of hand contact states, which involves identifying interactions between hands and objects or other entities, is essential for the development of human-computer interaction systems and the comprehension of social dynamics. Previous approaches have made progress in modeling hand-object interactions. Nonetheless, they neglect critical cues between their hands and bodies, as well as those of others, thus constraining their ability to accurately detect interpersonal contact. The task remains challenging due to frequent occlusions, especially in crowded multi-person scenarios with complex contexts. In this paper, a novel hand-object-person interaction network, called HOPNet, is proposed to model contextual information between hands and objects, as well as between hands and bodies. Specifically, HOPNet consists of two components: (i) the Hand-Object Relation (HOR) module analyzes interaction patterns between hands and objects, capturing spatial and semantic relationships; (ii) the Contrastive Spatial Refinement (CSR) module learns hand-body interactions through contrastive geometric embedding and relative spatial enhancement, improving interpersonal contact recognition in crowded scenarios. Experiments on ContactHands and 100DOH datasets demonstrate that HOPNet outperforms state-of-the-art methods.
In industrial scenarios, diverse anomalous images are difficult to acquire, significantly limiting the performance of industrial anomaly detection methods. Automatically generating anomalous images for anomaly detection has the potential to solve the above problem. However, existing anomaly generation models are still not satisfactory regarding the authenticity and controllability of anomaly generation. In this paper, we propose a controlled anomaly generation model named AnomalyControl to generate realistic anomalous images aligned highly with both text prompts and anomaly masks. First, we introduce a CLIP-guided anomaly prompt generator that leverages a CLIP text encoder to find anomaly text prompts most aligned with real anomalous images. Secondly, we propose an anomaly appearance and shape decoupling mechanism, which designs an embedding similarity loss to enforce the alignment between the anomaly text prompt and anomalies generated with different shapes at the same location, making the appearance of generated anomalies better maintain semantic consistency when the anomaly shape changes. Then, a training-free local control enhancement strategy is employed to provide stronger control intensity to anomaly regions during inference for finer alignment with anomaly masks. Finally, a hard sample generation module is proposed to create anomalous samples with subtle shapes and imperceptible anomaly appearances, enabling the downstream anomaly detection model to focus on learning low-saliency anomaly features. Extensive experiments demonstrate that anomalous images generated by our model outperform the state-of-the-art anomaly generation methods in terms of authenticity and consistency, and can significantly improve the performance of downstream anomaly detection tasks, especially anomaly localization.
Perceptual hashing has garnered significant attention for its wide-ranging applications in image retrieval and authentication domains. However, existing algorithms often struggle to detect subtle manipulations confined to small regions of an image. In this paper, we introduce a novel framework, Manipulation-Aware Deep Perceptual Hashing (MADPHash), which leverages feature consistency to enhance sensitivity to such subtle manipulations. MADPHash explicitly treats tampered images as a distinct category, incorporates a tampering detection objective into the perceptual hash generation process, and employs a Consistency Constraint Module to amplify discrepancies between tampered and untampered regions. Comprehensive experiments conducted on five benchmark datasets demonstrate that MADPHash significantly improves the detection of subtle manipulations while maintaining robustness against content-preserving transformations, outperforming several state-of-the-art perceptual hashing methods.
Lens flare removal remains an information confusion challenge in the underlying image background and the optical flares, due to the complex optical interactions between light sources and camera lens. While recent solutions have shown promise in decoupling the flare corruption from image, they often fail to maintain contextual consistency, leading to incomplete and inconsistent flare removal. To eliminate this limitation, we propose DeflareMamba, which leverages the efficient sequence modeling capabilities of state space models while maintains the ability to capture local-global dependencies. Particularly, we design a hierarchical framework that establishes long-range pixel correlations through varied stride sampling patterns, and utilize local-enhanced state space models that simultaneously preserves local details. To the best of our knowledge, this is the first work that introduces state space models to the flare removal task. Extensive experiments demonstrate that our method effectively removes various types of flare artifacts, including scattering and reflective flares, while maintaining the natural appearance of non-flare regions. Further downstream applications demonstrate the capacity of our method to improve visual object recognition and cross-modal semantic understanding. Code is available at https://github.com/BNU-ERC-ITEA/DeflareMamba.
Accurate prediction of placental diseases via whole slide images (WSIs) is critical for preventing severe maternal and fetal complications. However, WSI analysis presents significant computational challenges due to the massive data volume. Existing WSI classification methods encounter critical limitations: (1) inadequate patch selection strategies that either compromise performance or fail to sufficiently reduce computational demands, and (2) the loss of global histological context resulting from patch-level processing approaches. To address these challenges, we propose an Efficient multimodal framework for Patient-level placental disease Diagnosis, named EmmPD. Our approach introduces a two-stage patch selection module that combines parameter-free and learnable compression strategies, optimally balancing computational efficiency with critical feature preservation. Additionally, we develop a hybrid multimodal fusion module that leverages adaptive graph learning to enhance pathological feature representation and incorporates textual medical reports to enrich global contextual understanding. Extensive experiments conducted on both a self-constructed patient-level Placental dataset and two public datasets demonstrating that our method achieves state-of-the-art diagnostic performance. The code is available at https://github.com/ECNU-MultiDimLab/EmmPD.
Sketching is a quick ideation and multimedia tool for effectively expressing design intent. By translating simple strokes into CAD models, it allows non-expert users to create editable designs, reducing the learning curve associated with traditional CAD software. However, current sketch-based CAD modeling methods are often limited to basic shapes and require structured inputs, making them less robust when dealing with varied sketch styles. To overcome these challenges, we propose a novel sketch-based modeling framework DAFU-CAD, that is both efficient and robust. Our approach features a Depth-Assisted and Feature-Unraveling sketch classification module that categorizes sketches into corresponding modeling operations, independent of their drawing style. A parameter regression and optimization module then estimates the modeling parameters, ensuring consistent and stable model reconstruction across different sketch inputs. To support this, we compile a diverse sketch dataset with a range of modeling categories and abstraction levels. Experimental results show that our method outperforms existing approaches in terms of both robustness and versatility.
Short-video platforms have become a central part of digital content, with users rapidly engaging in various trending topics. Predicting the peak popularity of short-video topics is critical for understanding content dynamics and user behavior. This paper introduces the task of Short-Video Topic Peak Prediction (SVTPP) and proposes both a new dataset and an innovative method. We present the TopicVid dataset, designed to capture the peak trends of short-video topics across multiple platforms. The TopicVid dataset includes 7,701 topics, 58,539 users, and 96,936 videos, with data on views, comments, and shares. This dataset is the first to provide rich semantic information for short-video topic peak prediction, including content details, titles, and user interactions. We propose the Topic Large Graph Model (TLGM), a two-stage framework that integrates heterogeneous graph data with large language models. The TLGM model effectively analyzes the relationships within short-video topics to predict their peak popularity. Experimental results show that our method outperforms existing approaches for predicting short-video topic peaks. Our dataset and code is available at https://github.com/chensh911/TLGM.
Understanding 3D affordance is essential for agents to effectively interact with real-world environments, encompassing tasks such as manipulation and navigation. Existing methods typically support open-vocabulary queries through label-based language descriptions but often suffer from limited generalization and weak discriminative ability in their representations. However, affordance understanding requires constructing a coherent semantic landscape from fragmented linguistic expressions-one that preserves intra-class diversity while minimizing inter-class overlap. To address these challenges, we introduce Aff3DFunc, a framework designed to enhance the alignment between affordance and 3D geometry. It begins with a functional text enhancement module grounded in the Information Bottleneck (IB) principle, which strategically enriches affordance semantics by maximizing both relevance and diversity. A dual-encoder architecture is then employed to extract embeddings from both point clouds and text. To bridge the modality gap, we further propose a multilevel representation alignment strategy that incorporates supervised contrastive learning, reinforcing semantic-geometric correspondence in a part-to-whole manner. Extensive experiments demonstrate that our approach significantly enhances the understanding of affordance complexity. The learned representations exhibit high adaptability to diverse text queries, particularly in zero-shot settings. Furthermore, the real-world robot validation confirms that our method improves affordance understanding, enabling more fine-grained manipulation tasks.
Mental stress assessment is crucial for mental and physical well-being. However, it faces limitations due to domain fragmentation, in which contextual variations in stress triggers and demographics hinder the generalization of assessment models across real-world scenarios. Additionally, mental stress assessment is sensitive and human-centric due to its implications for mental health interventions, emphasizing the need for model transparency and trustworthiness. To address this gap, we propose Retrieval-Augmented Reasoning, a novel framework that bridges domain gaps in mental stress assessment through transparent step-by-step reasoning and dynamic in-context example retrieval. Our framework introduces two key components: (1) a ''detect-then-assess'' reasoning chain decouples stress-relevant facial action units (AUs) from domain-specific noise by first generating textual descriptions as intermediate reasoning step (e.g., ''eyebrow: inner portions raised''). The model then reflects on and learns to refine these descriptions via Direct Preference Optimization (DPO), ensuring faithfulness and helpfulness; (2) a dual-encoder multimodal retriever dynamically selects proper in-context examples from source domain to enhance target-domain assessments, leveraging feedback from the assessment model to optimize retrieval. Experimental results demonstrate that our framework consistently outperforms large multimodal foundation models, stress assessment baselines, and domain generalization methods.
Volumetric video enables immersive experiences by capturing dynamic 3D scenes, enabling diverse applications for virtual reality, education, and telepresence. However, traditional methods struggle with fixed lighting conditions, while neural approaches face trade-offs in efficiency, quality, or adaptability for relightable scenarios. To address these limitations, we present BEAM, a novel pipeline that bridges 4D Gaussian representations with physically-based rendering (PBR) to produce high-quality, relightable volumetric videos from multi-view RGB footage. BEAM recovers detailed geometry and PBR properties via a series of available Gaussian-based techniques. It first combines Gaussian-based human performance tracking with geometry-aware rasterization in a coarse-to-fine optimization framework to recover spatially and temporally consistent geometries. We further enhance Gaussian attributes by incorporating PBR properties step by step. We generate roughness via a multi-view-conditioned diffusion model, and then derive AO and base color using a 2D-to-3D strategy, incorporating a tailored Gaussian-based ray tracer for efficient visibility computation. Once recovered, these dynamic, relightable assets integrate seamlessly into traditional CG pipelines, supporting real-time rendering with deferred shading and offline rendering with ray tracing. By offering realistic, lifelike visualizations under diverse lighting conditions, BEAM opens new possibilities for interactive entertainment, storytelling, and creative visualization.
Recently, Learned Image Compression (LIC) models have garnered significant attention due to their superior performance in comparison to traditional image codecs. However, the growing complexity of these deep learning-based models results in high memory consumption and computational load, which limits their practical deployment. Quantization has emerged as a promising technique to reduce both the storage requirements and computational overhead. Despite its success in high-level vision tasks like image recognition and object detection, quantization techniques applied to LIC models remain underexplored. In this work, we identify the unique challenges of quantizing LIC models, specifically focusing on the impact of latent distribution ranges in high-bitrate. We observe that the activation layers of high-bitrate models exhibit a wider distribution range, which causes significant performance degradation after quantization. Furthermore, we explore the limitations of existing LIC quantization schemes, such as per-channel quantization for activation layers, which result in poor hardware acceleration performance and increased data storage overhead. To address these challenges, we propose the activation and weight distribution balancing post-training quantization (AWDB-PTQ) method for LIC models, which uses a coarse-to-fine strategy to optimize balancing coefficients. In addition, we employ per-tensor activation quantization and symmetric uniform quantization to better facilitate hardware acceleration. Experimental results demonstrate that our proposed method outperforms existing methods in terms of both compression performance and computational efficiency. Our code and data are available at: https://github.com/jie-yu16/AWDB-PTQ.
Video signals are vulnerable in multimedia communication and storage systems, as even slight bitstream-domain corruption can lead to significant pixel-domain degradation. To recover faithful spatio-temporal content from corrupted inputs, bitstream-corrupted video recovery has recently emerged as a challenging and understudied task. However, existing methods require time-consuming and labor-intensive annotation of corrupted regions for each corrupted video frame, resulting in a large workload in practice. In addition, high-quality recovery remains difficult as part of the local residual information in corrupted frames may mislead feature completion and successive content recovery. In this paper, we propose the first blind bitstream-corrupted video recovery framework that integrates visual foundation models with recovery model, which is adapted to different types of corruption and bitstream-level prompts. Within the framework, the proposed Detect Any Corruption (DAC) model leverages the rich priors of the visual foundation model while incorporating bitstream and corruption knowledge to enhance corruption localization and blind recovery. Additionally, we introduce a novel Corruption-aware Feature Completion (CFC) module, which adaptively processes residual contributions based on high-level corruption understanding. With VFM-guided hierarchical feature augmentation and high-level coordination in a mixture-of-residual-experts (MoRE) structure, our method suppresses artifacts and enhances informative residuals. Comprehensive evaluations show that the proposed method achieves outstanding performance in bitstream-corrupted video recovery without requiring a manually labeled mask sequence. The demonstrated effectiveness will help to realize improved user experience, wider application scenarios, and more reliable multimedia communication and storage systems.