Signed Graph Neural Networks (SGNNs) are widely adopted to analyze complex patterns in signed graphs with both positive and negative links. Given the noisy nature of real-world connections, the robustness of SGNN has also emerged as a pivotal research area. Under the supervision of empirical properties, graph structure learning has shown its robustness on signed graph representation learning, however, there remains a paucity of research investigating a robust SGNN with theoretical guidance. Inspired by the success of graph information bottleneck (GIB) in information extraction, we propose RIDGE, a novel framework for Robust sIgned graph learning through joint Denoising of Graph inputs and supervision targEts. Different from the basic GIB, we extend the GIB theory with the capability of target space denoising as the co-existence of noise in both input and target spaces. In instantiation, RIDGE effectively cleanses input data and supervision targets via a tractable objective function produced by reparameterization mechanism and variational approximation. We extensively validate our method on four prevalent signed graph datasets, and the results show that RIDGE clearly improves the robustness of popular SGNN models under various levels of noise.
论文检索
输入标题、作者或关键词，从 1,620 篇学术成果中精准定位
EgoHierMask: Hierarchical Semantic-Prior Guided Masked Autoencoder for Egocentric Action Recognition
Egocentric action recognition holds critical value in augmented reality, embodied AI, and human behavior analysis. While transformer-based masked autoencoders show potential in general video representation learning, their direct application to egocentric vision faces fundamental limitations --- random masking strategies disrupt crucial spatiotemporal features like hand-object interaction hierarchies and viewpoint dynamics by neglecting task-specific semantic priors. Through systematic analysis, this paper reveals three complementary semantic priors for egocentric video understanding: verb-centric motion patterns characterizing hand trajectories, noun-aware attention regions highlighting object contact points, and action-oriented global context integrating holistic semantics. These hierarchical cues address egocentric visual specificity through motion granularity, interaction locality, and semantic integrity. Building on this discovery, we propose EgoHierMask: a hierarchical semantic prior-guided masked autoencoder framework coordinating vision-language knowledge through differentiated masking strategies. The framework employs frozen vision-language teacher models to generate multi-level semantic attention maps, systematically guiding three specialized masking branches: a) dynamic motion masking preserves hand movement continuity through temporal verb attention, b) interaction-sensitive masking maintains object manipulation coherence via spatial noun saliency, and c) spatiotemporal joint masking encodes complete action semantics through global context alignment. Additionally, to enhance learning efficacy, we curate a distribution-balanced pretraining corpus and devise a unified architecture with dual-granularity supervision, combining pixel-level reconstruction with semantic-level distillation within the VideoMAE paradigm.Extensive experiments demonstrate state-of-the-art performance across major benchmarks, validating the crucial value of hierarchical prior injection for egocentric representation learning.
Temporal Sentence Grounding (TSG) aims to identify relevant moments in an untrimmed video that semantically correspond to a given textual query. Despite existing studies having made substantial progress, they often overlook the issue of spurious correlations between video and textual queries. These spurious correlations arise from two primary factors: (1) inherent biases in the textual data, such as frequent co-occurrences of specific verbs or phrases, and (2) the model's tendency to overfit to salient or repetitive patterns in video content. Such biases mislead the model into associating textual cues with incorrect visual moments, resulting in unreliable predictions and poor generalization to out-of-distribution examples. To overcome these limitations, we propose a novel TSG framework, causal intervention and counterfactual reasoning that utilizes causal inference to eliminate spurious correlations and enhance the model's robustness. Specifically, we first formulate the TSG task from a causal perspective with a structural causal model. Then, to address unobserved confounders reflecting textual biases toward specific verbs or phrases, a textual causal intervention is proposed, utilizing do-calculus to estimate the causal effects. Furthermore, visual counterfactual reasoning is performed by constructing a counterfactual scenario that focuses solely on video features, excluding the query and fused multi-modal features. This allows us to debias the model by isolating and removing the influence of the video from the overall effect. Experiments on public datasets demonstrate the superiority of the proposed method. The code is available at https://github.com/Tangkfan/CICR.
Graphical User Interface (GUI) automation holds significant promise for assisting users with complex tasks, thereby enhancing accessibility and productivity. However, existing works mainly concentrate on improving AI agent's capabilities to automate procedures for liberating humans from tedious complexities, lacking the evaluation of potential erroneous agent-generated actions that could lead to task failure and irreversible system damage, inherently involving a degree of risk particularly for fields where automated systems have control over critical operations. To address this issue, we introduce TrustScorer, which evaluates the trustworthiness of actions generated by AI agents, enabling a new human-AI collaboration paradigm in GUI task automation, i.e., actions with low predicted trust scores are redirected for human intervention, thereby mingling human precision with AI efficiency. We further construct TrustBench, which provides diverse real-world GUI automation tasks with per-step ground-truth action sequence and trust score annotations, benchmarking trustworthy GUI task automation. Experimental results show that our designed three types of exploratory TrustScorer methods can help identify erroneous actions and enhance the task success rate (up to 38.1% improvement). Moreover, comprehensive analysis highlights the critical role of action trustworthiness assessment in agent automation and provides valuable insights for future explorations into more advanced trustworthiness scoring techniques to further support the reliable human-AI collaboration. Code and data are available at https://github.com/showlab/TrustScorer.
End-to-end Multiple Object Tracking (MOT) frameworks integrate detection and tracking into a unified model, avoiding intermediate information loss and complicated post-processing. However, existing end-to-end MOT trackers rely on track queries of the previous frame to provide prior information. Their limited short-term temporal modeling struggle to cope with high dynamic tracking scenarios, where inter-frame target variations exhibit significant heterogeneity. To address these shortcomings, we propose a scene-perception MOT framework (SP-MOT) that encodes scene context understanding into long-term embedding and adaptively complements it with short-term cues, enabling discriminative and flexible instance representations. Specifically, SP-MOT introduces: (1) a learnable scene query that globally profiles foreground and background to capture short-term scene-level features; (2) a context understanding module to uncover long-term stable relationships across dynamic scenes based on multiple historical scene features; (3) scene-adaptive augmented decoding that leverages scene information as guidance, adaptively aggregating long-term and short-term information into object embeddings, improving the model's association ability and fault-tolerance. Extensive experiments on MOT benchmarks demonstrate that SP-MOT outperforms state-of-the-art end-to-end trackers across multiple metrics, particularly in challenging scenarios with high dynamics.
Image steganalysis is a detection task to distinguish whether a secret message is embedded in a digital image. Due to the domain inconsistency caused by Cover Source Mismatch(CSM) and Steganographic Algorithm Mismatch (SAM), most of them suffer from significant performance degradation. Recent mismatched steganalysis focused on extracting domain invariant features by domain adversarial training or feature alignment. However these schemes are limited to unstable performance in diverse domain mismatch scenarios, and are even ineffective in some cases. In this paper, we propose a Universal Mismatched Steganalysis PCD-UMS via pair-wise confidence difference-based pseudo-label selection from the perspective of optimizing target training data. Specifically, we reveal a strong positive correlation commonality between pair-wise confidence difference and the detection performance of steganalysis among various mismatch scenarios. Based on this, a novel pseudo-label selection strategy consisting of maximum confidence difference first (MCDF) rule and pair-wise label differential storage (PLDS) rule is designed to select and filter the reliable target pseudo-labels. Furthermore, a multi-perspective pair-wise feature alignment loss is designed to initially transfer the classification ability of source steganalysis, thus solving the problem that source steganalysis fails completely under some domain mismatch scenarios. Comprehensive experiments show that our PCD-UMS outperforms the existing mismatched steganalysis by 12.07% and 3.40% in terms of detection performance under CSM and SAM scenarios.
Photo-realistic novel view synthesis from multi-view images, such as neural radiance field (NeRF) and 3D Gaussian Splatting (3DGS), has gained significant attention for its superior performance. However, most existing methods rely on low dynamic range (LDR) images, limiting their ability to capture detailed scenes in high-contrast environments. While some prior works address high dynamic range (HDR) scene reconstruction, they typically require multi-view sharp images with varying exposure times captured at fixed camera positions-a process that is time-consuming and impractical. To make data acquisition more flexible, we propose Casual3DHDR, a robust one-stage method that reconstructs 3D HDR scenes from casually-captured auto-exposure (AE) videos, even under severe motion blur and unknown, varying exposure times. Our approach integrates a continuous camera trajectory into a unified physical imaging model, jointly optimizing exposure times, camera poses, and the camera response function (CRF). Extensive experiments on synthetic and real-world datasets demonstrate that Casual3DHDR outperforms existing methods in robustness and rendering quality.
Transfer-based adversarial attacks have endowed adversarial examples with the ability to transfer from a source model to an unknown target model, which poses a more realistic threat to security-critical applications. Existing transferable adversarial attacks generally suffer from overfitting to the source model, i.e., the perturbations are locally optimal in the source model and focus on the model-specific information. We demand the adversarial perturbation to contain more generalized knowledge, which reveals the intrinsic general properties and can introduce model-general optimum into adversarial examples, for improving transferability. To this end, we devise a Bi-level Bias Mitigated Attack (BBMA), which empowers the transferability of adversarial examples by exploring generalization in two levels: 1) Progressive filtering of high-frequency sample components. We first propose to remove the sample-specific high-frequency components of samples to explore model-level generation. To simulate how a model evaluates feature importance at different stages, we devise a stride-wise step-tuning strategy to progressively produce multiple samples for aggregating the gradients. 2) Accumulated gradient-guided model attention shift. To facilitate the sample-level bias mitigation, we employ an accumulated gradient-guided attention map to distort the more generalized features during perturbation generation. Comprehensive experiments on several benchmarks demonstrate the superiority of our method in attack transferability over state-of-the-art attacks.
Retrieving target pedestrians from cross-modal images captured by infrared and visible cameras is critical in 24-hour intelligent surveillance. The primary challenge lies in narrowing the modality gap between the visible and infrared modalities. In view of this, existing research tends to extract modality-shared features to bridge the modality gap. However, the extraction process and effectiveness of the shared features are often insufficiently justified. In contrast, we observe that a certain portion of semantics remains invariant across visible and infrared modalities. These invariant semantics provide the basis for extracting modality-shared features. Based on this criterion, we propose a novel method named Low-light Invariant Representation Learning (IRL), which aims to construct an invariant space shared between visible and infrared modalities. Specifically, we introduce a Modality Invariant Extractor, which divides invariance into modality invariance and scale invariance, and extracts the invariant features from different scales and dimensions respectively. Furthermore, a Low-light Representation Enhancement module is designed, which reuses the invariant features and shallow modality features through paired enhancement units and compensation units to highlight cross-modality shared features. Extensive experiments on SYSU-MM01, RegDB, and LLCM benchmarks demonstrate the effectiveness of our method. Code is available https://github.com/Mapzzone/IRL.
Galaxy morphology analysis involves studying galaxies based on their shapes and structures. For such studies, fundamental tasks include identifying and classifying galaxies in astronomical images, as well as retrieving visually or structurally similar galaxies through similarity search. Existing methods either directly train domain-specific foundation models on large, annotated datasets or fine-tune vision foundation models on a smaller set of images. The former is effective but costly, while the latter is more resource-efficient but often yields lower accuracy. To address these challenges, we introduce GalaxAlign, a multimodal approach inspired by how citizen scientists identify galaxies in astronomical images by following textual descriptions and matching schematic symbols. Specifically, GalaxAlign employs a tri-modal alignment framework to align three types of data during fine-tuning: (1) schematic symbols representing galaxy shapes and structures, (2) textual labels for these symbols, and (3) galaxy images. By incorporating multimodal instructions, GalaxAlign eliminates the need for expensive pretraining and enhances the effectiveness of fine-tuning. Experiments on galaxy classification and similarity search demonstrate that our method effectively fine-tunes general pre-trained models for astronomical tasks by incorporating domain-specific multi-modal knowledge. Code is available at https://github.com/RapidsAtHKUST/GalaxAlign.
Prevalent lossy image compression schemes can be divided into: 1) explicit image compression (EIC), including traditional standards and neural end-to-end algorithms; 2) implicit image compression (IIC) based on implicit neural representations (INR). The former is encountering impasses of leveling off bitrate reduction at a cost of tremendous complexity while the latter suffers from excessive smoothing quality as well as lengthy decoder models. In this paper, we propose an innovative paradigm, which we dub Unicorn (Unified Neural Image Compression with One Nnumber Reconstruction). By conceptualizing the images as index-image pairs and learning the inherent distribution of pairs in a subtle neural network model, Unicorn can reconstruct a visually pleasing image from a randomly generated noise with only one index number. The neural model serves as the unified decoder of images while the noises and indexes corresponds to explicit representations. As a proof of concept, we propose an effective and efficient prototype of Unicorn based on latent diffusion models with tailored model designs. Quantitive and qualitative experimental results demonstrate that our prototype achieves significant bitrates reduction compared with EIC and IIC algorithms. More impressively, benefitting from the unified decoder, our compression ratio escalates as the quantity of images increases. We envision that more advanced model designs will endow Unicorn with greater potential in image compression. The code will be made publicly available upon publication.
3D Gaussian Splatting (3DGS) has revolutionized 3D scene reconstruction, which effectively balances rendering quality, efficiency, and speed. However, existing 3DGS approaches usually generate plausible outputs and face significant challenges in complex scene reconstruction, manifesting as incomplete holistic structural outlines and unclear local lighting effects. To address these issues simultaneously, we propose a novel decoupled optimization framework, which integrates wavelet decomposition into 3D Gaussian Splatting and 2D sampling. Technically, through 3D wavelet decomposition, our approach divides point clouds into high-frequency and low-frequency components, enabling targeted optimization for each. The low-frequency component captures global structural outlines and manages the distribution of Gaussians through voxelization. In contrast, the high-frequency component restores intricate geometric and textural details while incorporating a relight module to mitigate lighting artifacts and enhance photorealistic rendering. Additionally, a 2D wavelet decomposition is applied to the training images, simulating radiance variations. This provides critical guidance for high-frequency detail reconstruction, ensuring seamless integration of details with the global structure. Extensive experiments on challenging datasets demonstrate our method achieves state-of-the-art performance across various metrics, surpassing existing approaches and advancing the field of 3D scene reconstruction.
Online Cardiac Monitoring (OCM) emerges as a compelling enhancement for the next-generation video streaming platforms. It enables various applications including remote health, affective computing, and deepfake detection. Yet the physiological information encapsulated in the video streams has been long neglected. In this paper, we present the design and implementation of CardioLive, the first online cardiac monitoring system in video streaming platforms. We leverage the naturally co-existed video and audio streams and devise CardioNet, the first audio-visual network to learn the cardiac series. It incorporates multiple unique designs to extract temporal and spectral features, ensuring robust performance under realistic streaming conditions. To enable the Service-On-Demand OCM, we implement CardioLive as a plug-and-play middleware service and develop systematic solutions to practical issues including changing FPS and unsynchronized streams. Extensive evaluations demonstrate the effectiveness of our system. We achieve a Mean Squared Error of 1.79 BPM error, outperforming the video-only and audio-only solutions by 69.2% and 81.2%, respectively. CardioLive achieves average throughput of 115.97 and 98.16 FPS in Zoom and YouTube. We believe our work opens up new applications for video stream systems. Code is available at https://github.com/aiot-lab/CardioLive.
Remote sensing image restoration under cloud and haze occlusions poses a significant challenge due to severe spectral degradation and spatial distortions. While recent generative models have shown promise in image restoration, they struggle with three key issues: (1) Lack of precise annotations, making supervised methods unreliable; (2) Unintended interference with clear regions, leading to distortion in unaffected areas; (3) Spectral and structural inconsistencies in heavily occluded regions, limiting realistic recovery. To address these challenges, we propose Saliency-Guided Adaptive Random Diffusion Strategy(SG-ARD), a novel blind restoration framework that integrates saliency-aware guidance with adaptive diffusion for enhanced reconstruction. First, we introduce a Saliency-Guided Pseudo-label Generation module (SGPG) to identify degraded regions and generate pseudo-labels for blind restoration. Second, we propose an Adaptive Random Diffusion Correction Strategy (ARDC), which employs a Random-Walk-based Diffusion and an Adaptive Enhancement module to refine local and global texture pseudo-labels. Lastly, we design a Spectral-Aware Consistency Loss (SAC) to improve spectral fidelity, ensuring that the generated content aligns with the real spectral distribution. Extensive experiments on three large-scale remote sensing datasets demonstrate that SG-ARD outperforms state-of-the-art generative restoration models, producing high-fidelity, visually coherent remote sensing images.
Electroencephalogram (EEG) signal classification faces significant challenges due to data distribution shifts caused by heterogeneous electrode configurations, acquisition protocols, and hardware discrepancies across domains. This paper introduces IMAC, a novel channel-dependent mask and imputation self-supervised framework that formulates the alignment of cross-domain EEG data shifts as a spatial time series imputation task. To address heterogeneous electrode configurations in cross-domain scenarios, IMAC first standardizes different electrode layouts using a 3D-to-2D positional unification mapping strategy, establishing unified spatial representations. Unlike previous mask-based self-supervised representation learning methods, IMAC introduces spatio-temporal signal alignment. This involves constructing a channel-dependent mask and reconstruction task framed as a low-to-high resolution EEG spatial imputation problem. Consequently, this approach simulates cross-domain variations such as channel omissions and temporal instabilities, thus enabling the model to leverage the proposed imputer for robust signal alignment during inference. Furthermore, IMAC incorporates a disentangled structure that separately models the temporal and spatial information of the EEG signals separately, reducing computational complexity while enhancing flexibility and adaptability. Comprehensive evaluations across 10 publicly available EEG datasets demonstrate IMAC's superior performance, achieving state-of-the-art classification accuracy in both cross-subject and cross-center validation scenarios. Notably, IMAC shows strong robustness under both simulated and real-world distribution shifts, surpassing baseline methods by up to 35% in integrity scores while maintaining consistent classification accuracy.
Accurately identifying correct correspondences in two images is a crucial task in computer vision. Current methods predominantly use PointCN blocks as feature extraction backbones and learn local-global consensus through a progressive learning strategy. However, such methods have two main drawbacks: First, PointCN blocks, composed of multilayer perceptrons and normalization layers, process spatial positions independently, leading to limited interaction between channel-wise and spatial-wise dimensions. Second, the progressive learning strategy primarily focuses on unidirectional transfer from local to global consensus, yet neglects the bidirectional interaction between local and global consensus. To address these issues, we propose the Channel-Spatial interaction and Bidirectional Consensus interaction-Based Network (CSBCNet), which contains three innovative blocks: Channel-Spatial Interaction (CSI), Local Consensus Mining (LCM), and Global Consensus-Aware Attention (GCAA). Specifically, CSI enhances interaction between channel-wise and spatial-wise dimensions through a dual-path attention mechanism, addressing the limited interaction caused by the independent processing of spatial positions in PointCN blocks. LCM extracts reliable local consensus by modeling geometric structures and spatial continuity within correspondences. GCAA captures global consensus by aggregating correspondences that are highly likely to be correct ones, and achieves bidirectional interaction between local and global consensus through cross attention. Experiments demonstrate our CSBCNet's superior performance in camera pose estimation and correspondence pruning. Notably, when the CSI block is applied to the existing OANet and MS2DGNet networks, it achieves significant performance improvements of 10.27% and 7.5%, respectively, on the mAP5° metric on the camera pose estimation task.
Reconstructing and segmenting scenes from unconstrained photo collections obtained from the Internet is a novel but challenging task. Unconstrained photo collections are easier to get than well-captured photo collections. These unconstrained images suffer from inconsistent lighting and transient occlusions, which makes segmentation challenging. Previous segmentation methods cannot address transient occlusions or accurately restore the scene's lighting conditions. Therefore, we propose Seg-Wild, an interactive segmentation method based on 3D Gaussian Splatting for unconstrained image collections, suitable for in-the-wild scenes. We integrate multi-dimensional feature embeddings for each 3D Gaussian and calculate the feature similarity between the feature embeddings and the segmentation target to achieve interactive segmentation in the 3D scene. Additionally, we introduce the Spiky 3D Gaussian Cutter (SGC) to smooth abnormal 3D Gaussians. We project the 3D Gaussians onto a 2D plane and calculate the ratio of 3D Gaussians that need to be cut using the SAM mask. We also designed a benchmark to evaluate segmentation quality in in-the-wild scenes. Experimental results demonstrate that compared to previous methods, Seg-Wild achieves better segmentation results and reconstruction quality. Our code will be available at https://github.com/Sugar0725/Seg-Wild.
Human-Object Interaction (HOI) detection involves detecting human-object pairs and predicting their interactions. However, it faces significant challenges due to the complexity of human behavior and the diverse contexts in which interactions occur. Contextual cues, such as the participants involved, body language, and the surrounding environment, are crucial for accurately identifying interactions, particularly those that are ambiguous or previously unseen. In this paper, we propose ConCue, a novel approach that integrates contextual cue generation with feature extraction to enhance HOI detection. Specifically, we design specialized prompts tailored for Large Vision-Language Models (VLMs), enabling the generation of rich contextual cues from images. These cues are then seamlessly integrated into HOI detection through a feature extraction module with a multi-tower architecture we developed, which effectively incorporates contextual information into both instance and interaction detection processes. Extensive experimental results demonstrate the effectiveness of ConCue. Integrating ConCue with state-of-the-art HOI methods leads to significant performance improvements on two widely used benchmark datasets, highlighting the potential of our approach in advancing HOI detection.
Temporal forgery in multimedia-where audio or video streams are subtly manipulated-poses critical challenges for content authenticity verification. While video-level detection has advanced, Temporal Forgery Localization (TFL) remains underexplored, often limited by weak audio-visual modeling and reliance on non-learnable post-processing. To address these challenges, we propose RegQAV, a Register-enhanced Query-based Audio-Visual framework for TFL. RegQAV exploits pretrained foundation models to capture fine-grained audio-visual correspondences and learnable registers are introduced to mitigate the model's tendency to overly focus on a limited set of temporal features. A query-based localization strategy enables end-to-end optimization without post-processing. We also introduce a Modality Fusion Adapter (MFA) for effective multi-scale integration of audio-visual data, a Deepfake Queries Generation (DQG) module for efficient query initialization, and a Poisson Count-Based Approach to dynamically predict the number of forgeries. Experiments on LAV-DF and AV-Deepfake1M show that RegQAV achieves state-of-the-art performance with fewer parameters, faster inference, and stronger generalization. This work offers significant potential for real-time deepfake detection and other multimedia verification applications. The code is available at https://github.com/zxd3099/RegQAV.
Previous research on Multimodal Relation Extraction (MRE) has primarily focused on identifying textual relations enhanced by static visual clues from images, benefiting fields such as multimedia analysis and knowledge graphs. With the rapid rise of video content on social media platforms, Multimodal Relation Extraction (MRE) systems face new challenges. To bridge this gap, we introduce Video-level Multimodal Relation Extraction (VMRE), a novel task aimed at extracting relational facts from videos. To advance this research, we present Vid-MRE, a new dataset containing 32 relation types and 12,402 multimodal relational facts, annotated across 3,970 pairs of textual news titles and corresponding videos. Since this task demands precise event and entity grounding to filter out excessive noise in the video, we propose an Event-Entity Semantic Consistency Network (E2SCN) to capture relational clues in the video effectively. Experimental results demonstrate that incorporating video content into the model significantly improves relation identification performance but also introduces more noise. Our E2SCN method effectively reduces the noise, enhancing fine-grained multimodal event and entity alignments while achieving state-of-the-art (SOTA) performance.