Recent advances in multi-view 3D multi-person pose estimation have led to significant progress. However, several critical challenges remain, including the limited extraction and integration of multi-domain information, as well as the high annotation costs associated with 3D data in multi-person scenarios. These issues hinder the broader applicability of current methods in complex computer vision tasks. In this paper, we propose a Dense-Sparse Parallel Networks (DSP) framework that jointly leverages spatial, temporal, and frequency-domain information through an adaptive geo-consistency self-supervised strategy. Specifically, we design a multi-view spatial feature extraction module that captures cross-view spatial distributions from dense multi-view feature maps. In parallel, we employ a local-global temporal attention module and a frequency-aware attention module to extract dynamic temporal patterns and localized frequency-domain features from sparse keypoint data. Furthermore, a multi-domain parallel fusion module is introduced to effectively integrate features across all domains, enabling accurate multi-person 3D pose regression. To enhance self-supervised learning, we employ a dynamic view selector guided by reinforcement learning, which reduces the impact of inaccurate pre-trained 2D poses. Experimental results on three benchmark datasets (i.e., CMU Panoptic, Campus, and Shelf) demonstrate that the proposed DSP framework achieves robust and accurate performance, as evidenced by comparisons with other state-of-the-art methods.
论文检索
输入标题、作者或关键词，从 1,620 篇学术成果中精准定位
Pre-trained Vision-Language Models (VLMs) have been exploited in various Computer Vision tasks (e.g., few-shot recognition) via model adaptation, such as prompt tuning and adapters. However, existing adaptation methods are designed by human experts, requiring significant time cost and experience. Inspired by recent advances in Large Language Models (LLMs) based code generation, we propose an Evolutionary Vision-Language Model Adaptation (EvoVLMA) method to automatically search training-free efficient adaptation algorithms for VLMs. We recognize feature selection and logits computation as the key functions in training-free VLM adaptation, and propose a two-stage LLM-assisted evolutionary algorithm for optimizing these parts in a sequential manner, effectively addressing the challenge posed by the expansive search space through a divide-and-conquer strategy. Besides, to enhance the stability and efficiency of searching process, we propose low-precision code conversion, web based code execution and process monitoring, leading to a highly effective automatic algorithm design system. Extensive experiments demonstrate that the algorithms found by EvoVLMA can obtain promising results compared to previous manually-designed ones. More specifically, in the 8-shot image classification setting, the classical APE algorithm can be improved by 1.91 points in recognition accuracy. This research opens new possibilities for automating the optimization of adaptation algorithms of pre-trained multimodal models. Code is available at: https://github.com/kding1225/EvoVLMA
3D Visual Question Answering (3D VQA) is crucial for enabling models to perceive the physical world and perform spatial reasoning. In 3D VQA, the free-form nature of answers often leads to improper annotations that can confuse or mislead models when training on the entire dataset. While other text generation tasks can mitigate this issue by learning on large-scale datasets, the scarcity of 3D scene data enlarges the negative effect of misleading annotations. Although active learning strategies can select valuable instances for training, they fail to identify and resolve misleading labels, which the oracle inevitably provides in practice. To address this issue, we propose a multi-turn interactive active learning strategy. This strategy selects data based on models' semantic uncertainty to form a solid knowledge foundation more effectively and actively requests reannotation from an oracle to resolve potentially misleading labels. For uncertainty assessment, we utilize a variance-based metric that takes semantic relationships between terms into consideration, thus avoiding the uniform inter-class similarity assumption of previous assessment metrics. Extensive experiments exhibit better model performance and a substantial reduction in training costs, with a halving of training costs for achieving relatively high accuracy. The code is available at https://github.com/fz-zsl/AQuA.
Large Language Models (LLM) can significantly enhance the Vision-Language Model's prompting capabilities (e.g. CLIP) by generating detailed and comprehensive prompts. However, the LLMs are prone to generating hallucinated text prompts, leading to misalignment between visual and textual representations in the semantic space. Moreover, sparse sampling of images in low-shot scenarios often leads to incomplete or biased concept representations between two modalities, as the limited data struggles to capture the full range of concept attributes. To this end, we propose Chain-of-Thought Guided Low-shot Debiasing (CoLD), which addresses two key challenges: the textual biases generated by the hallucination issues and the visual biases due to insufficient discriminative features. Specifically, a multi-granularity chain-of-thought (CoT) prompting strategy is proposed that reduces the negative impact of irrelevant textual information, thereby effectively mitigating hallucinations and alleviating the issue of textual bias. Additionally, we propose visual anchoring that utilizes CoT prompts with class attributes and characteristics to generate auxiliary visual features. The anchors optimize the existing semantic space by introducing additional visually-grounded concepts, thereby mitigating visual biases and enhancing low-shot performance. Extensive experiments demonstrate the effectiveness of the proposed method, showcasing notable performance improvements across various datasets.
Images captured in low-light nighttime scenes suffer from light effects. Existing nighttime visibility enhancement methods predominantly focus on low-light image enhancement (LLIE), neglecting light-effect suppression (LES). Current LES methods mainly rely on unsupervised or zero-shot learning due to the lack of paired nighttime light-effect datasets. We construct a large-scale nighttime dataset containing diverse light effects to enable supervised learning for joint LES and LLIE. We design a two-stage structural prior-guided diffusion model for nighttime visibility enhancement, proposing a Laplacian decomposition physical model and a dual-loop Receptance Weighted Key Value (RWKV) to separate light effects from structural features. Experimental results demonstrate that our method outperforms state-of-the-art (SOTA) methods in LLIE and LES tasks. Through supervised training on our dataset, our method achieves optimal performance in joint LES and LLIE while maintaining effectiveness across various real-world scenarios.
Weakly Supervised Object Localization (WSOL) relies only on image-level labels to realize object localization, significantly reducing the cost for fine-grained annotations. While traditional CAM-based methods excel at identifying the most prominent regions of objects, they frequently neglect other essential components, resulting in partial or incomplete object localization. The foreground prediction map (FPM) generates finer-grained activation maps using underlying features to address the shortcomings of CAM, but it may still have coverage blind spots. To this end, this paper proposes a collaborative optimization framework based on cross-modal semantic alignment that deeply integrates the saliency awareness of CAM with the refined representation capabilities of FPM. It introduces a multimodal pretrained model (CLIP) to construct a semantic-driven WSOL paradigm. By dynamically interacting CLIP's text embeddings with the semantic of image categories, a semantic-enhanced FPM based on similarity measurement is generated. Leveraging CLIP's cross-modal alignment capabilities, a targeted generation scheme is designed. On the one hand, the CLIP model is frozen and its features are refined through a decoder to obtain richer semantic representations; On the other hand, by using knowledge distillation, the CAM generated by CLIP is taken as a reference benchmark, guiding the network to learn more accurate target localization. Additionally, to enhance FPM's focus on foreground regions, the Exponential Decay Foreground Emphasis (EDFE) module is designed, which uses a differentiated excitation strategy to effectively suppress background interference and highlight target areas. Experimental results show that our method significantly improves the completeness and boundary accuracy of target localization under weak supervision, laying a solid foundation for subsequent downstream tasks.
Social group detection aims to identify groups of individuals exhibiting social behavior from multi-individual trajectory data. Recent approaches often determine group correlations based on global trajectory similarity, while temporal dynamics can cause diverging member trajectories and undermine similarity-based measures. Other methods model pairwise interaction strengths to capture group relations, focusing only on explicit direct interactions while ignoring implicit indirect interactions. To address temporal variability of group structures, we decompose long trajectories into multiple semantic sub-trajectories, enabling the capture of dynamic characteristics. Furthermore, to explore implicit indirect interactions, we introduce a unified spatio-temporal graph structure that models both direct and indirect interactions among individuals. In addition, considering the contextual influence of the neighborhood of an individual, we incorporate neighborhood information into the trajectory representation process. Based on these insights, we propose a Segmented-Trajectory-Aware Spatio-Temporal Graph Convolutional Network (SegTraj). This framework uniformly models explicit and implicit interactions through a spatio-temporal graph, and fuses individual trajectories with contextual neighborhood information for fine-grained representation of group relationships. Extensive experiments on three datasets covering both synthetic and real-world scenarios demonstrate that SegTraj significantly outperforms baseline methods. The code is available at https://github.com/DC0827/SegTraj.
Multi-domain Task Incremental Learning (MTIL) aims to continuously acquire knowledge from diverse domains while maintaining generalization capability. Recent works have demonstrated promising results by leveraging Vision-Language Models (VLMs) for continual learning. However, we identify a critical issue within this paradigm, termed the Evolving Semantic Entanglement. Specifically, VLMs tend to produce highly similar text features for semantically related categories, resulting in hard feature alignment and interference between related categories. This problem becomes increasingly pronounced as the category space expands. In this paper, we present a novel Dual-granularity Prompt Learning (DuPLe) framework to address this challenge. Our approach enhances text feature discriminability by leveraging complementary two-level prompts: category-level global prompts for holistic semantic concepts and attribute-level local prompts for fine-grained visual patterns. We further apply a Task Assignment-free Inference strategy that eliminates explicit task identification, simplifying the inference process and enabling extension to unseen categories. Extensive experiments on 11 diverse domains under MTIL and X-TAIL settings demonstrate that our method significantly mitigates the entanglement issue and outperforms previous state-of-the-art approaches.
Monocular 3D visual grounding is a novel task that aims to locate 3D objects in RGB images using text descriptions with explicit geometry information. Despite the inclusion of geometry details in the text, we observe that the text embeddings are sensitive to the magnitude of numerical values but largely ignore the associated measurement units. For example, simply equidistant mapping the length with unit 'meters' to 'decimeters' or 'centimeters' leads to severe performance degradation, even though the physical length remains equivalent. This observation signifies the weak 3D comprehension of pre-trained language model, which generates misguiding text features to hinder 3D perception. Therefore, we propose to enhance the 3D perception of model on text embeddings and geometry features with two simple and effective methods. Firstly, we introduce a pre-processing method named 3D-text Enhancement (3DTE), which enhances the comprehension of mapping relationships between different units by augmenting the diversity of distance descriptors in text queries. Next, we propose a Text-Guided Geometry Enhancement (TGE) module to further enhance the 3D-text information by projecting the basic text features into geometrically consistent space. These 3D-enhanced text features are then leveraged to precisely guide the attention of geometry features. We evaluate the proposed method through extensive comparisons and ablation studies on the Mono3DRefer dataset. Experimental results demonstrate substantial improvements over previous methods, achieving new state-of-the-art results with a notable accuracy gain of 11.94% in the 'Far' scenario. Our code will be made publicly available.
Most of existing weakly supervised GAR methods are typically bottom-up, automatically mining key areas by the attention mechanism. Due to the lack of a semantic connection to individual actions, some regions associated with these actions may be omitted, potentially impacting performance. In fact, a group activity is a combination of multiple individual actions, and the prototype of a specific action can be obtained from visual representations of individuals performing it, denoted as visual conceptual knowledge. In this paper, we propose a Visual Conceptual Knowledge Guided Action Map framework. It uses prototypes to produce individual action maps that indicate the likelihood of actions occurring at different locations. In some scenarios, the spatial distribution of actions shows strong regularity, which we compile as A-A Maps to enhance individual action maps. The action maps are integrated with action semantic representations for group activity recognition. Extensive experiments on two public benchmarks, the Volleyball and the NBA datasets, demonstrate the effectiveness of our proposed method, even in cases of limited training data.
Compositional Zero-Shot Learning (CZSL) aims to recognize unseen combinations of seen attributes and objects. Current CLIP-based methods in CZSL, despite their advancements, often fail to effectively understand and link the attributes and objects due to inherent limitations in CLIP's pretraining mechanisms. To address these shortcomings, this paper introduces a novel framework, Understanding and Linking Attributes and Objects (ULAO) in CZSL, which comprises two innovative modules. The Understanding Attributes and Objects (UAO) module improves primitive understanding by sequential primitive prediction and leveraging recognized objects as contextual hints for attribute classification. Concurrently, the Linking Attributes and Objects (LAO) module improves the attribute-object linkage understanding through a new contrastive learning strategy that incorporates tailored hard negative generation and adaptive loss adjustments. We demonstrate our model's superiority by showcasing its state-of-the-art performance across three benchmark datasets in both Closed-World (CW) and Open-World (OW) scenarios.
Natural Language-Guided Drones (NLGD) offer a novel and flexible interaction paradigm for tasks such as target matching and navigation. However, the wide field of view and complex compositional semantic relationships inherent in drone scenarios place greater demands on visual language understanding. First, mainstream Vision-Language Models (VLMs) primarily focus on global feature alignment and lack fine-grained semantic understanding. Second, existing hierarchical semantic modeling methods rely on precise entity partitioning and strict containment relationship constraints, which limits their effectiveness in complex drone environments. To address these challenges, we propose the Hierarchical Cross-Granularity Contrastive and Matching learning (HCCM) framework, comprising two core components: 1) Region-Global Image-Text Contrastive Learning (RG-ITC). Avoiding precise scene entity partitioning, RG-ITC models hierarchical local-to-global cross-modal semantics by contrasting local visual regions with global text semantics, and vice versa. 2) Region-Global Image-Text Matching Learning (RG-ITM). Instead of relying on strict relationship constraints, this component evaluates local semantic consistency within global cross-modal representations, improving the comprehension of complex compositional semantics. Furthermore, drone scenario textual descriptions are often incomplete or ambiguous, destabilizing global semantic alignment. To mitigate this, HCCM incorporates a Momentum Contrast and Momentum Distillation (MCD) mechanism, enhancing alignment robustness. Extensive experiments on the GeoText-1652 benchmark demonstrate HCCM significantly outperforms existing methods, achieving state-of-the-art Recall@1 scores of 28.8% (image retrieval) and 14.7% (text retrieval). Moreover, HCCM exhibits strong zero-shot generalization on the unseen ERA dataset, achieving 39.93% mean recall (mR), surpassing evaluated fine-tuned models. These results highlight the effectiveness and robustness of HCCM across diverse scenarios. Our implementation is available at https://github.com/rhao-hur/HCCM.
Video Moment Retrieval (MR) and Highlight Detection (HD) aim to pinpoint specific moments and assess clip-wise relevance based on the text query. While DETR-based joint frameworks have made significant strides, there remains untapped potential in harnessing the intricate relationships between temporal motion and spatial semantics within video content. In this paper, we propose the Motion-Semantics DETR (MS-DETR), a framework that captures rich motion-semantics features through unified learning for MR/HD tasks. The encoder first explicitly models disentangled intra-modal correlations within motion and semantics dimensions, guided by the given text queries. Subsequently, the decoder utilizes the task-wise correlation across temporal motion and spatial semantics dimensions to enable precise query-guided localization for MR and refined highlight boundary delineation for HD. Furthermore, we observe the inherent sparsity dilemma within the motion and semantics dimensions of MR/HD datasets. To address this issue, we enrich the corpus from both dimensions by generation strategies and propose contrastive denoising learning to ensure the above components learn robustly and effectively. Extensive experiments on four MR/HD benchmarks demonstrate that our method outperforms existing state-of-the-art models by a margin. Our code is available at https://github.com/snailma0229/MS-DETR.git.
Zero-shot object detection (ZSD) aims to leverage semantic descriptions to localize and recognize objects of both seen and unseen classes. Existing ZSD works are mainly coarse-grained object detection, where the classes are visually quite different, thus are relatively easy to distinguish. However, in real life we often have to face fine-grained object detection scenarios, where the classes are too similar to be easily distinguished. For example, detecting different kinds of birds, fishes, and flowers. In this paper, we propose and solve a new problem called Fine-Grained Zero-Shot Object Detection (FG-ZSD for short), which aims to detect objects of different classes with minute differences in details under the ZSD paradigm. We develop an effective method called MSHC for the FG-ZSD task, which is based on an improved two-stage detector and employs a multi-level semantics-aware embedding alignment loss, ensuring tight coupling between the visual and semantic spaces. Considering that existing ZSD datasets are not suitable for the new FG-ZSD task, we build the first FG-ZSD benchmark dataset FGZSD-Birds, which contains 148,820 images falling into 36 orders, 140 families, 579 genera and 1432 species. Extensive experiments on FGZSD-Birds show that our method outperforms existing ZSD models.
Editing long videos remains a challenging task due to the need for maintaining both global consistency and temporal coherence across thousands of frames. Existing methods often suffer from structural drift or temporal artifacts, particularly in minute-long sequences. We introduce AnchorSync, a novel diffusion-based framework that enables high-quality, long-term video editing by decoupling the task into sparse anchor frame editing and smooth intermediate frame interpolation. Our approach enforces structural consistency through a progressive denoising process and preserves temporal dynamics via multimodal guidance. Extensive experiments show that AnchorSync produces coherent, high-fidelity edits, surpassing prior methods in visual quality and temporal stability. The source code is available at https://github.com/VISION-SJTU/AnchorSync.
Multi-view 3D reconstruction has remained an essential yet challenging problem in the field of computer vision. While DUSt3R and its successors have achieved breakthroughs in 3D reconstruction from unposed images, these methods exhibit significant limitations when scaling to multi-view scenarios, including high computational cost and cumulative error induced by global alignment. To address these challenges, we propose Regist3R, a novel stereo foundation model tailored for efficient and scalable incremental reconstruction. Regist3R leverages an incremental reconstruction paradigm, enabling large-scale 3D reconstructions from unordered and many-view image collections. We evaluate Regist3R on public datasets for camera pose estimation and 3D reconstruction. Our experiments demonstrate that Regist3R achieves comparable performance with optimization-based methods while significantly improving computational efficiency, and outperforms existing multi-view reconstruction models. Furthermore, to assess its performance in real-world applications, we introduce a challenging oblique aerial dataset which has long spatial spans and hundreds of views. The results highlight the effectiveness of Regist3R. We also demonstrate the first attempt to reconstruct large-scale scenes encompassing over thousands of views through pointmap-based foundation models, showcasing its potential for practical applications in large-scale 3D reconstruction tasks, including urban modeling, aerial mapping, and beyond.
Sparse-view 3D reconstruction is a fundamental yet challenging task in practical 3D reconstruction applications. Recently, many methods based on 3D Gaussian Splatting (3DGS) have been proposed to address sparse-view 3D reconstruction. Although these methods have made considerable advancements, they still show significant issues with overfitting. To reduce the overfitting, we introduce VGNC, a novel Validation-guided Gaussian Number Control approach based on generative novel view synthesis (NVS) models. To the best of our knowledge, this is the first attempt to alleviate the overfitting issue of sparse-view 3DGS with generative validation images. Specifically, we first introduce a validation image generation method based on a generative NVS model. We then propose a Gaussian number control strategy that utilizes generated validation images to determine optimal Gaussian numbers, thereby reducing the issue of overfitting. We conducted detailed experiments on various sparse-view 3DGS baselines and datasets to evaluate the effectiveness of VGNC. Extensive experiments show that our approach not only reduces overfitting but also improves rendering quality on the test set while decreasing the number of Gaussians. This reduction lowers storage demands and accelerates both training and rendering. Our code is available at: https://github.com/LinLif1869/VGNC.
Text-Image Retrieval (TIR) retrieves a target image from a gallery based on an image-level description, while Referring Expression Comprehension (REC) localizes a target object within a given image using an instance-level description. However, real-world applications often present more complex demands. Users typically query an instance-level description across a large gallery and expect to receive both relevant image and the corresponding instance location. In such scenarios, TIR struggles with fine-grained descriptions and object-level localization, while REC is limited in its ability to efficiently search large galleries and lacks an effective ranking mechanism. In this paper, we introduce a new task called Referring Expression Instance Retrieval (REIR), which supports both instance-level retrieval and localization based on fine-grained referring expressions. First, we propose a large-scale benchmark for REIR, named REIRCOCO, constructed by prompting advanced vision-language models to generate high quality referring expressions for instances in the MSCOCO and RefCOCO datasets. Second, we present a baseline method, Contrastive Language Instance Alignment with Relation Experts (CLARE), which employs a dual-stream architecture to address REIR in an end-to-end manner. Given a referring expression, the textual branch encodes it into a query embedding, enhanced by a Mix of Relation Experts (MORE) module designed to better capture inter-instance relationships. The visual branch detects candidate objects and extracts their instance-level visual features. The most similar candidate to the query is selected for bounding box prediction. CLARE is first trained on object detection and REC datasets to establish initial grounding capabilities, then optimized via Contrastive Language Instance Alignment (CLIA) for improved retrieval across images. Experimental results demonstrate that CLARE outperforms existing methods on the REIR benchmark and generalizes well to both TIR and REC tasks, showcasing its effectiveness and versatility.
Existing Unsupervised Video Object Segmentation (UVOS) solutions primarily focus on frame-to-frame propagation and often struggle with extended sequences where objects undergo complex transformations. In this work, we observe that many natural and artificial motions exhibit inherent periodicity, where objects return to similar states across time, particularly in complex scenarios. Leveraging this insight, we present PeriodVOS, a novel framework that exploits recurring motion patterns to enhance segmentation quality across diverse video contexts. Specifically, we propose to establish intra-period consistency to enforce stable segmentation within short time windows, while mitigating the effects of temporary disturbances. Furthermore, to capture global dependencies, we present inter-period correlation to build associations between similar object states across different time periods. Additionally, an adaptive temporal contextual coupling is designed to dynamically adjust how temporal context is integrated based on video content. Through extensive evaluation on six standard benchmarks, including DAVIS-2016, FBMS, Youtube-Objects, DAVSOD, ViSal, and MCL datasets, our PeriodVOS outperforms state-of-thearts, demonstrating the potential of video periodic mining particularly in challenging scenarios. We have released source code on https://github.com/smdshzyjbr-qhw/PeriodVOS.
Decoding dynamic visual information from brain activity remains challenging due to inter-subject neural heterogeneity, limited per-subject data availability, and the substantial temporal resolution gap between fMRI signals (0.5Hz) and video dynamics (30Hz). Current approaches face persistent challenges in addressing these temporal mismatches, demonstrate limited capacity to integrate subject-specific neural patterns with shared representational frameworks, and lack adequate semantic granularity for aligning neural responses with visual content. To bridge these gaps, we propose CrossMind-VL, a framework addressing these limitations through three innovations: (1) a Dynamic Temporal Alignment module that resolves temporal mismatches via exponentially decayed multi-frame fusion with adaptive decay coefficients; (2) a Brain Mixture-of-Experts architecture that combines subject-specific extractors with shared expert layers through parameter-efficient tri-modal contrastive learning; and (3) a Multi-perspective Semantic Hyper-Anchoring module that resolves cross-subject attention bias via multi-dimensional semantic decomposition, leveraging multimodal LLMs for fine-grained video semantic extraction-enabling the model to match individual attention patterns as different subjects naturally focus on distinct aspects of the same visual stimulus. This module boosts Top-10/Top-100 retrieval by 17.7%/6.6%. Experiments on two video-fMRI datasets demonstrate state-of-the-art performance, with 39%/30% improvements in Top-10/Top-100 accuracy over single-subject baselines and 27% gains against multi-subject models. The framework exhibits remarkable few-shot adaptability, retaining 97% performance when using only 10% training data for new subjects. Visualization analysis confirms this generalization capability stems from effective disentanglement of subject-specific and shared neural representations.