Chain-of-Thought Guided Semantic Debiasing for Low-Shot Vision-Language Tasks

摘要

Large Language Models (LLM) can significantly enhance the Vision-Language Model's prompting capabilities (e.g. CLIP) by generating detailed and comprehensive prompts. However, the LLMs are prone to generating hallucinated text prompts, leading to misalignment between visual and textual representations in the semantic space. Moreover, sparse sampling of images in low-shot scenarios often leads to incomplete or biased concept representations between two modalities, as the limited data struggles to capture the full range of concept attributes. To this end, we propose Chain-of-Thought Guided Low-shot Debiasing (CoLD), which addresses two key challenges: the textual biases generated by the hallucination issues and the visual biases due to insufficient discriminative features. Specifically, a multi-granularity chain-of-thought (CoT) prompting strategy is proposed that reduces the negative impact of irrelevant textual information, thereby effectively mitigating hallucinations and alleviating the issue of textual bias. Additionally, we propose visual anchoring that utilizes CoT prompts with class attributes and characteristics to generate auxiliary visual features. The anchors optimize the existing semantic space by introducing additional visually-grounded concepts, thereby mitigating visual biases and enhancing low-shot performance. Extensive experiments demonstrate the effectiveness of the proposed method, showcasing notable performance improvements across various datasets.