From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models

摘要

Hallucination in large vision-language models (LVLMs) is a significant challenge, i.e., generating objects that are not present in the visual input, which significantly compromises the reliability of models. Recent studies often attribute hallucinations to a lack of visual understanding, yet ignore a more fundamental issue: the model's inability to effectively extract or decouple visual features. In this paper, we revisit the hallucinations in LVLMs from an architectural perspective, investigating whether the primary cause lies in the visual encoder (feature extraction) or the modal alignment module (feature decoupling). Motivated by our preliminary findings, we propose a parameter-efficient fine-tuning strategy, PATCH, to mitigate hallucinations in LVLMs. This plug-and-play method can be integrated into various LVLMs, leveraging adaptive virtual tokens to extract object features from bounding boxes, thereby addressing hallucinations stemming from inadequate feature decoupling. PATCH achieves state-of-the-art performance across multiple multi-modal hallucination datasets and demonstrates significant improvements in general capabilities. We hope this work provides deeper insights into the underlying causes of hallucinations in LVLMs, fostering further advancements and innovation in this field. The code will be available at https://github.com/YuyingShang/PATCH.