CoFiVLA: Synergistic Coarse-Fine Vision-Language Alignment for Image Aesthetic Assessment

摘要

Image aesthetic assessment (IAA) is challenging due to its subjective and diverse nature, making visual information alone insufficient. Existing methods using paired image and user comments, though effective, have limited practicality. Moreover, user comments contain both high- and weak-aesthetic-related textual information, thus directly integrating all textual information with visual information cannot guarantee the effectiveness. To address these issues, we propose a novel synergistic coarse-fine vision-language alignment (CoFiVLA) framework for IAA. It includes a CoFiVLA pretraining network and a CoFiVLA prediction network, aiming to effectively and comprehensively align and synergize image and text modalities at both coarse and fine granularities during training, while eliminating the dependence on user comments during inference. In the CoFiVLA pretraining network, the coarse- and fine-grained vision-language alignment branches work together, aligning the visual features with the high-aesthetic-related textual information and all textual information, respectively. In the coarse-grained alignment branch, we innovatively propose employing the large language model LLaMA to construct an aesthetic summary dataset, which extracts high-aesthetic-related text from user comments. Furthermore, in the CoFiVLA prediction network, we first extract features corresponding to learnable prompts of different aesthetic quality categories based on the CoFiVLA pretrained model, then fuse visual features with these learnable textual features. Thus we achieve aesthetic quality semantics embedded image aesthetic representations for effective IAA without requiring user comments during inference. Our aesthetic summary dataset and source code are available at https://github.com/lifusheng-chn/CoFiVLA.