Compositional Zero-shot Learning via Progressive Language-based Observations

摘要

Compositional zero-shot learning aims to recognize unseen stateobject compositions by leveraging known primitives (state and object) during training. However, effectively modeling interactions between primitives and generalizing knowledge to novel compositions remains a perennial challenge. There are two crucial factors: large object-conditioned and state-conditioned variance, i.e., the appearance of states (or objects) can vary significantly when combined with different objects (or states). For instance, the state "old" can signify vintage design for a "car" or advanced age for a "cat". In this paper, we argue that these variances can be mitigated by predicting composition categories based on salient observation cues. Therefore, we propose Progressive Language-based Observations (PLO), which can automatically determine the order of observation cues. These "observation cues" comprise a series of primitive concepts or graduated descriptions that allow the model to understand image content in a step-by-step manner. Specifically, PLO adopts pre-trained vision-language models (VLMs) to empower the model with observation capabilities.We further devise two variants: a twostep method (PLO-VLM) with a pre-observing classifier dynamically selecting the order of primitive concept-based cues, and a multistep approach (PLO-LLM) using large language models (LLMs) to craft graduated description-based cues. Extensive tests on three datasets show PLO's effectiveness in compositional recognition.