The Birth of Vision Language

摘要

Recent advancements in computer vision and Natural Language Processing (NLP) have transformed how machines interpret visual and textual information. While traditional image processing techniques rely on pixel-based analysis and feature extraction, and NLP leverages self-attention mechanisms for contextual understanding, the potential of viewing images as a language remains largely unexplored. In this work as a proof-of-concept, we introduce a novel ''Vision Language'' framework, transforming image pixels into sequences of alphanumeric characters, effectively treating images as ''text.'' This unique representation enables the direct application of NLP techniques to computer vision tasks, such as classification and segmentation, by utilizing text-based models for image analysis. We demonstrate this approach by adapting text classification methods for image classification on benchmark datasets. Furthermore, we construct a domain-specific medical language from the MedMNIST dataset, spanning 11 medical imaging sub-datasets, illustrating how a unified language can effectively generalize across distinct datasets. We create a broad, generalized language from ImageNet, leveraging approximately 1.3 million images to show that a language derived from extensive image data can simplify and unify image processing across tasks and datasets. Our findings reveal a promising cross-disciplinary approach that bridges computer vision and NLP. Our code is publicly available at https://github.com/mustakinalam/The-Birth-of-Vision-Language.