Finetuning CLIP (Contrastive Language-Image Pre-training)

The CLIP (Contrastive Language-Image Pre-training) model, developed by OpenAI, is a groundbreaking multimodal model that combines knowledge of English-language concepts with semantic knowledge of images. It consists of a text and an image encoder, which encodes textual and visual information into a multimodal embedding space. The model’s architecture aims to increase the cosine similarity score of images and associated text pairs. This is achieved through a contrastive objective, which enhances the efficiency of the model by 4x times.

The CLIP model’s forward pass involves running the input through the text and image encoder network, normalizing the embedded features, and using them as input to compute the cosine similarity. The resulting cosine similarity is then returned as logits.

CLIP’s versatility is evident in its ability to perform tasks such as zero-shot image classification, image generation, abstract task execution for robots, and image captioning. It has also been used for a wide variety of tasks beyond its original use cases, showcasing its adaptability and potential for diverse applications. The model has demonstrated significant flexibility, outperforming the best ImageNet model on various datasets, including tasks such as OCR, geolocalization, and action recognition. However, it has limitations in tasks requiring depth perception, object counting, and distinguishing between similar objects. Despite these limitations, CLIP’s zero-shot accuracy in OCR tasks is notable.