The manual labelling of high quality datasets for the purposes of training Computer Vision models remains one of the most time consuming tasks for Computer Vision research. Consequently, alternative methods of extracting label information from existing images and visual data is one of the areas of focus for recent research. In this blog we explore one of the state of the art models in pre-training Image Classification models, namely CLIP (Contrastive Language–Image Pre-training). Extracting latent labels from images already associated with text widely available on the internet is a promising method to fast-track the training of Computer Vision models using text and image encoders. These models demonstrate the power of Contrastive Pre-training to perform well with “zero-shot” classification, or classifying images which the model has not been trained on or seen before.

• Introduction
• Background
  • Contrastive Representation Learning
  • NLP and Zero-Shot
  • CLIP
    • Encoders
    • Preparing CLIP for Zero-Shot
• Benchmarks
  • Model Types
  • Pre-training datasets
  • OpenCLIP Benchmarks
  • Our Zero-Shot Datasets
    • Tensorflow Datasets
    • Miniplaces
• Explainability
  • Setup
  • Results
  • Discussion
    • Code
• References
• Appendix

• Introduction
• Background
  • Contrastive Representation Learning
  • NLP and Zero-Shot
  • CLIP
    • Encoders
      • ResNets
      • Vision Transformers
      • Text Encoder
    • Preparing CLIP for Zero-Shot
• Benchmarks
  • Model Types
  • Pre-training datasets
  • OpenCLIP Benchmarks
  • Our Zero-Shot Datasets
    • Tensorflow Datasets
      • Code
    • Miniplaces
      • Results
• Explainability
  • Setup
  • Results
    • Experiment 1a
    • Experiment 1b
    • Experiment 1c
    • Experiment 2a
    • Experiment 2b
    • Experiment 2c
    • Experiment 3a
    • Experiment 3b
    • Experiment 3c
  • Discussion
    • Code
• References
• Appendix

Introduction

Our project aims to examine the robustness and explainability of novel contrastive approaches to pre-training image classification models. While the original CLIP model and associated paper were released in early 2021 by OpenAI, there have already been extensive efforts done by researchers to benchmark the model’s “zero-shot” classification capabilities [3] on standards like ImageNet. Nonetheless we expanded upon the existing benchmarks with more obscure datasets, including the Miniplaces dataset used this class to derive further insights into the model’s capabilities. Moreover, we attempt to visualize the self-attention of the Transformer modules within CLIP models to observe how model robustness is affected by the overall architecture as well as choice of pre-training dataset.

Background

Contrastive Representation Learning

In principle, contrastive representation learning seeks to make machine learning models more closely mimic the way humans learn to categorize the world around them. The standard approach of machine learning is to miminimize a cost function know as “loss” representing how far off a model was from predicting the correct class of input data. With contrastive learning, the loss is calculated both from the how closely the model predicted the correct class but also how “far” it was from predicting incorrect classes.

NLP and Zero-Shot

Rather than relying on a fixed set of class labels, the use of NLP with classification models allows for new labels to be introduced simply by learning a the textual encoding of said labels. This allows for zero-shot transfer learning, where models had the theoretical capability to predict classes on which they had not been trained for previously unseen data. However, the accuracy of the NLP based zero-shot method was poor on benchmark datasets such as ImageNet compared to state of the art (SoTA) methods. While the approach was improved by using a vastly larger datasets during training, training was slow and abandoned full capabilities of zero-shot transfer by restricting outputs to the output labels of the target benchmarks.

CLIP

The Contrastive Language–Image Pre-training approach united contrastive representation learning with the existing zero-shot approach to using NLP to classify images in the form of a joint embedding matrix between text encodings and image encodings [1]. The key advancement was to use the aforementioned encodings to perform contrastive learning rather than the standard prediction where only the correct class is considered. Not only does this approach improve the training efficiency on the ImageNet accuracy benchmark compared to the status quo NLP based models, but it retains the full zero-shot capabilities. During pre-training, the model learns to maximize the similarity scores between correct image and text encodings while maximimizing dissimilirity scores between incorrect pairings:

Figure 1: Summary of Contrastive Language–Image Pre-training (CLIP)

To adopt additional labels unseen during pre-training, all that needs to be done is to learn the encoding of these new potential classes. Then, during the forward pass, the model uses the newly augment set of text label encodings to sythesize a linear classifier which calculates the highest similarity score between input images and said labels. While the length of the text encodings does have a finite length (76 elements in the case of CLIP), this representation still allows for a broad range of labels to be introduced for almost any downstream image classification task.

Figure 2: Zero-shot predictive capabilities of CLIP

Encoders

The Image Encoder can either be a ResNet backbone or a Vision Transformer model. Both will function properly with a given text encoder so long as they extract image features properly and project them into the latent embedding space.

ResNets

ResNet models are deep convolutional neural networks that use residual connections proposed by Kaiming He, which allow deep neural networks to easily learn the identity function and reduces the impact of vanishing gradients [8]. Below is an example of a simple ResNet architecture, where the curved arrows bypassing stacked 3x3 convolutions represent the residual connections.

Figure 3: ResNet18 Architecture overview

Vision Transformers

Vision Transformers take inspiration from the success of Transformer models in NLP tasks [2]. The input image is first divided into small patches of fixed size, which are then fed into a Transformer encoder. The Transformer encoder consists of multiple layers of self-attention and feedforward neural networks, which allow the model to discover the importance of each patch and learn complex representations of the image. Finally, a classification head is added on top of the Transformer encoder to predict the class label of the image.

Figure 4: ViT Architecture overview

Text Encoder

The Text Encoder is a variant of a BERT (Bidirectional Encoder Representations from Transformers) model, specifically BERT-base-uncased. However, discussion of the specific mechanics of the Text Encoder are beyond the scope of this blog.

Preparing CLIP for Zero-Shot

In order to conduct zero-shot classification with a CLIP model, the model must be provided the class names for the target dataset, along with a template to convert the class names into captions. Most of the the time, the template is a basic abstraction of the original label like “A photo of a {classname},” but for other datasets the prompt may be more specific. Below is code (courtesy of LAION’s CLIP_benchmark repository) to retrieve the normalized text embeddings for a given set of classnames:

def zero_shot_classifier(model, tokenizer, classnames, templates, device, amp=True, cupl=False):
    """
    This function returns zero-shot vectors for each class in order
    to use it for zero-shot classification.
    
    model:
        CLIP-like model with `encode_text`
    
    tokenizer:
        text tokenizer, i.e. convert list of strings to torch.Tensor of integers
    
    classnames: list of str
        name of classes
    
    templates: list of str
        templates to use.
    
    Returns
    -------
    
    torch.Tensor of shape (N,C) where N is the number
    of templates, and C is the number of classes.
    """
    autocast = torch.cuda.amp.autocast if amp else suppress
    with torch.no_grad(), autocast():
        zeroshot_weights = []
        for classname in tqdm(classnames):
            if cupl:
                texts = templates[classname]
            else:
                texts = [template.format(c=classname) for template in templates]
            texts = tokenizer(texts).to(device)  # tokenize
            class_embeddings = model.encode_text(texts)
            class_embedding = F.normalize(class_embeddings, dim=-1).mean(dim=0)
            class_embedding /= class_embedding.norm()
            zeroshot_weights.append(class_embedding)
        zeroshot_weights = torch.stack(zeroshot_weights, dim=1).to(device)
    return zeroshot_weights

Then during evaluation, the text encodings (referred to above as “zeroshot_weights”) are multiplied with the image encodings to create probabilities for each class. Below is the code (courtesy of LAION’s CLIP_benchmark repository) to run zero-shot classification on a batch of images:

def run_classification(model, classifier, dataloader, device, amp=True):
    """
    Run zero-shot classifcation
    model: torch.nn.Module
        CLIP-like model with `encode_image` and `encode_text`
    
    classifier: torch.Tensor
        obtained from the function `zero_shot_classifier`
    
    dataloader: torch.utils.data.Dataloader 
    
    Returns
    -------
    (pred, true)  where
        - pred (N, C) are the logits
        - true (N,) are the actual classes
    """
    autocast = torch.cuda.amp.autocast if amp else suppress
    pred = []
    true = []
    nb = 0
    with torch.no_grad():
        for images, target in tqdm(dataloader):
            images = images.to(device)
            target = target.to(device)

            with autocast():
                # predict
                image_features = model.encode_image(images)
                image_features = F.normalize(image_features, dim=-1)
                logits = 100. * image_features @ classifier
            
            true.append(target.cpu())
            pred.append(logits.float().cpu())

    pred = torch.cat(pred)
    true = torch.cat(true)
    return pred, true

Benchmarks

Although OpenAI released their pre-trained CLIP models to the public along with the paper publishing their findings [1], they did not publish the dataset they used to pre-train the model itself. In an effort to make public a dataset comparable to the one used by OpenAI originally, the LAION (Large-scale Artificial Intelligence Open Network) team created their own datasets of varying sizes and pretrained all the different CLIP model architectures themselves. In this section we will discuss the results from the LAION clip-benchmark repository in addition to our own benchmarks results on other datasets, including Miniplaces.

Model Types

The model types relevant to the discussion of CLIP benchmarks are the Vision Transformer (ViT) based models, which both in the original CLIP paper and LAION benchmarks achieve higher zero-shot accuracy across the board compared to ResNet based models. The relevant models along their trainable parameter counts are:

• ViT-B/16
  • 177.35 million params (87.85 million image encoder params)
• ViT-B/32
  • 151.28 million params (86.19 million image encoder params)
• ViT-L/14
  • 427.62 million params (303.97 million image encoder params)
• ViT-H/14
  • 986.11 million params (632.08 million image encoder params)
• ViT-g/14
  • 1366.68 million params (1012.65 million image encoder params)

Here “B” refers to the “baseline” CLIP models with the smallest number of total parameters across the Image and Text encoders, while “L”, “H”, and “g” are shorthand “large,” “huge,” and “giant” CLIP models. The last number denotes the patch size used for the ViT encoder in pixels (e.g. ViT-L/14 uses a patch size of 14x14 pixels). The standard image size for most image benchmarks and past image classifcation models has been 224x224 px, which is evenly divisible by all the patch sizes above. However, there is also a version of the ViT-L/14 model which requires 336x336 input images that’s included in our analysis.

Pre-training datasets

Although the OpenAI models were trained on an a dataset never fully disclosed, the original paper details how the dataset consisted of 400 million image and text pairs scraped from the internet, typically images with associated captions [1]. The LAION created datasets of various sizes seeking to create an open source replication of the results from OpenAI’s CLIP models. We’ll explore the models trained on LAION 400m, a dataset of 400 million text image pairs [5], as well as LAION 2b with 2 billion pairs.

Note that while ViT-g/14 is the largest the researchers at LAION ran into errors during training which forced them to prematurely stop training, leaving the model to have observed a number of images comparable to that of LAION 400m. ViT-L has versions trained on both LAION 400m and LAION 2b, while and ViT-H was trained exclusively on LAION 2b.

OpenCLIP Benchmarks

First let’s briefly examine the OpenCLIP benchmarks results. The first important metric is top1 accuracy, or the percentage of samples where the model’s top prediction was the correct class label.

Figure 4: OpenCLIP Top1 accuracy benchmarks

No one model holds consistent superiority across all the VTAB (Visual Task Adaptation Benchmark) datasets along the x-axis in Figure 4. However, the larger models (ViT-L, ViT-H, and ViT-g) generally perform higher on regular classification tasks. There are some interesting spikes in variance between models in the imagenet-a dataset for instance, where the ViT-L/14 model from OpenAI far exceeds the accuracy of the ViT-L/14 from trained on LAION 2b by nearly 0.3, even though both have the exact same architecture. It even exceeded the accuracy of the ViT-g and ViT-H models, both of which are larger models. ImageNet-A consists of unmodified real-world pictures that are frequently misclassified by standard ResNet models. As both OpenAI’s and LAION’s datasets were scraped from image-text pairs on the internet, we can only speculate that the data used by OpenAI may have been more diverse in including images frequently misclassified by standard CNN models.

(https://paperswithcode.com/dataset/imagenet-a)

Figure 4: OpenCLIP averaged Top1 accuracy across architectures

We can already observe contrasts in general performance for the same classification sub-task. The VTAB/flowers dataset (known better as Oxford Flowers102) and cars dataset (Stanford Cars) both test intra-class differentiation, where objects sharing a greater category (flowers and cars) must be further differentiated (flower species and car make/model/year). Here the CLIP models perform fairly well, reaching over ~70% accuracy for the Flowers102 dataset and over ~80% on the Stanford Cars dataset averaged across all models. While these metrics pale in comparison to standard classification SoTA is 99.76% for Flowers102 and 96.32% for Stanford Cars, they’re competive for zero-shot. The single zero-shot data point on Papers with Code is for VAE-GAN model at 70.8% top1 accuracy. ViT-H/14 from LAION reaches 80.2% top1 accuracy, making it SoTA on Papers with Code. The Stanford Cars doesn’t have a current leaderboard on Papers with Code, but the 80%+ zero-shot top1 accuracy surpases the SoTA for few-shot classification on Papers with Code, which is currently only 73.15%.

The zero-shot accuracies of the aforementioned datasets compete with or surpass the SoTA for zero-shot and few-shot learning and are within a reasonable margin of models trained on the dataset itself.

Lastly, all CLIP models struggle on the dsprites datasets, all failing to surpass 10% accuracy. However, this is not surprising considering that dataset is position and pose focused and is a nontraditional classification task.

Generally speaking, CLIP performs very well on regular zero-shot classification tasks like ImageNet and CIFAR, as well as intra-class differentiation when there are significant physical difference between classes like with cars. It performs less than optimal on non-traditional classification tasks like predicting sprite coordinates or character recognition.

Our Zero-Shot Datasets

Tensorflow Datasets

The original CLIP paper includes a suite of benchmarks of their pretrained models on various zero shot tasks. We performed a similar benchmark analysis across a different set of datasets provided by the Tensorflow Datsets package. Our benchmarking differs both by including a wider diversity of datasets to test robustness, as well as including all pretrained Laion models for comparison. We make changes to key metrics from OpenCLIP that are better suited for comparing performance across datasets. In addition, to accomplish a wide breadth of datasets on a lower computational budget, we selected datasets that were smaller in size but retained much of the attributes of larger datasets:

The above histogram is similar to the one from the OpenCLIP benchmark results, although using completely different datsets. We make one key change to the y-axis, which uses acc1_adjusted. acc1_adjusted takes into account the random-chance accuracies of multi-class classifcation. For example, a random model on 5-class task could achieve 20% accuracy on average. As our datasets have category counts from 2-200, it is important to adjust for random chance. The computation of acc1_adjusted is simply:

$$acc_{adjusted}\ =\ acc\ -\ \frac{1}{N}$$

Where N is the number of categories. This metric enables us to better compare performance across classification tasks and avoid bias towards binary classification.

We included the imagenette and imagewang datasets as a control datasets to ensure our results align with the ones produced by OpenCLIP. Notably, imagenette and imagewang are subsets of the original imagenet and only include 10 categories. Similar to OpenCLIP’s benchmarks, all CLIP models excel at at the imagenet tasks, acheiving >95% accuracy. This both confirms clip models ability to handle variations in imagenet tasks, but also the validity of our own benchmarks.

We can better explore CLIP’s performance on high-level tasks by grouping datsets into broad types:

As a result of the subtraction in acc1_adjusted, we gain the benefit of having bars that go in the negative direction, which indicate that the model performs worse than random chance. Further exploration of the tasks where negative performance occur reveals they tend have labels that are more scientific or esoteric. For example, the casava dataset contains 4 labels: cbsd, cgm, cmd, healthy, depicting different diseases inflicting a plant in the image. Besides healthy, all of these terms are abbreviations of diseases. The negative performance is likely a result of CLIP’s dependency on the semantic meaning of labels. As these labels have no common sensical meaning, CLIP appears to have inferred counterproductive patterns from the abbreviations, resulting in negative performance. For example, cmd may be interpreted as the word command rather than the actual medical term Cassava Mosaic Disease in this context. Cassava belongs in the medical category, and we can see that similarly for plant and typography based tasks, CLIP has difficulty capturing the particular labeling scheme and domain knowledge of the task, resulting in poor or negative performance.

Code

A more detailed view of our zero-shot benchmarking can be found in this colab. A full table of results are available in tables 1 and 2 in the appendix.

Miniplaces

We also decided it would be interesting to conduct some zero-shot analysis on Miniplaces, the toy dataset used for the class classification competition created originally in part by Dr. Zhou at MIT [7].

We constructed our notebook using the zero-shot classifier setup from CLIP_benchmark repository from LAION. We also download the Miniplaces dataset from the google drive link in the assignments and perform the necessary unzipping setup. The template we used to convert the labels into captions was “A photo of a {classname}”. After encoding the class captions, we ran zero-shot classification on the Miniplaces test set and submitted to the Kaggle competition to get top1 accuracy results.

Results

The zero-shot accuracies above validate previous findings that CLIP models with ResNet based image encoders, even deep ResNets like ResNet101, fail to achieve nearly as high zero-shot accuracies as models with ViT image encoders. In the context of the class competition, which was to train on Miniplaces to evaluate standard classification accuracy, the ResNet models are still impressive, with the ResNet50 based model matching the highest standard classification accuracy and the ResNet101 surpassing all submissions. However, they’re clearly not as versatile as ViT-L/14. One possible explanation for the relatively poor ResNet CLIP performance is that those architectures were designed to be good feature extractors for a finite number of classes, where the number of convolutional filters particularly in deeper layers is partially influenced by the final number of target classes. On the other hand the patchification and subsequent tokenization of images in transformer encoders are known to be better at extracting global image features and capturing long-range dependencies.

Explainability

Now we conduct an analysis of the self-attention across different CLIP architectures and pre-trainings. Explainability is an important part of modern deep learning research, as the community moves away from treating models solely as black boxes. Similar to the class activation mappings done for ResNets in class, attention visualization attempts to visualize the sections of the image the model considers important for a given class label.

Setup

We base our notebooks off the CLIP-explainability notebook [6] from Chefer et al’s paper [5] and corresponding repository Transformer-MM-Explainability. We modified it to include images of our own choosing to probe specificity and limits of CLIP’s visual and text encoders. While the original notebook only used ViT models from OpenAI, we also included a variety of models from OpenCLIP to allow for a more complete analysis.

Results

Experiment 1a

Prompt: “A photo of a pair of elephants”

Model	Original/Attention
OpenAI ViT-B/16
LAION ViT-B/16-400m
OpenAI ViT-B/32
LAION ViT-B/32-400m
LAION ViT-B/32-2b
OpenAI ViT-L/14
LAION ViT-L/14-400m

Here we can see the model architecture affects the attention the most. The models with patch size 16px (ViT-B/16) from both OpenAI and LAION highlight the attention on the left elephant, although not the right one.

The attention is about the same for the OpenAI B/32 model, focusing on the left elephant. The LAION ViT-B/32-400m model struggles, focusing on a patch of the lake with very weak attention on the left elephant. Interestingly, the LAION ViT-B/32-2b model is the only model whose main attention hotspot contains both elephants. This implies the text encoding for the word “pair” had more correspondance with the ViT model when pretrained on a larger dataset.

The attention results for the ViT-L/14 models are consistent between OpenAI and LAION.

Now let’s see what happens when we change the prompt.

Experiment 1b

Prompt: “A photo of a pond”

Model	Original/Attention
OpenAI ViT-B/16
LAION ViT-B/16-400m
OpenAI ViT-B/32
LAION ViT-B/32-400m
LAION ViT-B/32-2b
OpenAI ViT-L/14
LAION ViT-L/14-400m

There are some more interesting differences here. The ViT-B/32-2b from LAION once again appears to have the most accurate attention, with the hotspot existing across the whole pond fairly continuously. The ViT-B/32 from OpenAI is a close second, also being fairly uniform and spanning the whole pond. The ViT-B/32-400m struggles once again with the hotspot in a single location in the pond matching its attention for the elephant prompt, suggesting the image encoding for this image is poor given that the prompt didn’t change its attention much.

It’s also interesting to see the ViT-L/14 from LAION perform the worst here, with weak attention across the pond and two random hotspots in the sky. The fact that this particular ViT-L/14, the ViT-B/16, and the ViT-B/32 specified as 400m were all pretrained on the same dataset, yet have such drastically different results, suggests architecture does matter significantly even when dealing with datasets with 400 million image-text pairs. Even that large a dataset can be considered relatively small in the context of CLIP models.

Experiment 1c

Prompt: “A photo of a group of ducks”

Model	Original/Attention
OpenAI ViT-B/16
LAION ViT-B/16-400m
OpenAI ViT-B/32
LAION ViT-B/32-400m
LAION ViT-B/32-2b
OpenAI ViT-L/14
LAION ViT-L/14-400m

All of the models perform surprisingly well here considering the ducks themselves are tiny relative to the size of the photo. Perhaps the lack of detail due to the ducks being small and low resolution are outweighed by the context of them floating in the pond. The only model which perform mediocre are the ViT-B/32 from OpenAI which misses the two left duck groups, and the ViT-B/32-400m from LAION which does the same and has the same attention artifact at the bottom of the image as the first two prompts. The two left groups are in the area of the water where the trees are being reflected, so the water is green rather than blue. Perhaps the poorly performing models missed the context of ducks in non-blue water during their pre-training.

Experiment 2a

Prompt: “sushi rolls”

Model	Original/Attention
OpenAI ViT-B/16
LAION ViT-B/16-400m
OpenAI ViT-B/32
LAION ViT-B/32-400m
LAION ViT-B/32-2b
OpenAI ViT-L/14
LAION ViT-L/14-400m

All models do fairly well here, with the majority of the attention hotspot focusing on the sushi rolls at the bottom left of the photo. One potential reason for the consistency is that food, particularly sushi, is generally well photographed and published on social media, perhaps increasing the representation of sushi-like photos in the pre-training datasets. The models do well despite the fact that we didn’t include the prefix “A photo of” in the prompt, demonstrating the CLIP text encoder’s capability to perform without caption-like text encodings.

For some reason, the LAION ViT-L/14 has many small attention artifacts throughout the image. There’s no direct explanation that can be derived from this, other than the fact that no attention mechanism is foolproof. Maybe the larger model size and increased number of tranformer models caused the LAION ViT-L/14 to learn incorrect context associations, such as identifying white areas (the table) with sushi.

Let’s see how the attention shifts with other prompts:

Experiment 2b

Prompt: “an egg”

Model	Original/Attention
OpenAI ViT-B/16
LAION ViT-B/16-400m
OpenAI ViT-B/32
LAION ViT-B/32-400m
LAION ViT-B/32-2b
OpenAI ViT-L/14
LAION ViT-L/14-400m

This prompt leads to more differentiation between models. The ViT-B/32 from OpenAI is particulaly bad here, with attention hotspots on the soy sauce container and table larger than the egg at the bottom right of the photo. In contrast, the LAION ViT-L/14 was nearly perfect, with attention almost exclusively on the egg. Another interesting observation is the fact the ViT-B/32-2b has attention artifacts on the sushi and potatos on the left of the image despite having been trained on a larger dataset.

Experiment 2c

Prompt: “sriracha”

Model	Original/Attention
OpenAI ViT-B/16
LAION ViT-B/16-400m
OpenAI ViT-B/32
LAION ViT-B/32-400m
LAION ViT-B/32-2b
OpenAI ViT-L/14
LAION ViT-L/14-400m

Here the CLIP demonstrates its ability to learn the meaning of more obscure words. Six out of seven models have a large attention hotspot on the sriracha container at the top of the image. While sriracha is a distinctly red sauce, the lighting in the photo changes the color to be a darker red. Moreover, the other red areas of the image near the meat on the bottom right are not positively identified as sriracha. We can infer that sriracha was a image-text pair probably included in all pretraining datasets, because its meaning can’t be derived from other english words; its originally a Thai word.

The ViT-B/32 models continue to have the largest attention hotspots, which makes intuitive sense because of its larger patch size and thus lower level of granularity; a positive attention hit on a patch will occupy at minimum the entirety of the 32x32 patch.

The poor performance of LAION ViT-L/14 on this prompt despite being pretrained on the same 400 million text image pairs are the smaller ViT-B models suggests that larger CLIP models may require a larger pretraining dataset or longer training to perform well in all situations.

Let’s examine another image.

Experiment 3a

Prompt: “a photo of a coke bottle”

Model	Original/Attention
OpenAI ViT-B/16
LAION ViT-B/16-400m
OpenAI ViT-B/32
LAION ViT-B/32-400m
LAION ViT-B/32-2b
OpenAI ViT-L/14
LAION ViT-L/14-400m

The results here demonstrate the power of having a larger pretraining dataset. The LAION ViT-B/32-2b has the best attention heatmap focusing almost entirely on the word “Coca” in the Coca-Cola logo on the bottle. While the ViT-B/32 from OpenAI also has a strong emphasis on the logo, it also has attention distributed around the whole image, including the hand and random patches of the background. The LAION ViT-B/32-400m continues to have attention hotspot artifacts in places irrelevant to the prompt, in this case in the sky. Lastly, the LAION ViT-L/14-400m performs the worse once again, despite having the largest image encoding ViT.

Experiment 3b

Prompt: “a photo of the ocean”

Model	Original/Attention
OpenAI ViT-B/16
LAION ViT-B/16-400m
OpenAI ViT-B/32
LAION ViT-B/32-400m
LAION ViT-B/32-2b
OpenAI ViT-L/14
LAION ViT-L/14-400m

Here we test the models’ ability to positively assign attention to the background of the image which is also partially out of focus. The ViT-B/32 models all perform the best, even when considering LAION ViT-B/32-400m’s persistent visual artifact in the sky. The larger patch size appears to be better at synthesizing more global information. It’s also interesting that LAION ViT-B/32-400m has attention on the ocean when it was unable to do so for the pond. In this case, we can’t determine the reason for this. It could be a difference in word understanding for “ocean” and “pond” in the text encoder, the particular images chosen, or another reason.

Experiment 3c

Prompt: “a photo of a hand”

Model	Original/Attention
OpenAI ViT-B/16
LAION ViT-B/16-400m
OpenAI ViT-B/32
LAION ViT-B/32-400m
LAION ViT-B/32-2b
OpenAI ViT-L/14
LAION ViT-L/14-400m

Here we test the ability of CLIP models to see objects through distorted glass. All models are able to achieve some level of attention localization on the hand at the bottom of the image. In line with previous results, the ViT-B/32 models from both organizations are the best at the broad strokes heatmaps, with the hotspot covering the entire hand. Where the LAION ViT-L/14-400m struggled before with this image, it now succeeds in localizing the hand. The inconsistent results for the same model, same image, with different prompts, suggests that further training on a larger dataset would benefit this model.

Discussion

This method of attention visualization for the purposes of explainability is useful to observe qualitative differences between models. We observed how some models consistently included visual attention artifacts across multiple images. We also saw how the patch size impacted the downstream attention heatmap, with larger patch sizes having larger “blobs” of attention. We also saw how smaller patch sizes allow attention to be more localized for smaller objects in an image. Finally, even our simple attention visualization comparison was able to reveal which models had shortcomings in their trainings, despite the fact they scores similarly on standard benchmarks across model sizes. The ViT-L/14 model trained on 400 million paris from LAION either performed very well or very poorly, suggesting the dataset size wasn’t enought to make the larger model robust. This is supported by the fact the LAION ViT-B/32-2b consistently outperformed the LAION ViT-B/32-400m.

For CLIP based models, patch size may be adjusted to control the granularity of details the model is better at finding, whereas model size (mainly related to Vision Transformer size) can generally improve model performance but requires more data to be robust. Perhaps in the future, attention heatmap precision and consistency may become a new benchmark for the quality of CLIP and other largely ViT based models.

Architectural extensions of CLIP are already being rapidly explored, including CoCa (Contrastive Captioners) which use a unimodal and multimodal text decoder [10]. CoCa combines the contrastive learning methods of CLIP with techniques from Generative Adversarial Networks. It also improves up on the contrastive loss of CLIP by also using a captioning loss, for another one of CLIP’s downstream tasks (generative captioning) that was not explored in this blog. The inclusion of both the captioning loss and the contrastive loss diversifies the training process and likely allows the model to extract more semantic meaning from the latent text labels. Needless to say the future of CLIP and it’s extensions continues to be bright.

Code

A full implentation of our explainability analysis can be found at the following colabs:

• ViT/B16 OpenAI
• ViT-B/32 OpenAI
• ViT-L/14 OpenAI
• ViT-B/16-400m LAION
• ViT-B/32-400m LAION
• ViT-B/32-2b LAION
• ViT-L/14 LAION

References

[1] Radford, Alec, et al. “Learning transferable visual models from natural language supervision.” International conference on machine learning. PMLR, 2021.

[2] Dosovitskiy, Alexey, et al. “An image is worth 16x16 words: Transformers for image recognition at scale.” arXiv preprint arXiv:2010.11929 (2020).

[3] LAION-AI. CLIP_benchmark. GitHub, 2021, https://github.com/LAION-AI/CLIP_benchmark.

[4] Schuhmann, Christoph, et al. “LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs.” CoRR, vol. abs/2111.02114, 2021, arXiv:2111.02114.

[5] Chefer, Hila, et al. “Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers.” CoRR, vol. abs/2103.15679, 2021, arXiv:2103.15679.

[6] Chefer, Hila. Transformer-MM-Explainability. GitHub, 2021, https://github.com/hila-chefer/Transformer-MM-Explainability.

[7] CSAILVision. miniplaces. GitHub, 2016, https://github.com/CSAILVision/miniplaces.

[8] He, Kaiming, et al. “Deep Residual Learning for Image Recognition.” CoRR, vol. abs/1512.03385, 2015, arXiv:1512.03385.

[9] Yu, Jiahui, et al. “Coca: Contrastive captioners are image-text foundation models.” arXiv preprint arXiv:2205.01917 (2022).

Appendix

Table 1: Top1 Accuracy across all tested models (OpenCLIP)

*Note that the term following the model name denotes the pre-training dataset used.
*QuickGELU refers to a fast implementation of GELU (Gaussian Error Linear Unit), an activation function used mainly in Transformer architectures.

Table 2: Mean Accuracy per Class Recall (OpenCLIP)

Table 3: Top1 Accuracy across all tested models (Our TFDS results)

Table 4: Mean Accuracy per Class Recall (Our TFDS results)