# OpenCLIP

> Open implementation of CLIP for training and running image-text embeddings

- Category: Retrieval & Knowledge (RAG) / Embedding Models & Inference
- Language: Python
- Licence: MIT
- Repository: https://github.com/mlfoundations/open_clip
- Canonical page: https://ai-tldr.dev/tools/open-clip/

## In short

OpenCLIP is an open-source implementation of OpenAI's CLIP, the contrastive language-image pre-training method that learns a shared embedding space for pictures and text. It sits in the Embedding Models & Inference category of the Retrieval & Knowledge (RAG) tools and is open-source, written in Python and licensed MIT. To get started: install OpenCLIP.

## Overview

OpenCLIP is an open-source implementation of OpenAI's CLIP, the contrastive language-image pre-training method that learns a shared embedding space for pictures and text. Because images and captions land in the same space, you can classify an image against arbitrary label strings without training a classifier, or search a picture library with a sentence — the two are the same dot product.

The project is as much a model zoo as a codebase. The maintainers have trained models across a wide range of data sources and compute budgets, including runs on LAION-400M, LAION-2B and DataComp-1B, and the README publishes the zero-shot ImageNet-1k accuracy of each. Third-party families such as SigLIP, SigLIP2, DFN and Perception Encoder checkpoints load through the same API, and model cards live on the Hugging Face Hub under the `open_clip` library tag.

Beyond plain CLIP, the repository covers a widening set of contrastive and generative multimodal objectives: CoCa, MaMMUT, NaFlex variable-resolution image towers, and CLAP-style audio-text training. The `main` branch carries a refactored training stack organised around task wrappers, dict-based batches and FSDP2; the release-stable training API remains available on the `v3` branch and the 3.x releases on PyPI.

## What it does

- One API for loading dozens of pretrained image-text checkpoints, including CLIP, SigLIP/SigLIP2, DFN and Perception Encoder weights
- Zero-shot image classification and image-text retrieval with no task-specific training
- A full training stack for contrastive pre-training, with DDP or FSDP2, torch.compile strategies and WebDataset pipelines
- Reproducible scaling-law experiments — the published models document training data, resolution, samples seen and zero-shot accuracy
- Additional objectives beyond CLIP: CoCa and MaMMUT captioning, NaFlex variable-aspect towers, and CLAP audio-text training
- Checkpoints published on the Hugging Face Hub with per-model cards

## Getting started

Install the PyPI package, load a pretrained model with its preprocessing transform and tokenizer, then encode an image and a few candidate labels. The example below is the README's usage snippet.

1. Install OpenCLIP

   The package on PyPI is open_clip_torch. It requires torch 2.6 or newer on the current main branch.

```bash
pip install open_clip_torch
```

2. Run zero-shot classification

   create_model_and_transforms returns the model and the image preprocessing pipeline; get_tokenizer returns the matching text tokenizer. Normalize both embeddings before comparing them.

```python
import torch
from PIL import Image
import open_clip

model, _, preprocess = open_clip.create_model_and_transforms('ViT-B-32', pretrained='laion2b_s34b_b79k')
model.eval()
tokenizer = open_clip.get_tokenizer('ViT-B-32')

image = preprocess(Image.open("docs/CLIP.png")).unsqueeze(0)
text = tokenizer(["a diagram", "a dog", "a cat"])

with torch.no_grad(), torch.autocast("cuda"):
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)

    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs)
```

3. Pick a different checkpoint, or train your own

   The full list of pretrained tags is in docs/PRETRAINED.md, with zero-shot results for 38 datasets in docs/openclip_results.csv. For training, note that main uses the post-refactor stack — pin the v3 branch or a 3.x release if you need the older training API.

## When to use it

- Classify images against label sets that change at runtime, without collecting data and training a classifier
- Build image search over a photo or product catalogue where the query is a sentence
- Generate image embeddings to feed a vector database for multimodal retrieval
- Pre-train or fine-tune your own contrastive image-text model on a domain-specific dataset