OpenCLIP

// Overview

OpenCLIP is an open-source implementation of OpenAI's CLIP, the contrastive language-image pre-training method that learns a shared embedding space for pictures and text. Because images and captions land in the same space, you can classify an image against arbitrary label strings without training a classifier, or search a picture library with a sentence — the two are the same dot product.

The project is as much a model zoo as a codebase. The maintainers have trained models across a wide range of data sources and compute budgets, including runs on LAION-400M, LAION-2B and DataComp-1B, and the README publishes the zero-shot ImageNet-1k accuracy of each. Third-party families such as SigLIP, SigLIP2, DFN and Perception Encoder checkpoints load through the same API, and model cards live on the Hugging Face Hub under the `open_clip` library tag.

Beyond plain CLIP, the repository covers a widening set of contrastive and generative multimodal objectives: CoCa, MaMMUT, NaFlex variable-resolution image towers, and CLAP-style audio-text training. The `main` branch carries a refactored training stack organised around task wrappers, dict-based batches and FSDP2; the release-stable training API remains available on the `v3` branch and the 3.x releases on PyPI.

// What it does

• One API for loading dozens of pretrained image-text checkpoints, including CLIP, SigLIP/SigLIP2, DFN and Perception Encoder weights
• Zero-shot image classification and image-text retrieval with no task-specific training
• A full training stack for contrastive pre-training, with DDP or FSDP2, torch.compile strategies and WebDataset pipelines
• Reproducible scaling-law experiments — the published models document training data, resolution, samples seen and zero-shot accuracy
• Additional objectives beyond CLIP: CoCa and MaMMUT captioning, NaFlex variable-aspect towers, and CLAP audio-text training
• Checkpoints published on the Hugging Face Hub with per-model cards

// Getting started

Install the PyPI package, load a pretrained model with its preprocessing transform and tokenizer, then encode an image and a few candidate labels. The example below is the README's usage snippet.

pip install open_clip_torch

import torch
from PIL import Image
import open_clip

model, _, preprocess = open_clip.create_model_and_transforms('ViT-B-32', pretrained='laion2b_s34b_b79k')
model.eval()
tokenizer = open_clip.get_tokenizer('ViT-B-32')

image = preprocess(Image.open("docs/CLIP.png")).unsqueeze(0)
text = tokenizer(["a diagram", "a dog", "a cat"])

with torch.no_grad(), torch.autocast("cuda"):
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)

    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs)

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Classify images against label sets that change at runtime, without collecting data and training a classifier
• Build image search over a photo or product catalogue where the query is a sentence
• Generate image embeddings to feed a vector database for multimodal retrieval
• Pre-train or fine-tune your own contrastive image-text model on a domain-specific dataset

// How OpenCLIP compares

OpenCLIP alongside other open-source embedding models & inference tools AI/TLDR tracks, ranked by GitHub stars.