Segment Anything 2 (SAM 2)

// Overview

SAM 2 (Segment Anything Model 2) is a model from Meta's FAIR team for promptable visual segmentation. You give it a prompt — a click, a set of points, or a box — and it returns a mask for the object you pointed at. It extends the original Segment Anything model from single images to video by treating an image as a one-frame video.

The main step beyond SAM is video tracking. SAM 2 uses a transformer with streaming memory, so once you mark an object in one frame it can follow that object across the rest of the clip. The 2.1 checkpoints come in four sizes (tiny, small, base-plus, large) so you can trade accuracy for speed.

It sits in the computer vision space and is meant for developers and researchers who need object masks rather than just bounding boxes. Typical users are people building annotation tools, video editing features, or datasets, and it ships with both an image predictor and a video predictor you call from Python.

// What it does

• Promptable segmentation from clicks, point sets, or boxes on static images
• Object tracking across video frames using a streaming memory architecture
• Multi-object tracking with per-object inference, including adding new objects after tracking has started
• Four SAM 2.1 checkpoint sizes (tiny, small, base-plus, large) to balance speed and accuracy
• torch.compile support for the full model on video for faster VOS inference (vos_optimized=True)
• Separate SAM2ImagePredictor and video predictor APIs, plus released training and fine-tuning code

// Getting started

SAM 2 needs Python 3.10+, PyTorch 2.5.1+, and torchvision 0.20.1+, and is best run on a GPU machine. Clone the repo, install it, download a checkpoint, then call the predictor.

git clone https://github.com/facebookresearch/sam2.git && cd sam2

pip install -e .

cd checkpoints && \
./download_ckpts.sh && \
cd ..

import torch
from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor

checkpoint = "./checkpoints/sam2.1_hiera_large.pt"
model_cfg = "configs/sam2.1/sam2.1_hiera_l.yaml"
predictor = SAM2ImagePredictor(build_sam2(model_cfg, checkpoint))

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    predictor.set_image(<your_image>)
    masks, _, _ = predictor.predict(<input_prompts>)

import torch
from sam2.build_sam import build_sam2_video_predictor

checkpoint = "./checkpoints/sam2.1_hiera_large.pt"
model_cfg = "configs/sam2.1/sam2.1_hiera_l.yaml"
predictor = build_sam2_video_predictor(model_cfg, checkpoint)

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = predictor.init_state(<your_video>)

    # add new prompts and instantly get the output on the same frame
    frame_idx, object_ids, masks = predictor.add_new_points_or_box(state, <your_prompts>)

    # propagate the prompts to get masklets throughout the video
    for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
        ...

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Speed up dataset annotation by letting annotators click an object and get a clean mask instead of drawing by hand
• Track a selected object across the frames of a video for editing, effects, or analysis
• Build interactive segmentation into a tool where a user clicks or drags a box to select things
• Fine-tune the model on a custom domain using the released training code

// How Segment Anything 2 (SAM 2) compares

Segment Anything 2 (SAM 2) alongside other open-source vision & understanding tools AI/TLDR tracks, ranked by GitHub stars.