Grounded-SAM

// Overview

Grounded-SAM is an open-source pipeline from IDEA-Research that chains two vision models together: Grounding DINO finds objects in an image that match a text prompt, and Segment Anything (SAM) turns each detection into a precise pixel mask. The result is open-vocabulary segmentation — you describe what you want in plain words like "bear" or "the person on the left" and get back boxes and masks for it.

It is aimed at computer vision researchers and engineers who need to label or segment objects without training a model for each new class. Because it works zero-shot from text, it suits open-world tasks where the list of categories is not fixed in advance.

Within the multimodal vision space, Grounded-SAM acts as a building block rather than a single model. Each part can be used on its own or swapped for a similar model (for example replacing Grounding DINO with another detector), and the project ships extra demos that pair it with tools like RAM for automatic labeling and Stable Diffusion for inpainting.

// What it does

• Open-vocabulary detection and segmentation driven by free-text prompts, no per-class training
• Combines Grounding DINO (text-to-box) with Segment Anything (box-to-mask) in one pipeline
• Modular design — each model can run standalone or be replaced with a similar one
• RAM-Grounded-SAM demo for automatic image labeling and annotation pipelines
• EfficientSAM variants (FastSAM, MobileSAM, HQ-SAM) for faster or higher-quality masks
• Also available through Hugging Face Transformers for Grounding DINO and Grounded SAM

// Getting started

Grounded-SAM runs locally with Python 3.8+, PyTorch 1.7+, and torchvision 0.8+. Clone the repo, install the two submodules and dependencies, download the pretrained weights, then run the demo with a text prompt.

git clone https://github.com/IDEA-Research/Grounded-Segment-Anything
cd Grounded-Segment-Anything

python -m pip install -e segment_anything
pip install --no-build-isolation -e GroundingDINO
pip install opencv-python pycocotools matplotlib onnxruntime onnx ipykernel

wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth
wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth

python grounded_sam_demo.py \
  --config GroundingDINO/groundingdino/config/GroundingDINO_SwinT_OGC.py \
  --grounded_checkpoint groundingdino_swint_ogc.pth \
  --sam_checkpoint sam_vit_h_4b8939.pth \
  --input_image assets/demo1.jpg \
  --output_dir "outputs" \
  --text_prompt "bear"

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Auto-labeling image datasets by describing target classes in text instead of drawing masks by hand
• Zero-shot segmentation of objects that were never in a fixed training label set
• Building annotation pipelines, for example pairing RAM-Grounded-SAM to tag and segment images automatically
• Prototyping open-world vision workflows that combine detection, segmentation, and downstream editing or inpainting

// How Grounded-SAM compares

Grounded-SAM alongside other open-source vision & understanding tools AI/TLDR tracks, ranked by GitHub stars.