Overview
Roboflow Inference is an open-source server that turns any computer or edge device into a place to run computer vision models. You can self-host your own fine-tuned models or use foundation models such as Florence-2, CLIP, and SAM2, and call them over an HTTP API.
It is aimed at developers who want to deploy vision models without sending images to a third-party cloud. The server pulls the right image for your machine, runs locally (by default on localhost:9001), and supports GPU acceleration when you have a CUDA-enabled GPU.
Within the computer vision space, it fits as the runtime layer. On top of model inference it adds Workflows, composable blocks that let you chain models, mix ML with classic CV (OCR, barcode and QR reading, template matching), and process video streams.
What it does
- Runs your own fine-tuned models plus foundation models like Florence-2, CLIP, and SAM2
- Exposes models over a local HTTP API (default localhost:9001) with a Python SDK
- Workflows: composable blocks to track, count, time, measure, and visualize detections
- Combines ML with traditional CV such as OCR, barcode, QR, and template matching
- Manages cameras and video streams for real-time processing
- GPU acceleration via the NVIDIA Container Toolkit on CUDA-enabled hardware
Getting started
Inference runs as a Docker container that you start with its CLI. Install Docker first (plus the NVIDIA Container Toolkit if you want GPU acceleration), then start the server and connect to it from Python.
Install the CLI and start the server
Install the inference CLI with pip, then start the server in development mode. This pulls the right image for your machine and runs it.
pip install inference-cli && inference server start --devOpen the quickstart notebook
In development mode a Jupyter notebook server with a quickstart guide runs at http://localhost:9001/notebook/start, where you can tour the server's functionality.
Install the Python SDK
To talk to your running server from Python, install the SDK. By default the server listens on localhost:9001.
pip install inference-sdkCommands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Self-host a fine-tuned object detection model and call it over a local API instead of a cloud service
- Build a Workflow that detects, tracks, and counts objects on a live video stream
- Read license plates or barcodes by combining a detection model with OCR in one Workflow
- Run foundation models like CLIP or SAM2 on an edge device for on-site image analysis
How Roboflow Inference compares
Roboflow Inference alongside other open-source vision & understanding tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| OpenCV | ★ 90.9k | The long-standing open-source computer-vision library for C++, Python and Java; version 5 rewrote the DNN engine and added ONNX-based LLM and VLM inference in-library. |
| PaddleOCR | ★ 90.1k | A toolkit for detecting and recognizing text in images across many languages, plus document parsing. |
| Ultralytics YOLO | ★ 61.9k | A framework for training and running YOLO models for real-time object detection, segmentation, and tracking. |
| Supervision | ★ 51k | A Python toolkit for processing, annotating, and visualizing detections and segmentations from many vision models. |
| MediaPipe | ★ 37k | Google's on-device ML framework: ready-to-run vision, text and audio tasks with one cross-platform API for Android, iOS, web, desktop and edge. |
| MMDetection | ★ 33k | An OpenMMLab toolbox with many object detection and instance segmentation algorithms for research and production. |
| vit-pytorch | ★ 25.5k | A single pip package with readable PyTorch implementations of the Vision Transformer and dozens of its research variants, for training image models from scratch. |
| Roboflow Inference | ★ 2.5k | Run computer vision models like YOLO, SAM, and CLIP locally or at the edge |