What is NVIDIA VSS Blueprint?
NVIDIA VSS Blueprint is NVIDIA's deployable reference stack for video AI agents that search, summarize, answer questions about and raise verified alerts on live or recorded video. It sits in the Vision & Understanding category of the Multimodal Generation tools and is open-source, written in Python and licensed Apache-2.0 for the repository (Elasticsearch, Kibana and Redis config files dual-licensed SSPLv1 / AGPLv3; data under the NVIDIA Asset License); NIM models and NGC containers carry NVIDIA's own terms, and hosting NIM locally needs an NVIDIA AI Enterprise developer licence.
Overview
The NVIDIA AI Blueprint for Video Search and Summarization (VSS) is a set of reference architectures for building vision agents and video analytics applications. It combines vision language models (VLMs), large language models, retrieval and NVIDIA NIM microservices so that people can search video streams or archives in natural language, summarize hours of footage, ask visual questions, generate reports, and detect and verify real-time alerts. The repository implements the blueprint and powers the hosted demo on build.nvidia.com.
VSS is organised into three layers. Real-time video intelligence extracts features, embeddings and stream understanding from video and publishes the results to a message broker. Downstream analytics turns that metadata into trajectories, incidents and VLM-verified alerts. On top sits an agent that reaches video analytics data, incident records and vision tools through the Model Context Protocol (MCP), including VLM video understanding, embedding-based video search, long-video summarization and snapshot or clip retrieval. The pieces can be used inside existing applications, as standalone microservices, or as part of a larger agent.
It is deployed with Docker Compose developer profiles (base, search, alerts and long-video summarization) or as a Brev Launchable, on NVIDIA hardware ranging from H100, RTX PRO 6000 Blackwell and L40S servers to DGX Spark and Jetson Thor edge devices, which run with a remote LLM. NVIDIA's prerequisites list an 18-core x86 CPU, 128 GB of RAM and a 1 TB SSD as the minimum, plus one to four GPUs depending on the profile. The repository code is Apache-2.0, but deployment pulls models and containers from NVIDIA's NGC registry with an API key, and hosting the NIM microservices locally requires an NVIDIA AI Enterprise developer licence.
What it does
- Reference agent workflows for Q&A and report generation, alert verification, real-time VLM alerts, natural-language video search and long-video summarization
- Real-time video intelligence microservices that publish CV metadata, alerts and embeddings to a message bus for downstream use
- Behavior analytics and alert verification that enrich detections into incidents and use a VLM to cut false positives
- An MCP-based agent with tools for video understanding, semantic search, summarization, clip retrieval and Markdown or PDF reports
- A web UI for uploading videos and chatting with the agent, with intermediate steps and a reasoning trace shown for each answer
- Docker Compose profiles and Helm charts with shared, dedicated or remote LLM and VLM options, plus agentskills.io-compatible skills for deploying and operating VSS
Getting started
The quickstart deploys the base developer profile (video upload, Q&A and report generation) with Docker Compose on Ubuntu 24.04 or a supported NVIDIA edge OS. You need the NVIDIA Container Toolkit, Docker Engine 28.3.3 or later but below 29.5.0, Docker Compose v2.39.1+, the NGC CLI, an NGC API key and a Hugging Face token. If you lack the hardware, NVIDIA also documents a Brev Launchable deployment from a notebook in deploy/docker/scripts.
Clone the repository at a release tag
Git LFS is required for the large files in the repo.
git clone https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization.git
cd video-search-and-summarization
git checkout tags/v3.2.1
git lfs install
git lfs pullSet your credentials
The deploy script reads your NGC and Hugging Face keys from the environment; --help lists the profiles and hardware options.
export NGC_CLI_API_KEY='<your-ngc-api-key>'
export HF_TOKEN='<hf-token>'
deploy/docker/scripts/dev-profile.sh --help
Bring up the base profile for your GPU
Pick the -H value that matches your hardware. On an H100 the LLM and VLM can share one GPU or get one each; set LLM_ENDPOINT_URL and add --use-remote-llm to point at a remote LLM instead (required on DGX Spark and Thor).
# shared GPU
deploy/docker/scripts/dev-profile.sh up -p base -H H100
# dedicated GPUs for LLM and VLM
deploy/docker/scripts/dev-profile.sh up -p base -H H100 --llm-device-id 0 --vlm-device-id 1
# remote LLM
export LLM_ENDPOINT_URL=https://<llm-endpoint>
deploy/docker/scripts/dev-profile.sh up -p base -H H100 --use-remote-llmCheck the agent is healthy, then open the UI
Once the containers are up, the agent answers on port 8000 and the VSS UI is at http://<HOST_IP>:7777/. Drag a video into the chat and ask questions such as "Is the worker wearing PPE?" or "Can you generate a report for this video?".
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
HOST_IP=$(ip route get 1.1.1.1 | awk '{print $7; exit}')
curl -fsS "http://${HOST_IP}:8000/health"
Tear it down
deploy/docker/scripts/dev-profile.sh downCommands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Let operators ask plain-language questions of warehouse, smart-space or SOP-validation footage instead of scrubbing through hours of video
- Cut false positives from computer-vision alerts by having a VLM verify each detection before anyone is paged
- Summarize long recordings into a report by chunking the video and aggregating dense captions
- Start a custom video analytics agent from NVIDIA's reference microservices rather than wiring detection, embeddings, storage and an LLM together from scratch
How NVIDIA VSS Blueprint compares
NVIDIA VSS Blueprint alongside other open-source vision & understanding tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| PaddleOCR | ★ 92.1k | A toolkit for detecting and recognizing text in images across many languages, plus document parsing. |
| OpenCV | ★ 91.4k | The long-standing open-source computer-vision library for C++, Python and Java; version 5 rewrote the DNN engine and added ONNX-based LLM and VLM inference in-library. |
| Ultralytics YOLO | ★ 63k | A framework for training and running YOLO models for real-time object detection, segmentation, and tracking. |
| YOLOv5 | ★ 58.2k | Ultralytics' PyTorch repo for training, running and exporting YOLOv5 object-detection, instance-segmentation and classification models with plain scripts and PyTorch Hub. |
| Supervision | ★ 51.5k | A Python toolkit for processing, annotating, and visualizing detections and segmentations from many vision models. |
| MediaPipe | ★ 37.5k | Google's on-device ML framework: ready-to-run vision, text and audio tasks with one cross-platform API for Android, iOS, web, desktop and edge. |
| MMDetection | ★ 33k | An OpenMMLab toolbox with many object detection and instance segmentation algorithms for research and production. |
| NVIDIA VSS Blueprint | ★ 2k | NVIDIA's deployable reference stack for video AI agents that search, summarize, answer questions about and raise verified alerts on live or recorded video |