Overview
NeMo Curator is an NVIDIA toolkit that helps ML engineers and data teams build repeatable data pipelines for AI training. It loads, filters, deduplicates, and transforms large datasets across four modalities: text, image, video, and audio. The same pipeline can run on a single laptop or scale out across a multi-node Ray cluster.
It is aimed at teams who need production-grade curation rather than one-off notebooks or ad-hoc scripts. It leans on GPU and distributed execution for data-heavy stages like deduplication, classification, embedding, and inference, using NVIDIA RAPIDS (cuDF, cuML, cuGraph) together with Ray.
As a synthetic-data and labeling tool, it provides modality-aware building blocks plus recipes that map to NVIDIA training workflows such as Nemotron and Nemotron-CC, including LLM-based synthetic data generation.
What it does
- Four modalities in one framework: text, image, video, and audio curation
- Text operations include deduplication, classification, quality filtering, and language detection
- GPU and multi-node execution via NVIDIA RAPIDS (cuDF, cuML, cuGraph) and Ray
- Image operations such as aesthetic filtering, NSFW detection, embedding generation, and deduplication
- Video and audio pipelines for scene detection, clip extraction, ASR transcription, and WER filtering
- Recipes that reproduce NVIDIA training workflows like Nemotron and Nemotron-CC
Getting started
NeMo Curator uses the uv package manager. Install uv once, then pick the path that matches your hardware. The CPU smoke test below needs no GPU.
Install uv
NeMo Curator uses uv for installation. Install it once on your machine.
curl -LsSf https://astral.sh/uv/install.sh | shCPU smoke test (no GPU required)
Create a virtual environment, install the CPU text extra, and verify the import.
uv venv && source .venv/bin/activate
uv pip install "nemo-curator[text_cpu]"
python -c "import nemo_curator; print(nemo_curator.__version__)"GPU text pipeline (CUDA 12, Linux)
On a supported Linux box with a CUDA 12 driver, install the CUDA extra and run the bundled quickstart, which starts Ray and runs a GPU sentiment classification pipeline.
uv venv && source .venv/bin/activate
uv pip install "nemo-curator[text_cuda12]"
python tutorials/quickstart.pyDocker for video and audio
Video and audio pipelines need system codec libraries, so the published NGC container is recommended. See the Installation Guide for setup.
Commands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Curating large web datasets (quality filtering, deduplication, domain classification) to pre-train an LLM
- Scaling fuzzy and exact deduplication across multiple GPUs instead of slow CPU-based jobs
- Reproducing NVIDIA Nemotron-CC style pipelines, including LLM-based synthetic data generation
- Building image, video, or audio training sets with modality-specific filtering and embedding steps
How NeMo Curator compares
NeMo Curator alongside other open-source synthetic data tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| Easy Dataset | ★ 15k | An application that parses your documents and converts them into structured question-answer datasets for fine-tuning, RAG, and model evaluation. |
| SDV (Synthetic Data Vault) | ★ 3.6k | A Python library that learns patterns from real tabular data and generates realistic synthetic tables. |
| distilabel | ★ 3.4k | A framework for building synthetic-data and AI-feedback pipelines that call any LLM to generate and judge datasets. |
| NeMo Data Designer | ★ 2.4k | An NVIDIA library for orchestrating LLM-based synthetic-data pipelines with validation and quality scoring. |
| Augmentoolkit | ★ 1.9k | A tool that turns raw documents into synthetic instruction and conversation datasets for fine-tuning models. |
| NeMo Curator | ★ 1.8k | GPU-accelerated pipelines to clean, dedupe, and curate training data at scale |
| Curator | ★ 1.8k | A Python library for generating and curating large synthetic datasets to train and post-train language models. |
| ydata-synthetic | ★ 1.7k | A collection of generators for creating synthetic tabular and time-series data from real datasets. |