█

AI/TLDR

Buy me a coffee
Buy me a coffee

DVC

Git-like version control for data, models, and ML experiments

Experiment TrackingOpen source
Language
Python
$pip install dvc
IN SHORT

What is DVC?

DVC (Data Version Control) is a command-line tool and VS Code extension for building reproducible machine learning projects. It lets you version data and models by keeping the large files in cloud or network storage while tracking their version info inside your Git repository. It sits in the Experiment Tracking category of the Training & Fine-Tuning tools and is open-source, written in Python.

Overview

DVC (Data Version Control) is a command-line tool and VS Code extension for building reproducible machine learning projects. It lets you version data and models by keeping the large files in cloud or network storage while tracking their version info inside your Git repository.

It is built for ML engineers and data science teams who already use Git for code but need a way to track datasets, model files, pipelines, and experiments. Small placeholder meta-files live in Git, and DVC handles moving the actual data to and from a cache and remote storage.

As an experiment-tracking tool, DVC works locally in your existing Git repo with no separate server. You can define pipelines that rerun only the affected steps, run and compare many experiments by their parameters and metrics, and reproduce anyone's run from the same Git history.

What it does

  • Version datasets and model files in cloud or on-prem storage while keeping version info in Git
  • Define lightweight ML pipelines that rerun only the steps affected by your changes
  • Track experiments locally in your Git repo with no extra servers
  • Compare data, code, parameters, models, and performance plots across runs
  • Push and pull data to remote storage such as S3, Azure, Google Cloud, or SSH
  • Optional VS Code extension to drive DVC from a GUI inside the IDE

Getting started

Install the DVC CLI, initialize it inside an existing Git repository, then start tracking data and running experiments.

Install DVC

Install the CLI from PyPI with pip. DVC is also available via Homebrew, Conda, Chocolatey, and Snap.

bashbash
pip install dvc

Track a dataset

Use git for code and dvc add for data. DVC stores the data in its cache and creates a small meta-file you commit to Git.

bashbash
git add train.py params.yaml
dvc add images/

Define pipeline stages

Connect code and data into a pipeline so DVC knows how outputs are produced from inputs and commands.

bashbash
dvc stage add -n featurize -d images/ -o features/ python featurize.py
dvc stage add -n train -d features/ -d train.py -o model.p -M metrics.json python train.py

Run and compare experiments

Run experiments, then list and apply the one you want. Push data and models to a remote to share them.

bashbash
dvc exp run -n exp-baseline
dvc exp show
dvc remote add myremote -d s3://mybucket/image_cnn
dvc push

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

When to use it

  • Version large training datasets and model files alongside code without bloating the Git repo
  • Build reproducible ML pipelines that rerun only the steps impacted by a change
  • Track and compare many experiments by hyperparameters and metrics in a local Git repo
  • Share data and models with a team via S3, Azure, Google Cloud, or SSH remote storage

How DVC compares

DVC alongside other open-source experiment tracking tools AI/TLDR tracks, ranked by GitHub stars.

ToolStarsWhat it does
MLflow★ 28.9kAn open-source ML and GenAI platform that, alongside experiment tracking, provides evaluation and monitoring tools for scoring and comparing LLM and agent outputs.
DVC★ 16kGit-like version control for data, models, and ML experiments
Weights & Biases (wandb)★ 11.3kAn open-source client library for logging experiments, running hyperparameter sweeps, and versioning models and datasets.
ClearML★ 6.9kAn open-source MLOps suite whose experiment manager auto-logs runs, metrics, and artifacts and ties them into data management and orchestration.
Aim★ 6.3kA self-hosted, open-source experiment tracker that logs and compares thousands of training runs through a fast web UI.
Sacred★ 4.4kA Python tool for configuring, organizing, logging, and reproducing computational research experiments.
Polyaxon★ 3.7kAn open-source platform for tracking, scheduling, and managing large-scale deep learning experiments on Kubernetes.
Trackio★ 1.8kA lightweight open-source experiment tracker from Hugging Face that stores runs locally in SQLite as a drop-in alternative to wandb.