Kedro

// Overview

Kedro is a toolbox for production-ready data engineering and data science pipelines. It applies software engineering practice — modularity, separation of concerns, testing, linting, documentation — to analytics code, so that pipelines come out reproducible, maintainable and modular instead of accumulating as notebooks, one-off scripts and glue code. It is an open-source Python framework hosted by the LF AI & Data Foundation.

The framework is built around a few pieces that fit together. A project template based on Cookiecutter Data Science gives every project the same shape. The Data Catalog is a set of lightweight connectors for saving and loading data across many file formats and file systems — local and network filesystems, cloud object stores and HDFS — and it includes data and model versioning for file-based systems. The pipeline abstraction resolves dependencies between pure Python functions automatically, and the companion Kedro-Viz project renders the resulting graph.

Kedro is deliberately deployment-agnostic. The same project can run on a single machine or be distributed, with additional support for deploying on Argo, Prefect, Kubeflow, AWS Batch and Databricks. The team frames the motivation as fixing the shortcomings of notebooks and glue code, improving collaboration between team members with different levels of software-engineering exposure, and making analytics code reusable.

// What it does

• Project template based on Cookiecutter Data Science, so every Kedro project has the same structure
• Data Catalog: lightweight connectors for many file formats and file systems, including cloud object stores and HDFS, with data and model versioning for file-based systems
• Pipeline abstraction that resolves dependencies between pure Python functions automatically
• Pipeline visualisation through Kedro-Viz
• Coding-standards support out of the box — pytest, Sphinx docs, ruff linting and the standard Python logging library
• Flexible deployment: single machine or distributed, with support for Argo, Prefect, Kubeflow, AWS Batch and Databricks

// Getting started

Kedro is a Python package on PyPI and conda-forge. The commands below are the ones given in the project README; the official Get Started guide covers virtual-environment setup in full.

uv pip install kedro

conda install -c conda-forge kedro

uv pip install git+https://github.com/kedro-org/kedro@main

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Reach for it when notebooks and one-off scripts have stopped scaling and the pipeline needs to be maintainable
• Reach for it when a team mixes data scientists and engineers and needs one agreed project structure
• Reach for it when the same pipeline has to run locally and on Argo, Prefect, Kubeflow, AWS Batch or Databricks
• Reach for it when reproducibility matters — versioned data and models, and a dependency graph you can visualise

// How Kedro compares

Kedro alongside other open-source gpu & compute clouds tools AI/TLDR tracks, ranked by GitHub stars.