Dagster

// Overview

Dagster is an open-source orchestrator for data and machine learning pipelines. Instead of writing tasks, you declare the data assets you want to build (tables, datasets, ML models, reports) as Python functions, and Dagster figures out when to run them and how to keep them up to date.

It is built for data and ML engineers who manage pipelines across the whole development lifecycle, from local development and unit tests through staging and production. The same asset definitions run in each environment, which makes pipelines easier to test and review.

As a compute-orchestration tool, Dagster centralizes pipeline metadata in one place. It tracks lineage between assets, exposes a web UI to inspect runs, and offers observability so you can spot data quality issues and performance problems early.

// What it does

• Declarative asset model: define data and ML outputs as Python functions and let Dagster schedule and materialize them
• Built-in lineage that maps dependencies between assets, with a graph view in the web UI
• Observability, diagnostics, and cataloging from a single control plane
• Designed for the full lifecycle: local dev, unit tests, integration tests, staging, and production
• A growing library of integrations for common data stack tools
• Web UI (dagster-webserver) for inspecting pipelines, runs, and asset status

// Getting started

Dagster runs on Python 3.9 through 3.14 and installs from PyPI. The example below mirrors the quickstart and a minimal asset graph from the project README.

uv add dagster dagster-webserver dagster-dg-cli

import dagster as dg
import pandas as pd

@dg.asset
def country_populations() -> pd.DataFrame:
    df = pd.read_html("https://tinyurl.com/mry64ebh")[0]
    df.columns = ["country", "pop2022", "pop2023", "change", "continent", "region"]
    df["change"] = df["change"].str.rstrip("%").astype("float")
    return df

@dg.asset
def continent_stats(country_populations: pd.DataFrame) -> pd.DataFrame:
    return country_populations.groupby("continent").sum()

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Orchestrate ETL and analytics pipelines that build and refresh data warehouse tables
• Train and update machine learning models on a schedule with tracked dependencies
• Track lineage and observe data quality across a multi-tool data stack
• Develop and unit-test pipelines locally before promoting the same code to production

// How Dagster compares

Dagster alongside other open-source gpu & compute clouds tools AI/TLDR tracks, ranked by GitHub stars.