# Dagster

> Build, run, and observe data and ML pipelines as declarative Python assets

- Category: Models & Inference / GPU & Compute Clouds
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/dagster-io/dagster
- Website: https://dagster.io
- Canonical page: https://ai-tldr.dev/tools/dagster/

## In short

Dagster is an open-source orchestrator for data and machine learning pipelines. It sits in the GPU & Compute Clouds category of the Models & Inference tools and is open-source, written in Python and licensed Apache-2.0. To get started: install Dagster.

## Overview

Dagster is an open-source orchestrator for data and machine learning pipelines. Instead of writing tasks, you declare the data assets you want to build (tables, datasets, ML models, reports) as Python functions, and Dagster figures out when to run them and how to keep them up to date.

It is built for data and ML engineers who manage pipelines across the whole development lifecycle, from local development and unit tests through staging and production. The same asset definitions run in each environment, which makes pipelines easier to test and review.

As a compute-orchestration tool, Dagster centralizes pipeline metadata in one place. It tracks lineage between assets, exposes a web UI to inspect runs, and offers observability so you can spot data quality issues and performance problems early.

## What it does

- Declarative asset model: define data and ML outputs as Python functions and let Dagster schedule and materialize them
- Built-in lineage that maps dependencies between assets, with a graph view in the web UI
- Observability, diagnostics, and cataloging from a single control plane
- Designed for the full lifecycle: local dev, unit tests, integration tests, staging, and production
- A growing library of integrations for common data stack tools
- Web UI (dagster-webserver) for inspecting pipelines, runs, and asset status

## Getting started

Dagster runs on Python 3.9 through 3.14 and installs from PyPI. The example below mirrors the quickstart and a minimal asset graph from the project README.

1. Install Dagster

   Add Dagster along with the web UI and CLI packages. The README uses uv.

```bash
uv add dagster dagster-webserver dagster-dg-cli
```

2. Define your assets

   Declare each data asset as a Python function decorated with @dg.asset. Dependencies are passed as function arguments.

```python
import dagster as dg
import pandas as pd

@dg.asset
def country_populations() -> pd.DataFrame:
    df = pd.read_html("https://tinyurl.com/mry64ebh")[0]
    df.columns = ["country", "pop2022", "pop2023", "change", "continent", "region"]
    df["change"] = df["change"].str.rstrip("%").astype("float")
    return df

@dg.asset
def continent_stats(country_populations: pd.DataFrame) -> pd.DataFrame:
    return country_populations.groupby("continent").sum()
```

3. Explore in the web UI

   Open the docs and tutorial to load your assets into Dagster's web UI, where you can view the asset graph, run pipelines, and inspect lineage.

## When to use it

- Orchestrate ETL and analytics pipelines that build and refresh data warehouse tables
- Train and update machine learning models on a schedule with tracked dependencies
- Track lineage and observe data quality across a multi-tool data stack
- Develop and unit-test pipelines locally before promoting the same code to production