Prefect

// Overview

Prefect is a workflow orchestration framework for building data pipelines in Python. You add `@flow` and `@task` decorators to ordinary functions, and Prefect handles scheduling, retries, caching, and event-based automations so a plain script can run as a production workflow.

It is aimed at data and ML teams who already write Python and want their pipelines to recover from failures, run on a schedule, and be observable without rewriting everything in a separate DSL. Workflow runs are tracked in a UI that you can host yourself with a self-hosted Prefect server or use through the managed Prefect Cloud dashboard.

Within the compute orchestration space, Prefect sits alongside tools like Airflow and Dagster but keeps the authoring experience close to regular Python code, using decorators instead of static DAG definitions.

// What it does

• Define workflows by decorating Python functions with `@flow` and `@task` - no separate DAG syntax
• Built-in scheduling, caching, retries, and event-based automations
• Self-hosted Prefect server with a UI at http://localhost:4200, or managed Prefect Cloud
• Turn any flow into a scheduled deployment with `.serve()` and a cron expression
• Run deployments manually from the UI or CLI, or trigger them in response to events
• Lightweight `prefect-client` package for ephemeral environments talking to Cloud or a remote server

// Getting started

Prefect requires Python 3.10+. Install it, write a flow, then start a server to watch runs in the UI.

pip install -U prefect

from prefect import flow, task
import httpx

@task(log_prints=True)
def get_stars(repo: str):
    url = f"https://api.github.com/repos/{repo}"
    count = httpx.get(url).json()["stargazers_count"]
    print(f"{repo} has {count} stars!")

@flow(name="GitHub Stars")
def github_stars(repos: list[str]):
    for repo in repos:
        get_stars(repo)

if __name__ == "__main__":
    github_stars(["PrefectHQ/prefect"])

prefect server start

if __name__ == "__main__":
    github_stars.serve(
        name="first-deployment",
        cron="* * * * *",
        parameters={"repos": ["PrefectHQ/prefect"]}
    )

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Schedule recurring ETL or data ingestion jobs to run every minute, hour, or day
• Add retries and error handling to flaky pipeline steps that hit external APIs
• Orchestrate multi-step ML pipelines and monitor their runs in a single UI
• Trigger workflows in response to events instead of fixed schedules

// How Prefect compares

Prefect alongside other open-source gpu & compute clouds tools AI/TLDR tracks, ranked by GitHub stars.