Kestra

// Overview

Kestra is an open-source orchestration platform for data, AI, and infrastructure workflows. You describe each pipeline as declarative YAML, and Kestra runs it on a schedule or in response to events. It brings Infrastructure as Code practices to pipelines you would otherwise wire together by hand.

It is aimed at data engineers, platform teams, and anyone who needs to coordinate scripts, database extracts, API calls, and microservices across systems. Workflows can be built and edited in the browser through a built-in code editor, and the YAML stays in sync whether you change a flow from the UI, an API call, or your Git repository.

Within the workflow-automation and low-code space, Kestra fits teams that want a UI to build and watch workflows but still keep the orchestration logic as version-controlled code. A plugin ecosystem lets it talk to many databases, cloud stores, and APIs, and run scripts in any language.

// What it does

• Declarative YAML interface for defining workflows, kept in sync with any UI, API, or CI/CD change
• Both scheduled and event-driven runs, configured through a simple trigger definition
• Built-in web UI and code editor with syntax highlighting, auto-completion, and real-time validation
• Plugin ecosystem to pull data from databases, cloud storage, and APIs, and run scripts in any language
• Git version control integration so workflows built in the UI can be pushed to a branch
• Workflow structure and resilience: namespaces, subflows, retries, timeouts, error handling, inputs/outputs, and conditional branching

// Getting started

The fastest way to try Kestra locally is to run it with Docker, then open the UI to build your first flow.

docker run --pull=always -it -p 8080:8080 --user=root \
  --name kestra --restart=always \
  -v kestra_data:/app/storage \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /tmp:/tmp \
  kestra/kestra:latest server local

id: hello_world
namespace: company.team
tasks:
  - id: hello
    type: io.kestra.plugin.core.log.Log
    message: Hello, World!

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Schedule recurring data pipelines that extract from databases or cloud storage and run scripts in any language
• Trigger workflows in real time from events such as a new file, an API call, or a message
• Coordinate AI and data-processing steps across multiple services with retries, timeouts, and error handling
• Manage orchestration logic as version-controlled YAML while still building and monitoring flows from a UI

// How Kestra compares

Kestra alongside other open-source workflow automation tools AI/TLDR tracks, ranked by GitHub stars.