# Pathway

> Python ETL and RAG pipelines that stay live as the data changes

- Category: Retrieval & Knowledge (RAG) / RAG Frameworks & Platforms
- Language: Python, Rust
- Licence: BSL 1.1 (free for non-commercial use and most commercial purposes; converts to Apache 2.0 after 4 years)
- Repository: https://github.com/pathwaycom/pathway
- Website: https://pathway.com
- Canonical page: https://ai-tldr.dev/tools/pathway/

## In short

Pathway is Python ETL and RAG pipelines that stay live as the data changes. It sits in the RAG Frameworks & Platforms category of the Retrieval & Knowledge (RAG) tools and is open-source, written in Python, Rust and licensed BSL 1.1 (free for non-commercial use and most commercial purposes; converts to Apache 2.0 after 4 years).

## Overview

Pathway is a Python framework for stream processing, real-time analytics, LLM pipelines and RAG. You write ordinary Python — schemas, filters, joins, reducers — and the framework runs it as an incremental computation, so results are recomputed only where the input changed instead of being rebuilt from scratch on a schedule.

The Python API sits on top of a Rust engine based on Differential Dataflow, which handles multithreading, multiprocessing and distributed computation. The same code runs in both batch and streaming mode, so a pipeline developed against local files or a CI fixture can be pointed at a live Kafka topic without a rewrite. Pipeline state is held in memory, with persistence available so a pipeline can restart after a crash or an update, and the engine handles late and out-of-order records by updating results when the data arrives.

For AI work, Pathway ships an LLM extension (the "LLM xpack") with wrappers, parsers, embedders and splitters, an in-memory real-time vector index, and integrations with LangChain and LlamaIndex — which is what makes it a RAG option rather than only a data-engineering one: the index tracks your documents as they change instead of being rebuilt by a nightly job. Connectors cover Kafka, Google Drive, PostgreSQL and SharePoint, plus an Airbyte connector for 300+ more sources and a Python connector for anything custom.

## What it does

- One codebase for batch and streaming — the same pipeline runs on local files, CI fixtures, stream replays, and live topics
- Rust engine based on Differential Dataflow doing incremental computation, with multithreading, multiprocessing and distributed execution
- Stateful transformations implemented in Rust — joins, windowing, sorting — alongside arbitrary Python functions and libraries
- LLM xpack: LLM wrappers, parsers, embedders, splitters and an in-memory real-time vector index, with LangChain and LlamaIndex integrations
- Connectors for Kafka, Google Drive, PostgreSQL and SharePoint, an Airbyte connector for 300+ sources, and a custom Python connector
- Persistence to save computation state and restart a pipeline after a crash or update, plus a monitoring dashboard for connector throughput and latency
- Deploys as a plain Python process, a Docker image (pathwaycom/pathway), or on Kubernetes

## Getting started

Pathway needs Python 3.10 or above and runs on macOS and Linux (other systems should use a VM). Install it with pip, describe the pipeline, then call pw.run() to start the computation.

1. Install Pathway

```bash
pip install -U pathway
```

2. Write a pipeline

   Connect to a source, transform the table, write the result, and run. This example from the README sums the positive values arriving in a CSV directory and keeps the output file up to date.

```python
import pathway as pw

# Define the schema of your data (Optional)
class InputSchema(pw.Schema):
  value: int

# Connect to your data using connectors
input_table = pw.io.csv.read(
  "./input/",
  schema=InputSchema
)

# Define your operations on the data
filtered_table = input_table.filter(input_table.value>=0)
result_table = filtered_table.reduce(
  sum_value = pw.reducers.sum(filtered_table.value)
)

# Load your results to external systems
pw.io.jsonlines.write(result_table, "output.jsonl")

# Run the computation
pw.run()
```

3. Run it

   A Pathway project is a normal Python script. The pathway CLI adds a spawn command when you want multiple threads.

```bash
python main.py

# or, with the Pathway CLI and 3 threads
pathway spawn --threads 3 python main.py
```

4. Containerize it

   For single-file projects you can skip the Dockerfile and run the script directly in the official image.

```bash
docker run -it --rm --name my-pathway-app -v "$PWD":/app pathwaycom/pathway:latest python my-pathway-app.py
```

5. Add LLM and RAG tooling

   The LLM xpack adds model wrappers, parsers, embedders, splitters and a real-time vector index on top of the same pipelines. Pathway's documentation carries runnable templates — adaptive RAG, multimodal RAG, and a private RAG stack running on Ollama with Mistral — at pathway.com/developers/user-guide/llm-xpack/overview.

## When to use it

- Reach for it when a RAG index must reflect documents that keep changing — the in-memory vector index updates with the source instead of waiting for a rebuild job.
- Reach for it for real-time ETL and event-driven pipelines over Kafka, object storage or a database, when you want Python rather than Flink or Spark.
- Reach for it when the same transformation has to serve both a historical backfill and a live stream without maintaining two implementations.
- Look elsewhere if you need a batteries-included RAG application with a UI — Pathway is a pipeline framework, not a chat product.
