Pathway

// Overview

Pathway is a Python framework for stream processing, real-time analytics, LLM pipelines and RAG. You write ordinary Python — schemas, filters, joins, reducers — and the framework runs it as an incremental computation, so results are recomputed only where the input changed instead of being rebuilt from scratch on a schedule.

The Python API sits on top of a Rust engine based on Differential Dataflow, which handles multithreading, multiprocessing and distributed computation. The same code runs in both batch and streaming mode, so a pipeline developed against local files or a CI fixture can be pointed at a live Kafka topic without a rewrite. Pipeline state is held in memory, with persistence available so a pipeline can restart after a crash or an update, and the engine handles late and out-of-order records by updating results when the data arrives.

For AI work, Pathway ships an LLM extension (the "LLM xpack") with wrappers, parsers, embedders and splitters, an in-memory real-time vector index, and integrations with LangChain and LlamaIndex — which is what makes it a RAG option rather than only a data-engineering one: the index tracks your documents as they change instead of being rebuilt by a nightly job. Connectors cover Kafka, Google Drive, PostgreSQL and SharePoint, plus an Airbyte connector for 300+ more sources and a Python connector for anything custom.

// What it does

• One codebase for batch and streaming — the same pipeline runs on local files, CI fixtures, stream replays, and live topics
• Rust engine based on Differential Dataflow doing incremental computation, with multithreading, multiprocessing and distributed execution
• Stateful transformations implemented in Rust — joins, windowing, sorting — alongside arbitrary Python functions and libraries
• LLM xpack: LLM wrappers, parsers, embedders, splitters and an in-memory real-time vector index, with LangChain and LlamaIndex integrations
• Connectors for Kafka, Google Drive, PostgreSQL and SharePoint, an Airbyte connector for 300+ sources, and a custom Python connector
• Persistence to save computation state and restart a pipeline after a crash or update, plus a monitoring dashboard for connector throughput and latency
• Deploys as a plain Python process, a Docker image (pathwaycom/pathway), or on Kubernetes

// Getting started

Pathway needs Python 3.10 or above and runs on macOS and Linux (other systems should use a VM). Install it with pip, describe the pipeline, then call pw.run() to start the computation.

pip install -U pathway

import pathway as pw

# Define the schema of your data (Optional)
class InputSchema(pw.Schema):
  value: int

# Connect to your data using connectors
input_table = pw.io.csv.read(
  "./input/",
  schema=InputSchema
)

# Define your operations on the data
filtered_table = input_table.filter(input_table.value>=0)
result_table = filtered_table.reduce(
  sum_value = pw.reducers.sum(filtered_table.value)
)

# Load your results to external systems
pw.io.jsonlines.write(result_table, "output.jsonl")

# Run the computation
pw.run()

python main.py

# or, with the Pathway CLI and 3 threads
pathway spawn --threads 3 python main.py

docker run -it --rm --name my-pathway-app -v "$PWD":/app pathwaycom/pathway:latest python my-pathway-app.py

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Reach for it when a RAG index must reflect documents that keep changing — the in-memory vector index updates with the source instead of waiting for a rebuild job.
• Reach for it for real-time ETL and event-driven pipelines over Kafka, object storage or a database, when you want Python rather than Flink or Spark.
• Reach for it when the same transformation has to serve both a historical backfill and a live stream without maintaining two implementations.
• Look elsewhere if you need a batteries-included RAG application with a UI — Pathway is a pipeline framework, not a chat product.

// How Pathway compares

Pathway alongside other open-source rag frameworks & platforms tools AI/TLDR tracks, ranked by GitHub stars.