# Distributed Llama

> Split one model across the spare machines on your LAN and run it faster

- Category: Models & Inference / Local Runtimes
- Language: C++
- Licence: MIT
- Repository: https://github.com/b4rtaz/distributed-llama
- Canonical page: https://ai-tldr.dev/tools/distributed-llama/

## In short

Distributed Llama takes a single model and spreads it across several ordinary machines on the same network — laptops, mini PCs, a stack of Raspberry Pis — using tensor parallelism with synchronisation over Ethernet. It sits in the Local Runtimes category of the Models & Inference tools and is open-source, written in C++ and licensed MIT.

## Overview

Distributed Llama takes a single model and spreads it across several ordinary machines on the same network — laptops, mini PCs, a stack of Raspberry Pis — using tensor parallelism with synchronisation over Ethernet. Each extra node contributes both memory and arithmetic, so the cluster can hold a model no single box could fit and generate faster than any one of them would alone. It is written in C++ and optimised for ARM and x86_64 AVX2 CPUs, with experimental Vulkan support for GPUs.

The topology is simple. One node is the root: it loads the weights, hands each worker its slice of the network, and synchronises state between steps — while also processing a slice itself. Workers need no model-specific configuration at all, only an address and port. Because the split is by KV head, the node count must be a power of two (1, 2, 4, 8…) and cannot exceed the model's head count, and only two precision combinations are supported: a `q40` model with a `q80` synchronisation buffer, or `f32` with `f32`.

Getting a model running is a single command: `launch.py` downloads and converts the weights and the tokenizer for a named model. Llama 3.1, 3.2 and 3.3, DeepSeek R1 Distill Llama 8B and the Qwen 3 family from 0.6B to 30B-A3B are published as one-command targets, and there is a documented path for converting arbitrary Hugging Face checkpoints. Four commands drive it afterwards: `dllama inference` for a benchmarked run, `dllama chat` for a CLI conversation, `dllama worker` to join a node to the cluster, and `dllama-api` for an HTTP server. The project is MIT-licensed.

## What it does

- Tensor parallelism across 2^n nodes over plain Ethernet, adding RAM and throughput with each device
- Root node distributes weights and synchronises state; workers need only an IP and port, no model configuration
- One-command setup that downloads, converts and launches a named model via `launch.py`
- Llama 3.1/3.2/3.3, DeepSeek R1 Distill Llama 8B and Qwen 3 (0.6B–30B-A3B) as published targets, plus a Hugging Face conversion guide
- CLI inference, CLI chat, worker and HTTP API server modes from the same build
- Runs on Linux, macOS and Windows; optimised for ARM and x86_64 AVX2, with experimental Vulkan GPU support

## Getting started

Python 3 and a C++ compiler are the only prerequisites on the root node. Documented walkthroughs exist for Linux/macOS/Windows, for Raspberry Pi clusters and for GPU (Vulkan) runs.

1. Launch a model on the root node

   The launcher downloads the weights and tokenizer, converts them and starts the run. Pick the target that fits the RAM you have across the cluster.

```bash
python launch.py llama3_1_8b_instruct_q40   # 6.32 GB
python launch.py qwen3_8b_q40               # 6.7 GB
python launch.py llama3_3_70b_instruct_q40  # 40 GB
```

2. Start the workers

   On each other device, run a worker bound to a port. Workers take no model arguments — the root sends them their slice.

```bash
dllama worker --host 0.0.0.0 --port 9999 --nthreads 4
```

3. Point the root at them

   List every worker as `ip:port`. Node count must be a power of two counting the root, and `--max-seq-len` is the main lever for cutting RAM use.

```bash
dllama inference \
  --model dllama_model_meta-llama-3-8b_q40.m \
  --tokenizer dllama_tokenizer_llama3.t \
  --buffer-float-type q80 \
  --workers 10.0.0.2:9999 10.0.0.3:9999 10.0.0.4:9999 \
  --max-seq-len 4096 \
  --nthreads 4 \
  --prompt "Hello World" --steps 256
```

4. Chat, or serve an API

   The same cluster backs an interactive CLI chat or an HTTP endpoint.

```bash
dllama chat --model ... --tokenizer ... --workers ...
dllama-api --model ... --tokenizer ... --workers ... --host 0.0.0.0 --port 9990
```

## When to use it

- Running a 70B model on four mid-range machines instead of buying one large one
- Getting usable token rates out of a Raspberry Pi cluster or a pile of old laptops
- Keeping inference entirely on hardware you own and a LAN you control
- Measuring how tensor parallelism scales across devices, with the built-in benchmark in `dllama inference`