Distributed Llama

// Overview

Distributed Llama takes a single model and spreads it across several ordinary machines on the same network — laptops, mini PCs, a stack of Raspberry Pis — using tensor parallelism with synchronisation over Ethernet. Each extra node contributes both memory and arithmetic, so the cluster can hold a model no single box could fit and generate faster than any one of them would alone. It is written in C++ and optimised for ARM and x86_64 AVX2 CPUs, with experimental Vulkan support for GPUs.

The topology is simple. One node is the root: it loads the weights, hands each worker its slice of the network, and synchronises state between steps — while also processing a slice itself. Workers need no model-specific configuration at all, only an address and port. Because the split is by KV head, the node count must be a power of two (1, 2, 4, 8…) and cannot exceed the model's head count, and only two precision combinations are supported: a `q40` model with a `q80` synchronisation buffer, or `f32` with `f32`.

Getting a model running is a single command: `launch.py` downloads and converts the weights and the tokenizer for a named model. Llama 3.1, 3.2 and 3.3, DeepSeek R1 Distill Llama 8B and the Qwen 3 family from 0.6B to 30B-A3B are published as one-command targets, and there is a documented path for converting arbitrary Hugging Face checkpoints. Four commands drive it afterwards: `dllama inference` for a benchmarked run, `dllama chat` for a CLI conversation, `dllama worker` to join a node to the cluster, and `dllama-api` for an HTTP server. The project is MIT-licensed.

// What it does

• Tensor parallelism across 2^n nodes over plain Ethernet, adding RAM and throughput with each device
• Root node distributes weights and synchronises state; workers need only an IP and port, no model configuration
• One-command setup that downloads, converts and launches a named model via `launch.py`
• Llama 3.1/3.2/3.3, DeepSeek R1 Distill Llama 8B and Qwen 3 (0.6B–30B-A3B) as published targets, plus a Hugging Face conversion guide
• CLI inference, CLI chat, worker and HTTP API server modes from the same build
• Runs on Linux, macOS and Windows; optimised for ARM and x86_64 AVX2, with experimental Vulkan GPU support

// Getting started

Python 3 and a C++ compiler are the only prerequisites on the root node. Documented walkthroughs exist for Linux/macOS/Windows, for Raspberry Pi clusters and for GPU (Vulkan) runs.

python launch.py llama3_1_8b_instruct_q40   # 6.32 GB
python launch.py qwen3_8b_q40               # 6.7 GB
python launch.py llama3_3_70b_instruct_q40  # 40 GB

dllama worker --host 0.0.0.0 --port 9999 --nthreads 4

dllama inference \
  --model dllama_model_meta-llama-3-8b_q40.m \
  --tokenizer dllama_tokenizer_llama3.t \
  --buffer-float-type q80 \
  --workers 10.0.0.2:9999 10.0.0.3:9999 10.0.0.4:9999 \
  --max-seq-len 4096 \
  --nthreads 4 \
  --prompt "Hello World" --steps 256

dllama chat --model ... --tokenizer ... --workers ...
dllama-api --model ... --tokenizer ... --workers ... --host 0.0.0.0 --port 9990

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Running a 70B model on four mid-range machines instead of buying one large one
• Getting usable token rates out of a Raspberry Pi cluster or a pile of old laptops
• Keeping inference entirely on hardware you own and a LAN you control
• Measuring how tensor parallelism scales across devices, with the built-in benchmark in `dllama inference`

// How Distributed Llama compares

Distributed Llama alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.