Mesh LLM

// Overview

Mesh LLM is a distributed inference runtime that turns several machines into one inference endpoint. Start a node, and it exposes an OpenAI-compatible API at `http://localhost:9337/v1`; add more nodes, and the mesh decides for each request whether the model runs locally, routes to a peer that already hosts it, or is split into layer stages across several boxes. A web console runs alongside the API on port 3131.

Peer traffic — inference requests, responses and split-model activations — travels over QUIC and is end-to-end encrypted, with iroh relays forwarding packets they cannot read. Published meshes advertise themselves through Nostr discovery; private meshes stay invite-token based, joined with `--join <token>`. Operator configuration and inventory actions run on a separate control lane from the public mesh plane, so nodes on different versions stay compatible.

For models too large for any single machine, Mesh LLM uses what the project calls Skippy stage splits: a coordinator plans contiguous layer ranges, starts downstream stages first, waits for readiness, then publishes the stage-0 route. Layer packages contain a `model-package.json` plus GGUF fragments, so each peer downloads only the pieces its assigned stage needs. The CLI also has one-command hand-offs to coding agents — `mesh-llm goose`, `mesh-llm opencode`, `mesh-llm claude` and `mesh-llm pi`.

// What it does

• One OpenAI-compatible `/v1` API per node, with requests routed by the `model` field to whichever peer can serve it
• Single-machine fit first: if one node can host the whole model, it serves locally with no stage traffic
• Skippy stage splits spread a model too large for one box across contiguous layer ranges on several peers
• QUIC end-to-end encryption between nodes, with iroh relays forwarding encrypted packets without reading them
• Public mesh discovery over Nostr, or private invite-token meshes joined with a single flag
• A mixture-of-agents gateway: send `"model": "mesh"` to fan one prompt out to every model in the mesh
• Built-in launchers that point Goose, OpenCode, Claude Code or Pi at the mesh endpoint

// Getting started

Install the release executable, run setup once, then join or start a mesh. Packages, checksums, SBOMs and OCI images are produced by the separate Mesh-LLM/mesh-packaging repository.

curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.sh | bash
# Apple Silicon: brew install Mesh-LLM/tap/mesh-llm

mesh-llm setup

mesh-llm serve --auto

curl -s http://localhost:9337/v1/models | jq '.data[].id'

curl http://localhost:9337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"GLM-4.7-Flash-Q4_K_M","messages":[{"role":"user","content":"hello"}]}'

mesh-llm serve --model Qwen3-8B-Q4_K_M
mesh-llm serve --model Qwen3-8B-Q4_K_M --publish
mesh-llm serve --model hf://meshllm/<repo>@<rev> --split

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Pool the GPUs already sitting in a team's workstations into one shared inference endpoint instead of buying a dedicated server
• Run a model that does not fit on any single machine you own by splitting its layers across several peers
• Give a coding agent a local, OpenAI-compatible endpoint backed by whatever hardware is currently online
• Share compute with a private group over an encrypted, invite-only mesh rather than a public API

// How Mesh LLM compares

Mesh LLM alongside other open-source serving & deployment tools AI/TLDR tracks, ranked by GitHub stars.