Shimmy

// Overview

Shimmy is a local inference server for GGUF models that ships as one binary and speaks the OpenAI API. You point an existing client — an SDK, an editor plugin, anything that already targets `/v1/chat/completions` — at a Shimmy port, and it serves completions from a model file on your own disk instead of a hosted endpoint.

What distinguishes it from the other local runtimes is that the stack is Rust the whole way down. Shimmy is the server; the engine underneath is Airframe, a pure-Rust transformer implementation whose compute shaders are written in WGSL and executed through WebGPU. That means no Python runtime, no C++ toolchain, and no backend flags to pick: the same build targets NVIDIA, AMD, Intel, integrated GPUs and Apple Silicon, because WebGPU is the abstraction layer rather than a vendor SDK. Shimmy 2.0 removed the earlier llama.cpp, MLX, Hugging Face and RustChain backends to make this the only path.

The project leans hard on reproducibility. Model specs are derived from GGUF metadata rather than hardcoded per-model constants, accumulation happens in F32, and the README states that the same model with the same seed and parameters produces the same output. Every supported model/quantisation pair goes through a three-part certification regimen covering maths, inference and determinism before it is listed, and the README is explicit that architecture recognition alone does not imply certification.

It is published on crates.io as `shimmy` and licensed Apache-2.0. The maintainer states in the README that the project is independently maintained and will remain free.

// What it does

• OpenAI-compatible endpoints — chat completions, text completions, streaming and model listing — so existing SDKs and tools work unchanged
• Pure-Rust WebGPU (WGSL) engine via Airframe: no Python runtime, no C++ toolchain, no per-vendor backend selection
• Runs across NVIDIA, AMD, Intel, integrated GPUs and Apple Silicon from one build
• Deterministic output: F32 accumulation, and the same model plus seed plus parameters returns the same result
• TurboShimmy INT4 KV cache, which the docs report cuts KV-cache memory around 7x in tested configurations
• YaRN RoPE scaling for extended context, enabled through the `SHIMMY_MAX_CTX` environment variable
• Model specs auto-derived from GGUF metadata, with a documented certification ledger covering 12 model families

// Getting started

Shimmy installs as a Rust crate and serves a GGUF file you already have on disk. You need a stable Rust toolchain and a GGUF model; the project's quickstart doc covers GPU setup and VRAM sizing in more detail.

cargo install shimmy

shimmy serve --model-path /absolute/path/to/model.gguf --bind 127.0.0.1:11435

shimmy list --short

curl -s http://127.0.0.1:11435/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"tinyllama-1.1b","messages":[{"role":"user","content":"Say hi in 5 words."}],"max_tokens":32}'

SHIMMY_MAX_CTX=16384 shimmy serve --model-path /absolute/path/to/model.gguf

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Run an OpenAI-compatible endpoint on a laptop or workstation when you cannot send prompts to a hosted API
• Deploy local inference onto machines where installing Python or a CUDA toolchain is impractical — the binary has no runtime dependencies
• Get reproducible generations for testing or evaluation, where the same seed and parameters must return the same tokens
• Serve a small model on a constrained GPU using the INT4 KV cache instead of provisioning more VRAM

// How Shimmy compares

Shimmy alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.