# Shimmy

> A single Rust binary that serves GGUF models on an OpenAI-compatible API

- Category: Models & Inference / Local Runtimes
- Language: Rust
- Licence: Apache-2.0
- Repository: https://github.com/Michael-A-Kuykendall/shimmy
- Canonical page: https://ai-tldr.dev/tools/shimmy/

## In short

Shimmy is a single Rust binary that serves GGUF models on an OpenAI-compatible API. It sits in the Local Runtimes category of the Models & Inference tools and is open-source, written in Rust and licensed Apache-2.0. To get started: install the binary.

## Overview

Shimmy is a local inference server for GGUF models that ships as one binary and speaks the OpenAI API. You point an existing client — an SDK, an editor plugin, anything that already targets `/v1/chat/completions` — at a Shimmy port, and it serves completions from a model file on your own disk instead of a hosted endpoint.

What distinguishes it from the other local runtimes is that the stack is Rust the whole way down. Shimmy is the server; the engine underneath is Airframe, a pure-Rust transformer implementation whose compute shaders are written in WGSL and executed through WebGPU. That means no Python runtime, no C++ toolchain, and no backend flags to pick: the same build targets NVIDIA, AMD, Intel, integrated GPUs and Apple Silicon, because WebGPU is the abstraction layer rather than a vendor SDK. Shimmy 2.0 removed the earlier llama.cpp, MLX, Hugging Face and RustChain backends to make this the only path.

The project leans hard on reproducibility. Model specs are derived from GGUF metadata rather than hardcoded per-model constants, accumulation happens in F32, and the README states that the same model with the same seed and parameters produces the same output. Every supported model/quantisation pair goes through a three-part certification regimen covering maths, inference and determinism before it is listed, and the README is explicit that architecture recognition alone does not imply certification.

It is published on crates.io as `shimmy` and licensed Apache-2.0. The maintainer states in the README that the project is independently maintained and will remain free.

## What it does

- OpenAI-compatible endpoints — chat completions, text completions, streaming and model listing — so existing SDKs and tools work unchanged
- Pure-Rust WebGPU (WGSL) engine via Airframe: no Python runtime, no C++ toolchain, no per-vendor backend selection
- Runs across NVIDIA, AMD, Intel, integrated GPUs and Apple Silicon from one build
- Deterministic output: F32 accumulation, and the same model plus seed plus parameters returns the same result
- TurboShimmy INT4 KV cache, which the docs report cuts KV-cache memory around 7x in tested configurations
- YaRN RoPE scaling for extended context, enabled through the `SHIMMY_MAX_CTX` environment variable
- Model specs auto-derived from GGUF metadata, with a documented certification ledger covering 12 model families

## Getting started

Shimmy installs as a Rust crate and serves a GGUF file you already have on disk. You need a stable Rust toolchain and a GGUF model; the project's quickstart doc covers GPU setup and VRAM sizing in more detail.

1. Install the binary

   Install from crates.io with cargo.

```bash
cargo install shimmy
```

2. Serve a GGUF model

   Point Shimmy at an absolute path to a .gguf file and bind it to a local port.

```bash
shimmy serve --model-path /absolute/path/to/model.gguf --bind 127.0.0.1:11435
```

3. Check what is loaded

   From another terminal, list the models Shimmy is serving.

```bash
shimmy list --short
```

4. Call it like OpenAI

   Send a normal chat-completions request. Any client that speaks that surface can be pointed at the same URL.

```bash
curl -s http://127.0.0.1:11435/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"tinyllama-1.1b","messages":[{"role":"user","content":"Say hi in 5 words."}],"max_tokens":32}'
```

5. Stretch the context window

   Extended context is opt-in through an environment variable that turns on YaRN RoPE scaling. The extended-context doc works through the VRAM arithmetic before you raise it.

```bash
SHIMMY_MAX_CTX=16384 shimmy serve --model-path /absolute/path/to/model.gguf
```

## When to use it

- Run an OpenAI-compatible endpoint on a laptop or workstation when you cannot send prompts to a hosted API
- Deploy local inference onto machines where installing Python or a CUDA toolchain is impractical — the binary has no runtime dependencies
- Get reproducible generations for testing or evaluation, where the same seed and parameters must return the same tokens
- Serve a small model on a constrained GPU using the INT4 KV cache instead of provisioning more VRAM