# Magnitude

> A local inference engine that tunes its kernels to your hardware and serves open models to coding agents

- Category: Models & Inference / Local Runtimes
- Language: Rust
- Licence: Apache-2.0
- Repository: https://github.com/magnitudedev/magnitude
- Website: https://magnitude.dev
- Canonical page: https://ai-tldr.dev/tools/magnitude/
- Updated: 2026-09-30

## In short

Magnitude is an open-source inference engine for running open-weight models locally, built for AI agents rather than chat. Its makers describe the problem it solves this way: engines like vLLM and SGLang are built for batched datacenter serving, while llama.cpp and Ollama trade peak speed for broad compatibility. It sits in the Local Runtimes category of the Models & Inference tools and is open-source, written in Rust and licensed Apache-2.0.

## Overview

Magnitude is an open-source inference engine for running open-weight models locally, built for AI agents rather than chat. Its makers describe the problem it solves this way: engines like vLLM and SGLang are built for batched datacenter serving, while llama.cpp and Ollama trade peak speed for broad compatibility. Magnitude aims at long, concurrent agent sessions on one personal machine, on macOS, Linux and Windows with Apple Silicon, NVIDIA, AMD or CPU-only hardware.

The engine is written in Rust with a custom GPU kernel runtime and autotuner. Kernels are written with flexible parameters that are tuned on your actual device before the model runs, and they are hand-optimised for the most popular open-weight model families. Memory is reserved only for the model weights up front; the heap grows as agent sessions grow and is freed when agents stop. A hybrid paged attention design lets concurrent sessions share prefix caches without slowing a single session down.

Magnitude ships as a desktop app that includes the magnitude CLI. It downloads models from a built-in catalog, starts them on demand when a connected agent needs them, and shuts them down after inactivity. The local API speaks both the OpenAI and the Anthropic formats, and one-click connections exist for Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi and Cline. The project is licensed Apache-2.0 and is backed by Y Combinator (S25 batch).

## What it does

- On-device kernel compilation and tuning for Apple Silicon (Metal), NVIDIA (CUDA), AMD and CPU
- Measured by the makers against llama.cpp on Qwen 3.6 35B A3B: 92% faster decode on an M4 Pro and 19% faster decode on a DGX Spark
- Dynamic memory that grows with agent sessions and is freed when agents stop, with 27% less memory per agent in the makers' tests
- Hybrid paged attention so concurrent agent sessions share prefix caches
- OpenAI-compatible and Anthropic-compatible local API on port 10100
- One-click connections for Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi and Cline
- Desktop app plus a CLI with a headless magnitude serve mode for macOS, Windows and Linux
- Speculative decoding support (DFlash, DSpark and DFlash2)

## Getting started

Magnitude is installed from the desktop app download at magnitude.dev/download; the app includes the magnitude CLI. In the app you pick a model under Discover and connect your agent under Connections. The same steps work from the terminal.

1. Install on Linux

   Download the package for your distribution from magnitude.dev/download, then install it. Ubuntu 22.04, Debian 12 and Fedora/RHEL 9+ with glibc 2.35 or later are supported.

```bash
sudo apt install ./magnitude-desktop.deb
# or on Fedora
sudo dnf install ./magnitude-desktop.rpm
```

2. Check your hardware and pick a model

   Magnitude ranks catalog models that fit your machine, then downloads the one you choose.

```bash
magnitude hardware
magnitude catalog recommendations
magnitude catalog pull <model-id>
```

3. Run it without the desktop window

   magnitude serve runs the service in the foreground; magnitude status shows service and model state.

```bash
magnitude serve
magnitude status
```

4. Connect an agent or call the API

   Configure a supported agent, or point any OpenAI-compatible client at the local endpoint.

```bash
magnitude connections add <harness>
curl http://127.0.0.1:10100/inference/v1/models
```

## When to use it

- Running local coding agents such as Pi, OpenCode or Codex on open-weight models with no token costs
- Running several agent sessions at once on one laptop or workstation while still using the machine for other work
- Getting faster local decode than llama.cpp on Apple Silicon for long-context agent work
- Serving an OpenAI- or Anthropic-compatible endpoint from a headless Linux box with magnitude serve

## Version history

- 2026-09-30 v0.2.0: Magnitude launched on Hacker News and v0.2.0 replaced the llama.cpp-based backend with Magnitude's own self-tuning engine. The release also added the headless magnitude serve command.
