# vllm-mlx

> vLLM-style inference server for Apple Silicon with OpenAI and Anthropic APIs

- Category: Models & Inference / Local Runtimes
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/waybarrios/vllm-mlx
- Website: https://vllm-mlx.is-a.dev/
- Canonical page: https://ai-tldr.dev/tools/vllm-mlx/
- Updated: 2026-09-17

## In short

vllm-mlx is a vLLM-style inference server for Apple Silicon Macs. It sits in the Local Runtimes category of the Models & Inference tools and is open-source, written in Python and licensed Apache-2.0. To get started: install into a fresh environment.

## Overview

vllm-mlx is a vLLM-style inference server for Apple Silicon Macs. It runs language, vision, audio and embedding models on Metal through Apple's MLX framework, using unified memory with no model conversion step, and exposes them from a single process through both OpenAI-compatible `/v1/*` endpoints (chat completions, completions, embeddings, rerank, responses) and the Anthropic-compatible `/v1/messages` endpoint. Any client that accepts a custom base URL, including the OpenAI and Anthropic SDKs and Claude Code, can point at it.

What sets it apart from running `mlx-lm` directly or using Ollama, in the project's own words, is the serving layer borrowed from vLLM's design: continuous batching for concurrent requests, a paged KV cache with prefix sharing, a trie-based prefix cache, and an SSD-tiered KV cache that spills prefixes to disk for long-context agents. Underneath, it routes requests to mlx-lm for LLMs, mlx-vlm for vision models, mlx-audio for speech and mlx-embeddings for embeddings.

The server also handles tool calling (with 19 parsers covering formats such as OpenAI, Anthropic, Gemini, Qwen, DeepSeek and Gemma), JSON Schema structured output, reasoning extraction for Qwen3 and DeepSeek-R1, speculative decoding for supported models, and Prometheus metrics. The README reports that OpenCode, pi, Codex, Claude Code, GitHub Copilot CLI, Cline CLI and OpenClaw's embedded agent each completed a streamed tool interaction and a file edit against it in a local test run. The project is Apache-2.0 licensed and written in Python; Rapid-MLX began as a community fork of it.

## What it does

- OpenAI-compatible `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/rerank` and `/v1/responses`, plus Anthropic-compatible `/v1/messages` from one process
- Continuous batching, paged KV cache, trie-based prefix cache and an SSD-tiered KV cache (`--ssd-cache-dir`)
- Text, image, video and audio input, native text-to-speech and Whisper-family speech-to-text
- Tool calling with 19 parsers, JSON Schema structured output and reasoning extraction (`--reasoning-parser`)
- Built-in `vllm-mlx bench-serve` benchmarker and a Prometheus `/metrics` endpoint
- `vllm-mlx model inspect / acquire / convert` commands for fetching and converting Hugging Face weights

## Getting started

vllm-mlx requires a Mac with Apple Silicon (M1 to M5) and an arm64-native Python 3.10 or later. The project recommends installing a pinned release into a fresh virtual environment; the README pins 0.4.1 as its reference release.

1. Install into a fresh environment

   Check that Python runs natively (not under Rosetta), then install a pinned release. If you manage CLI tools with uv, `uv tool install 'vllm-mlx==0.4.1'` is the isolated equivalent.

```bash
python3.12 -c 'import platform; print(platform.system(), platform.machine())'
# Expected: Darwin arm64
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install 'vllm-mlx==0.4.1'
```

2. Serve a model

   Models load straight from Hugging Face's mlx-community repos. Once the first response works, restart with `--continuous-batching` to enable batching and its cache options.

```bash
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --host 127.0.0.1 --port 8000
```

3. Call it with the OpenAI SDK

   Install `openai` in your client environment; the server package does not include it.

```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
r = client.chat.completions.create(model="mlx-community/Llama-3.2-3B-Instruct-4bit", messages=[{"role": "user", "content": "Hi!"}])
print(r.choices[0].message.content)
```

4. Point Claude Code at it

   The Anthropic-compatible endpoint lives at the server root, so Anthropic clients use the base URL without `/v1`.

```bash
vllm-mlx serve mlx-community/Qwen3-8B-4bit --port 8000
export ANTHROPIC_BASE_URL=http://localhost:8000
export ANTHROPIC_API_KEY=not-needed
claude
```

5. Benchmark and monitor

   `bench-serve` sweeps prompts at a given concurrency and writes CSV or JSON; `--enable-metrics` exposes Prometheus metrics.

```bash
vllm-mlx bench-serve --url http://localhost:8000 --concurrency 5 --prompts prompts.txt --output results.csv

vllm-mlx serve <model> --enable-metrics
curl http://localhost:8000/metrics
```

## When to use it

- Run Claude Code, Codex, OpenCode or another coding agent against a local model on a Mac
- Serve several concurrent users or agent threads from one Mac with continuous batching and shared prefix caching
- Give existing OpenAI- or Anthropic-SDK code a private local backend by changing only the base URL
- Run vision, speech-to-text, text-to-speech, embeddings and reranking locally through one API server

## Version history

- 2026-09-17 v0.5.0: Added opt-in prefix-trie conversation caching, DeepSeek-V4-Flash support with reasoning and tool calls, and assistant drafting for multimodal models with continuous batching. Structured output now rejects requests whose constraints cannot be enforced.
