oMLX

// Overview

oMLX is an inference server for Apple Silicon Macs built on MLX. It serves text LLMs, vision-language models, OCR models, embedding models and rerankers from a directory of downloaded models, and any OpenAI-compatible client can talk to it at http://localhost:8000/v1. The distinguishing feature is its KV cache: oMLX persists cached context across a hot in-memory tier and a cold SSD tier, so past context stays reusable across requests even when the conversation changes mid-flight — the case that makes local models practical for repeated coding-agent turns.

It is managed like a Mac app rather than a daemon you babysit. The macOS build installs a menu-bar app with a welcome flow (model directory, server start, first download), in-app auto-update, and a lightweight omlx CLI shim so terminal commands and Apple Shortcuts can drive the app-managed server. A web admin dashboard at /admin gives real-time monitoring, model management, a chat UI, a benchmark tool and per-model settings, localised into eight languages with all CDN dependencies vendored so it works offline.

The project is Apache-2.0 and installs from a DMG, from Homebrew, or from source. It requires macOS 15.0 or newer, Python 3.11–3.13 and an Apple Silicon chip. Some model families (GLM-5.2, MiniMax M3, Qwen3.5) have native custom Metal kernels that a plain pip install does not build; the README is explicit that the affected families silently fall back to much slower generic paths, so use the official DMG (which ships the kernels precompiled) or build with the Metal toolchain from full Xcode.

// What it does

• Continuous batching plus a two-tier KV cache — hot in memory, cold on SSD — so prior context stays reusable across requests
• OpenAI-compatible endpoint at http://localhost:8000/v1 for any existing client, plus a built-in chat UI at /admin/chat
• Serves text LLMs, vision-language models, OCR models, embeddings and rerankers, auto-discovered from subdirectories of a model folder
• macOS menu-bar app with in-app auto-update, a welcome wizard and an omlx CLI shim for terminal commands and Apple Shortcuts
• Admin dashboard for real-time monitoring, model management, benchmarking and per-model settings, vendored for offline use
• Native Metal custom kernels for specific model families, plus optional MCP support and experimental multi-Mac inference in source builds

// Getting started

Install the Mac app for the simplest path (it ships the precompiled Metal kernels), or use Homebrew if you want the server as a background service. Requires macOS 15.0+, Apple Silicon and Python 3.11–3.13.

brew tap jundot/omlx https://github.com/jundot/omlx
brew install jundot/omlx/omlx
omlx start

git clone https://github.com/jundot/omlx.git
cd omlx
OMLX_WITH_CUSTOM_KERNEL=1 pip install -e .

omlx serve --model-dir ~/models

python -c "from omlx.custom_kernels import native_kernel_status; print(native_kernel_status())"

curl http://localhost:8000/v1/models

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Run a coding agent against a local model on a Mac, where cross-request KV caching is what makes repeated long-context turns bearable
• Serve LLMs, VLMs, embeddings and rerankers for a local app from one OpenAI-compatible endpoint instead of several runtimes
• Keep a couple of everyday models pinned in memory and let heavier ones swap in on demand, controlled from the menu bar
• Benchmark and compare local models on your own hardware from the admin dashboard before committing to one

// How oMLX compares

oMLX alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.