llama.cpp

// Overview

llama.cpp is a plain C/C++ inference engine for running large language models locally and in the cloud. It loads models in the GGUF format and runs them on a wide range of hardware with minimal setup, from a laptop CPU to NVIDIA, AMD, and Apple GPUs.

It is built for developers who want to run open models on their own machines without a heavy Python stack or external dependencies. Integer quantization (from 1.5-bit up to 8-bit) lowers memory use, and CPU+GPU hybrid inference lets you partially accelerate models that are larger than your total VRAM.

As a local runtime in the inference and serving space, llama.cpp gives you both a command-line tool (llama-cli) for one-off prompts and an OpenAI-compatible server (llama-server) you can point existing client code at. It is also the main playground for the underlying ggml library.

// What it does

• Plain C/C++ implementation with no external dependencies
• Runs GGUF models on CPU, Apple Silicon (Metal/NEON/Accelerate), and GPUs via CUDA, HIP, MUSA, Vulkan, and SYCL
• Integer quantization from 1.5-bit to 8-bit for faster inference and reduced memory use
• CPU+GPU hybrid inference to partially accelerate models larger than total VRAM
• Built-in OpenAI-compatible REST API server (llama-server), with multimodal support
• Download and run models directly from Hugging Face with the -hf flag

// Getting started

Install a pre-built binary or build from source, then point llama.cpp at a GGUF model file or a Hugging Face repo.

brew install llama.cpp

# Use a local model file
llama-cli -m my_model.gguf

# Or download and run a model directly from Hugging Face
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

llama-server -hf ggml-org/gemma-3-1b-it-GGUF

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Run open models offline on a laptop or workstation without a Python stack
• Serve a local OpenAI-compatible API for apps and agents during development
• Fit larger models on limited hardware using quantization and CPU+GPU hybrid inference
• Run inference on Apple Silicon or non-NVIDIA GPUs via Metal, Vulkan, HIP, or SYCL

// Version history

Every verified update to llama.cpp that AI/TLDR tracked, newest first — each links to our coverage and the official changeset.

1. 2026-09-14v0.4.1

   Adds architecture support for Maple 20B-A1B (a ternary mixture of experts, on CPU), Tencent Hy 4 as a preview architecture, and Spark2.5. ggml moves to v0.24.0, JSON schema handling is rebuilt around a common_schema representation, and the deprecated --mmap, --mlock, and --direct-io flags are removed in favour of --load-mode.

   our coverage →changeset ↗
2. 2026-09-04v0.4.0

   Adds initial support for the Qwen3.8-Flash-Next (qwen4exp), NVIDIA Nemotron-3-Puzzle-75B-A9B, Nemotron 3.5 DSpark, and nanbeige4.2-3B architectures. A new --lazy-mode flag reads tensors from disk on demand, the server gains per-slot context limits, and ggml moves to 0.23.0 with sparse flash attention and Apple RDMA as an RPC transport.

   our coverage →changeset ↗

// llama.cpp in the news

1. 2026-09-14MAJOROllama 0.34.1 — MLX safetensors leave experimental, GGUF needs llama.cpp
2. 2026-09-14MAJORllama.cpp v0.4.1 — Maple 20B-A1B and Tencent Hy 4 now run locally
3. 2026-09-04MAJORllama.cpp v0.4.0 — Qwen3.8-Flash-Next support and lazy tensor loading
4. 2026-08-11MAJORMetal Capability Shim — llama.cpp runs up to 16x faster inside macOS VMs
5. 2026-04-09MAJORllama.cpp Build b8738 — Vendor-Agnostic Tensor Parallelism, 1-Bit Quantization, AMD CDNA4
6. 2026-04-07NOTABLEOpenVINO 2026.1 — llama.cpp backend for Intel CPUs, GPUs, and NPUs

From the AI/TLDR release feed — every item is source-verified when it ships.

// How llama.cpp compares

llama.cpp alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.