# llama.cpp

> Run LLMs in C/C++ on CPU, Apple Silicon, and GPU with low memory use

- Category: Models & Inference / Local Runtimes
- Language: C/C++
- Licence: MIT
- Repository: https://github.com/ggml-org/llama.cpp
- Canonical page: https://ai-tldr.dev/tools/llama-cpp/
- Updated: 2026-09-14

## In short

llama.cpp is a plain C/C++ inference engine for running large language models locally and in the cloud. It loads models in the GGUF format and runs them on a wide range of hardware with minimal setup, from a laptop CPU to NVIDIA, AMD, and Apple GPUs. It sits in the Local Runtimes category of the Models & Inference tools and is open-source, written in C/C++ and licensed MIT.

## Overview

llama.cpp is a plain C/C++ inference engine for running large language models locally and in the cloud. It loads models in the GGUF format and runs them on a wide range of hardware with minimal setup, from a laptop CPU to NVIDIA, AMD, and Apple GPUs.

It is built for developers who want to run open models on their own machines without a heavy Python stack or external dependencies. Integer quantization (from 1.5-bit up to 8-bit) lowers memory use, and CPU+GPU hybrid inference lets you partially accelerate models that are larger than your total VRAM.

As a local runtime in the inference and serving space, llama.cpp gives you both a command-line tool (llama-cli) for one-off prompts and an OpenAI-compatible server (llama-server) you can point existing client code at. It is also the main playground for the underlying ggml library.

## What it does

- Plain C/C++ implementation with no external dependencies
- Runs GGUF models on CPU, Apple Silicon (Metal/NEON/Accelerate), and GPUs via CUDA, HIP, MUSA, Vulkan, and SYCL
- Integer quantization from 1.5-bit to 8-bit for faster inference and reduced memory use
- CPU+GPU hybrid inference to partially accelerate models larger than total VRAM
- Built-in OpenAI-compatible REST API server (llama-server), with multimodal support
- Download and run models directly from Hugging Face with the -hf flag

## Getting started

Install a pre-built binary or build from source, then point llama.cpp at a GGUF model file or a Hugging Face repo.

1. Install llama.cpp

   Install with a package manager (brew, nix, or winget), run it with Docker, download a pre-built binary from the releases page, or build from source. See the project's install and build guides for details.

```bash
brew install llama.cpp
```

2. Run a model from the command line

   Use llama-cli with a local GGUF file, or pass -hf to download and run a model straight from Hugging Face.

```bash
# Use a local model file
llama-cli -m my_model.gguf

# Or download and run a model directly from Hugging Face
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF
```

3. Launch the OpenAI-compatible server

   Start llama-server to expose a REST API that OpenAI-compatible clients can call.

```bash
llama-server -hf ggml-org/gemma-3-1b-it-GGUF
```

## When to use it

- Run open models offline on a laptop or workstation without a Python stack
- Serve a local OpenAI-compatible API for apps and agents during development
- Fit larger models on limited hardware using quantization and CPU+GPU hybrid inference
- Run inference on Apple Silicon or non-NVIDIA GPUs via Metal, Vulkan, HIP, or SYCL

## Version history

- 2026-09-14 v0.4.1: Adds architecture support for Maple 20B-A1B (a ternary mixture of experts, on CPU), Tencent Hy 4 as a preview architecture, and Spark2.5. ggml moves to v0.24.0, JSON schema handling is rebuilt around a common_schema representation, and the deprecated --mmap, --mlock, and --direct-io flags are removed in favour of --load-mode.
- 2026-09-04 v0.4.0: Adds initial support for the Qwen3.8-Flash-Next (qwen4exp), NVIDIA Nemotron-3-Puzzle-75B-A9B, Nemotron 3.5 DSpark, and nanbeige4.2-3B architectures. A new --lazy-mode flag reads tensors from disk on demand, the server gains per-slot context limits, and ggml moves to 0.23.0 with sparse flash attention and Apple RDMA as an RPC transport.