# LocalAI

> Self-hosted, OpenAI-compatible API for running LLMs, vision, voice, and image models locally

- Category: Models & Inference / Local Runtimes
- Language: Go
- Licence: MIT
- Repository: https://github.com/mudler/LocalAI
- Website: https://localai.io
- Canonical page: https://ai-tldr.dev/tools/localai/

## In short

LocalAI is an open-source inference server you run on your own machines. It sits in the Local Runtimes category of the Models & Inference tools and is open-source, written in Go and licensed MIT. To get started: run the server (CPU only).

## Overview

LocalAI is an open-source inference server you run on your own machines. It exposes a drop-in OpenAI-compatible API (it also supports Anthropic and ElevenLabs API shapes), so existing clients and SDKs can talk to it by changing only the base URL. It can serve text, vision, voice, image, and video models, and it runs on CPU-only hardware as well as NVIDIA, AMD, Intel, Apple Silicon, and Vulkan GPUs.

Rather than shipping one large bundle, LocalAI keeps a small core and pulls each backend on demand. Every backend wraps a focused engine such as llama.cpp, vLLM, whisper.cpp, or stable-diffusion in its own image, so you only download what a given model needs. It detects your GPU capabilities and fetches the matching backend automatically.

It fits the local-runtime category for teams and developers who want to keep data on their own infrastructure. Built-in API key auth, per-user quotas, and role-based access make it usable beyond a single developer, and it ships features like agents with tool use, RAG, and MCP support.

## What it does

- Drop-in OpenAI-compatible API (also Anthropic and ElevenLabs) across every backend
- One API for many modalities: LLMs, vision, voice, image, and video
- Composable backends (llama.cpp, vLLM, whisper.cpp, stable-diffusion, MLX) pulled only when a model needs them
- Runs on CPU-only or NVIDIA, AMD, Intel, Apple Silicon, and Vulkan hardware with automatic backend detection
- Multi-user controls: API key auth, per-user quotas, and role-based access
- Load models from a built-in gallery, Hugging Face, Ollama registry, YAML config, or an OCI registry

## Getting started

The quickest way to start is the official Docker image, then load a model from the gallery and chat with it from a second terminal.

1. Run the server (CPU only)

   Start the container and expose the API on port 8080. For NVIDIA, AMD, Intel, or Vulkan GPUs, use the matching tagged image from the README instead.

```bash
docker run -ti --name local-ai -p 8080:8080 localai/localai:latest
```

2. Load a model

   Pull and run a model from the gallery. You can also load from Hugging Face, the Ollama registry, a YAML config URL, or an OCI registry.

```bash
local-ai run llama-3.2-1b-instruct:q4_k_m
```

3. Chat with the model

   From another shell, open an interactive chat session against the running server. Inside the prompt, /models lists installed models and /model <name> switches between them.

```bash
local-ai chat --model llama-3.2-1b-instruct:q4_k_m
```

## When to use it

- Give an existing OpenAI-SDK app a private, self-hosted backend by changing only the base URL
- Run LLMs, transcription, text-to-speech, and image generation behind a single local API
- Serve models on CPU-only or mixed-vendor GPU hardware where you can't rely on a cloud provider
- Stand up a shared internal inference endpoint with API keys, quotas, and role-based access for a team