# MLC LLM

> Compile and run LLMs natively on any GPU, browser, or phone

- Category: Models & Inference / Local Runtimes
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/mlc-ai/mlc-llm
- Website: https://llm.mlc.ai
- Canonical page: https://ai-tldr.dev/tools/mlc-llm/

## In short

MLC LLM is a machine-learning compiler and deployment engine for large language models. It compiles models down to native code with TVM and runs them on MLCEngine, a single inference engine that targets many platforms from one codebase. It sits in the Local Runtimes category of the Models & Inference tools and is open-source, written in Python and licensed Apache-2.0.

## Overview

MLC LLM is a machine-learning compiler and deployment engine for large language models. It compiles models down to native code with TVM and runs them on MLCEngine, a single inference engine that targets many platforms from one codebase.

It is aimed at developers who need to run LLMs outside a typical cloud GPU server. The same engine and compiler run on NVIDIA, AMD, Apple, and Intel GPUs, in the web browser through WebGPU and WASM, and on iOS and Android devices.

As a local runtime, it exposes an OpenAI-compatible API through a REST server, Python, and JavaScript, so you can swap it in for hosted APIs while keeping inference on the hardware you control.

## What it does

- Runs on NVIDIA, AMD, Apple, and Intel GPUs via CUDA, ROCm, Metal, Vulkan, and OpenCL
- Deploys to the web browser using WebGPU and WASM, plus iOS and Android
- MLCEngine gives one unified inference engine and compiler across every target platform
- OpenAI-compatible API through a REST server, Python, and JavaScript
- Built on Apache TVM machine-learning compilation for native code generation
- Ships a CLI for chat and for launching a local OpenAI-style server

## Getting started

Install the package, then either talk to a model from Python or launch a local OpenAI-compatible server. Models are pulled directly from Hugging Face with the HF:// prefix.

1. Install MLC LLM

   Install the Python package with pip.

```bash
pip install mlc-llm
```

2. Chat with a model from Python

   Create an MLCEngine, stream a chat completion, then terminate the engine. This downloads and runs a quantized Llama 3 8B model (about 6GB of free VRAM recommended).

```python
from mlc_llm import MLCEngine

model = "HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC"
engine = MLCEngine(model)

for response in engine.chat.completions.create(
    messages=[{"role": "user", "content": "What is the meaning of life?"}],
    model=model,
    stream=True,
):
    for choice in response.choices:
        print(choice.delta.content, end="", flush=True)

engine.terminate()
```

3. Or run an OpenAI-compatible server

   Launch a REST server at http://127.0.0.1:8000 that accepts OpenAI-style requests, so existing clients can point at it.

```bash
mlc_llm serve HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC
```

## When to use it

- Run an LLM locally on a laptop or workstation GPU instead of a hosted API
- Ship in-browser AI features that run client-side with WebGPU and WASM
- Embed on-device language models in iOS or Android apps
- Drop in an OpenAI-compatible local endpoint for development and testing