Overview
LiteRT-LM is Google's orchestration layer for running large language models on top of LiteRT, the on-device runtime that succeeded TensorFlow Lite. Where LiteRT executes a model graph, LiteRT-LM handles everything an LLM needs around it — prompt handling, decoding, KV cache, speculative decoding and tool calling — so an application can hold a conversation rather than run a single inference.
It targets the full spread of edge hardware: Android, iOS, web, desktop and IoT devices such as a Raspberry Pi, with GPU and NPU acceleration where available. Model support spans Gemma, Llama, Phi-4 and Qwen among others, and the runtime handles vision and audio inputs as well as text, plus function calling for agentic workflows. Language bindings are stable for Python and Kotlin, with Swift in early preview, and versioned C API prebuilts landed in v0.16.0 so the runtime can be embedded natively or wrapped in your own bindings without building shared libraries yourself.
This is production code, not a demo: Google states LiteRT-LM powers on-device generative AI in Chrome, Chromebook Plus and Pixel Watch, and it is the engine behind the Google AI Edge Gallery app. Recent releases have added Apple Foundation Framework integration, CLI configuration and JavaScript API updates in v0.15.0, and the experimental YNNPACK delegate for linux arm64 builds in v0.16.0.
What it does
- Cross-platform: Android, iOS, web, desktop and IoT devices including Raspberry Pi
- Hardware acceleration through GPU and NPU delegates
- Multimodal — vision and audio inputs alongside text
- Function calling for agentic workflows on-device
- Broad model support including Gemma, Llama, Phi-4 and Qwen
- Speculative decoding, including multi-token prediction drafters
- Stable Python and Kotlin APIs, Swift in early preview, plus versioned C API prebuilts for native embedding
Getting started
The fastest path is the CLI, which needs no code at all: install it with uv and point it at a model on Hugging Face. From there, move to the Python, Kotlin or Swift guides for embedding it in an app.
Install the CLI
Install the litert-lm tool with uv, then run a model straight from a Hugging Face repository.
uv tool install litert-lm
litert-lm run \
--from-huggingface-repo=google/gemma-3n-E2B-it-litert-lm \
gemma-3n-E2B-it-int4 \
--prompt="What is the capital of France?"Turn on acceleration and speculative decoding
Select a backend and enable speculative decoding for a larger model. This is the README's own example for running Gemma4-E4B with multi-token prediction on Linux, macOS, Windows or a Raspberry Pi.
litert-lm run \
--from-huggingface-repo=litert-community/gemma-4-E4B-it-litert-lm \
gemma-4-E4B-it.litertlm \
--backend=gpu \
--enable-speculative-decoding=true \
--prompt="What is the capital of France?"Embed it in an app
Pick a language guide from the docs: Python for prototyping and scripting, Kotlin for Android and JVM apps, Swift for native iOS and macOS. Each has its own setup guide at ai.google.dev/edge/litert-lm.
Bind it natively
From v0.16.0 there are versioned C API shared-library prebuilts for every supported platform, so you can integrate LiteRT-LM natively or create your own language bindings without building the shared libraries yourself.
See it running first
If you want to try models on a device before writing code, install the Google AI Edge Gallery app from Google Play or the App Store — it runs on LiteRT-LM. There is also a browser chat demo linked from the repository.
Commands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Ship an LLM feature inside a mobile app that keeps working with no network and sends no data off the device
- Run a small multimodal model on a Raspberry Pi or other IoT hardware with GPU or NPU acceleration
- Add on-device function calling so a local model can drive app features rather than just produce text
- Embed edge inference into a native C, Kotlin or Swift codebase using the prebuilt C API libraries
How LiteRT-LM compares
LiteRT-LM alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| Ollama | ★ 181k | A developer-friendly tool that downloads and runs local LLMs from the terminal with a built-in OpenAI-compatible API. |
| llama.cpp | ★ 128k | A C/C++ inference engine that runs LLMs in the GGUF format on CPUs, Apple Silicon, and GPUs with low memory use. |
| GPT4All | ★ 77.4k | GPT4All is a free desktop app and Python client that runs large language models locally on your own computer, with no API calls or GPU required. |
| LocalAI | ★ 49k | A self-hosted server that exposes an OpenAI-compatible API for running text, vision, voice, and image models on local hardware. |
| Jan | ★ 44.4k | An open-source desktop app that runs LLMs fully offline as a ChatGPT-style assistant on your own computer. |
| AirLLM | ★ 34k | A Python inference library that keeps only one transformer layer on the GPU at a time, so a 70B model runs on a single 4GB card and a 671B MoE model on about 12GB, without quantization. |
| Colibrì | ★ 27.3k | A pure-C inference engine that keeps a Mixture-of-Experts model's dense trunk resident in RAM and streams its routed experts from disk, so 744B-2.8T models run on consumer hardware. |
| LiteRT-LM | ★ 6.4k | Google's cross-platform orchestration layer for running LLMs on edge devices with LiteRT |