LiteRT

// Overview

LiteRT is Google's runtime for running machine-learning and generative-AI models directly on a device, and the continuation of TensorFlow Lite under a new name. It takes models exported from PyTorch, TensorFlow or JAX — as `.tflite` for classic models or `.litertlm` for LLMs — and executes them on whatever accelerator the hardware offers, from CPU via XNNPack to GPU via the ML Drift backend to vendor NPUs. Around the runtime sits a small toolchain: LiteRT Torch for conversion, the AI-Edge Quantizer for optimisation, and LiteRT-LM for on-device language models.

The V2 API is where the project's design shows. The Compiled Model API removes explicit delegate wiring in favour of automated accelerator selection, adds true asynchronous execution and efficient I/O buffer handling, and exposes NPUs from several chipset vendors through one consistent interface rather than per-vendor code paths. A tensor-centric C++ Tensor API is available when you need direct control over memory and graph execution.

Platform coverage is broad and explicitly tabulated in the repository: Android (OpenCL/OpenGL GPU, plus Broadcom, Google Tensor, Intel, MediaTek and Qualcomm NPUs), iOS and macOS (Metal), Linux, Windows and IoT (WebGPU), and the browser through LiteRT.js, which runs models client-side over WebGPU and WebAssembly. There is also a LiteRT CLI aimed at coding-agent workflows. The project publishes nightly builds and targets stable releases on a six-to-eight-week cadence, is written mainly in C++, and is Apache-2.0.

// What it does

• Compiled Model API with automated accelerator selection, asynchronous execution and efficient I/O buffer handling — no explicit delegates
• Unified NPU acceleration across Broadcom, Google Tensor, Intel, MediaTek and Qualcomm silicon through one API
• GPU acceleration via ML Drift, with buffer interoperability designed to cut latency between GPU buffer types
• Browser inference through LiteRT.js over WebGPU and WebAssembly
• On-device LLMs and diffusion models through LiteRT-LM, with `.litertlm` bindings for Python, C++, Kotlin, Swift and JS
• Conversion from PyTorch, TensorFlow and JAX via LiteRT Torch, plus the AI-Edge Quantizer for optimised artifacts
• Tensor API — a lightweight tensor-centric C++ library for direct memory and graph control

// Getting started

Which entry point you use depends on the target. The LiteRT CLI is the quickest thing to install locally; mobile and web integration are covered by the platform guides.

uv venv --clear --python=3.13 --seed
source .venv/bin/activate
uv pip install litert-cli-nightly
litert --help

cd docker_build
./build_with_docker.sh

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Reach for it when a model has to run on the user's phone, laptop or browser rather than on a server
• Reach for it when you are migrating an existing TensorFlow Lite deployment and want NPU support without per-vendor code
• Reach for it when you want a quantized LLM or diffusion model running fully on-device via LiteRT-LM
• Reach for it when a web app needs client-side inference over WebGPU without shipping data off the machine

// How LiteRT compares

LiteRT alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.