Nexa SDK

// Overview

Nexa SDK is an on-device inference framework for running language and multimodal AI models locally. It targets CPU, GPU, and Qualcomm NPU hardware and runs across Android, Windows, and Linux devices. Models load from GGUF and the project's own NEXA format.

It is aimed at developers who want to run models on the device itself rather than calling a hosted API—useful when you care about latency, offline operation, or keeping data on the machine. You can drive it from a command-line tool, a Python package, or an Android SDK.

As a local runtime, it sits alongside other on-device options but adds NPU support and a model range that goes past chat: it also covers ASR, OCR, reranking, object detection, image generation, and embeddings.

// What it does

• Runs models on CPU, GPU, and Qualcomm NPU across Android, Windows, and Linux
• Three ways to integrate: a CLI, a Python SDK (pip install nexaai), and an Android SDK
• Supports many model types: LLM, multimodal, ASR, OCR, rerank, object detection, image generation, and embeddings
• Loads models in GGUF and the project's NEXA format
• Streaming text generation via generate_stream in Python and Flow-based streaming on Android
• Multimodal chat from the CLI, including dragging images directly into the prompt

// Getting started

You can start from the Python SDK for a quick hello-world, or use the CLI to chat with a model in one command.

pip install nexaai

from nexaai import LLM, GenerationConfig, ModelConfig, LlmChatMessage

llm = LLM.from_(model="NexaAI/Qwen3-0.6B-GGUF", config=ModelConfig())

conversation = [
    LlmChatMessage(role="user", content="Hello, tell me a joke")
]
prompt = llm.apply_chat_template(conversation)
for token in llm.generate_stream(prompt, GenerationConfig(max_tokens=100)):
    print(token, end="", flush=True)

# Chat with Qwen3
nexa infer ggml-org/Qwen3-1.7B-GGUF

# Multimodal: drag images into the CLI
nexa infer NexaAI/Qwen3-VL-4B-Instruct-GGUF

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Add an offline chat or assistant feature to a desktop or mobile app without a hosted API
• Run on-device multimodal tasks such as OCR, ASR, or vision-language chat on laptops or phones
• Take advantage of a Qualcomm Snapdragon NPU for low-power local inference on Windows arm64 or Android
• Build embeddings, reranking, or object detection into a local pipeline that keeps data on the device

// How Nexa SDK compares

Nexa SDK alongside other open-source local runtimes tools AI/TLDR tracks, ranked by GitHub stars.