xiaozhi-esp32-server

// Overview

xiaozhi-esp32-server is the backend half of the open-source xiaozhi-esp32 project: the ESP32 device handles the microphone, speaker and wake word, and this server does everything after that. It terminates the device connection over WebSocket or MQTT+UDP, runs streaming speech recognition with voice-activity detection, calls a language model, synthesises the reply, and streams the audio back.

Its defining feature is that every stage is a pluggable provider. Speech recognition can run locally with FunASR or SherpaASR, or go out to Xunfei, Tencent, Alibaba, Baidu or OpenAI. The language model can be any OpenAI-compatible endpoint — DeepSeek, Zhipu, Gemini, Alibaba Bailian, Volcano Engine — a local Ollama model, or an application platform such as Dify, FastGPT, Coze or Xinference. Text-to-speech ranges from free options like EdgeTTS through paid vendor APIs to local FishSpeech, GPT-SoVITS, Index-TTS and PaddleSpeech. That matters for hardware projects, where cost per utterance and offline operation usually decide the design.

There are two deployment shapes. The minimal install runs a single agent with file-based storage and no database, needing roughly 2 cores and 2 GB of RAM for API-only providers or 4 GB with local FunASR. The full install adds multi-user and multi-agent management with a web dashboard, wanting about 4 cores and 8 GB with FunASR. Both ship Docker and from-source paths.

// What it does

• Streaming ASR and TTS with voice-activity detection, connected over WebSocket or MQTT+UDP
• Pluggable providers at every stage — local FunASR/SherpaASR or hosted ASR; any OpenAI-compatible, Ollama or platform LLM; free, paid or local TTS
• Voice fingerprint recognition via 3D-Speaker, plus multi-language support
• Memory options including mem0ai, PowerMem and a local short-term store, with a RAGFlow knowledge base
• Tool calling through IOT and MCP protocols as well as custom functions
• Web admin dashboard for devices and agents, in Simplified Chinese, Traditional Chinese and English
• Two deployment tiers: a database-free single-agent install, or a full multi-user install

// Getting started

Pick a deployment tier first — the minimal install is the fastest way to get one device talking, and needs no database. Both tiers are documented for Docker and for running from source in the repository's docs/ directory.

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Reach for it when building a physical voice assistant on ESP32 hardware and you need the server side already solved
• Reach for it when a smart speaker must run entirely on your own network, with local ASR, a local model and local TTS
• Reach for it to swap speech or model vendors without touching firmware, since providers are configuration
• Reach for it when managing several devices or agents at once and you want a dashboard rather than config files

// How xiaozhi-esp32-server compares

xiaozhi-esp32-server alongside other open-source assistants & chatbots tools AI/TLDR tracks, ranked by GitHub stars.