Categories 14
Models & Inference
Local RuntimesModels & Inference · 24 tools
Ollama180kA developer-friendly tool that downloads and runs local LLMs from the terminal with a built-in OpenAI-compatible API.ollama/ollama ↗details →
llama.cpp127kA C/C++ inference engine that runs LLMs in the GGUF format on CPUs, Apple Silicon, and GPUs with low memory use.ggml-org/llama.cpp ↗details →
GPT4All77.4kGPT4All is a free desktop app and Python client that runs large language models locally on your own computer, with no API calls or GPU required.nomic-ai/gpt4all ↗details →
LocalAI49kA self-hosted server that exposes an OpenAI-compatible API for running text, vision, voice, and image models on local hardware.mudler/LocalAI ↗details →
Jan44.4kAn open-source desktop app that runs LLMs fully offline as a ChatGPT-style assistant on your own computer.janhq/jan ↗details →
AirLLM33.8kA Python inference library that keeps only one transformer layer on the GPU at a time, so a 70B model runs on a single 4GB card and a 671B MoE model on about 12GB, without quantization.lyogavin/airllm ↗details →
llamafile25.9kA Mozilla project that packages a model and its runtime into one executable file you can copy and run on any OS.mozilla-ai/llamafile ↗details →
MLC LLM23.1kA machine-learning compiler that builds and runs LLMs across browsers, phones, and desktops using TVM-based code generation.mlc-ai/mlc-llm ↗details →
oMLX21.5kAn LLM inference server for Apple Silicon with continuous batching and tiered RAM/SSD KV caching, managed from the macOS menu bar.jundot/omlx ↗details →
KTransformers19.5kA framework for running large Mixture-of-Experts models locally by splitting work between CPU and GPU to fit limited VRAM.kvcache-ai/ktransformers ↗details →
ggml15.3kThe low-level C tensor library that powers llama.cpp and whisper.cpp for running models efficiently on local hardware.ggml-org/ggml ↗details →
mistral-inference10.8kMistral AI's minimal official library for running Mistral open-weight models, with CLI demo and chat commands plus a Python API for text and multimodal inference.mistralai/mistral-inference ↗details →
PowerInfer9.8kA local inference engine that speeds up LLM serving on consumer GPUs by keeping frequently used neurons on the GPU.Tiiny-AI/PowerInfer ↗details →
GenieX8.4kQualcomm's on-device inference runtime for Snapdragon devices: run GGUF models or precompiled AI Hub bundles on the Hexagon NPU, Adreno GPU, or CPU from a CLI, Python, Kotlin, Docker, or an OpenAI-compatible server.qualcomm/GenieX ↗details →
TurboFieldfare6.7kA Swift and Metal runtime that keeps a Mixture-of-Experts model's shared core in memory and streams its routed experts from SSD, running Gemma 4 26B-A4B in about 2 GB of RAM on Apple Silicon.drumih/turbo-fieldfare ↗details →
ExLlamaV24.6kA fast local inference library for running quantized LLMs in the EXL2 format on consumer NVIDIA GPUs.turboderp-org/exllamav2 ↗details →
LiteRT3.4kGoogle's on-device runtime and successor to TensorFlow Lite, running .tflite and .litertlm models with GPU and NPU acceleration on Android, iOS, desktop, web and IoT.google-ai-edge/LiteRT ↗details →
h3.c2.6kA C and Metal inference engine that runs the MiniMax H3 multimodal model natively on Apple Silicon, generating video and audio from a prompt in a single binary.antirez/h3.c ↗details →
WARP (SQLite AI)2.4kA dependency-free C inference engine (formerly WASTE) that keeps a model's dense trunk in RAM and streams routed Mixture-of-Experts weights from NVMe, so trillion-parameter models run on a laptop.sqliteai/warp ↗details →
Nativ1.4kA native macOS app that bundles an mlx-vlm server: chat with MLX models from your Hugging Face cache, watch tokens per second, and serve OpenAI- and Anthropic-compatible endpoints locally.Blaizzy/nativ ↗details →
ExLlamaV31.3kThe next-generation ExLlama runtime that adds the EXL3 quantization format for faster local LLM inference on GPUs.turboderp-org/exllamav3 ↗details →
Swiftlet625A Swift and Metal runtime that streams Mixture-of-Experts weights from storage so 35B and 80B Qwen models run on Apple Silicon Macs and on iPhones with little RAM.leonickson1/Swiftlet ↗details →
Open-source and commercial AI tools, grouped by what they do — filter by access (open source, freemium, commercial, enterprise) with the chips above. Open-source projects link to a detail page with a plain-English overview and a getting-started guide; their star counts are pulled live from GitHub and refreshed every few hours.