Categories 14
Models & Inference
Local RuntimesModels & Inference · 40 tools
Ollama182kA developer-friendly tool that downloads and runs local LLMs from the terminal with a built-in OpenAI-compatible API.ollama/ollama ↗details →
llama.cpp130kA C/C++ inference engine that runs LLMs in the GGUF format on CPUs, Apple Silicon, and GPUs with low memory use.ggml-org/llama.cpp ↗details →
GPT4All77.4kGPT4All is a free desktop app and Python client that runs large language models locally on your own computer, with no API calls or GPU required.nomic-ai/gpt4all ↗details →
LocalAI49.3kA self-hosted server that exposes an OpenAI-compatible API for running text, vision, voice, and image models on local hardware.mudler/LocalAI ↗details →
Jan44.7kAn open-source desktop app that runs LLMs fully offline as a ChatGPT-style assistant on your own computer.janhq/jan ↗details →
Colibrì37.7kA pure-C inference engine that keeps a Mixture-of-Experts model's dense trunk resident in RAM and streams its routed experts from disk, so 744B-2.8T models run on consumer hardware.JustVugg/colibri ↗details →
llmfit37.2kA Rust terminal tool that inspects your CPU, RAM, GPUs and VRAM and scores which open-weight models and quantizations will actually run well on that machine, with a TUI, CLI, REST API and local-runtime integrations.AlexsJones/llmfit ↗details →
AirLLM35kA Python inference library that keeps only one transformer layer on the GPU at a time, so a 70B model runs on a single 4GB card and a 671B MoE model on about 12GB, without quantization.lyogavin/airllm ↗details →
llamafile26.1kA Mozilla project that packages a model and its runtime into one executable file you can copy and run on any OS.mozilla-ai/llamafile ↗details →
Google AI Edge Gallery24.8kGoogle app for Android, iOS and macOS that runs open models such as Gemma 4 fully on-device — chat, image questions, transcription, agent skills and per-device benchmarks, with no network.google-ai-edge/gallery ↗details →
MLC LLM23.2kA machine-learning compiler that builds and runs LLMs across browsers, phones, and desktops using TVM-based code generation.mlc-ai/mlc-llm ↗details →
DwarfStar22.7kA narrow native inference engine for DeepSeek V4 and GLM 5.x on Metal, CUDA and ROCm, with SSD streaming, tensor parallelism over RDMA and a built-in coding agent.antirez/ds4 ↗details →
oMLX22.3kAn LLM inference server for Apple Silicon with continuous batching and tiered RAM/SSD KV caching, managed from the macOS menu bar.jundot/omlx ↗details →
KTransformers19.5kA framework for running large Mixture-of-Experts models locally by splitting work between CPU and GPU to fit limited VRAM.kvcache-ai/ktransformers ↗details →
ggml15.4kThe low-level C tensor library that powers llama.cpp and whisper.cpp for running models efficiently on local hardware.ggml-org/ggml ↗details →
mistral-inference10.8kMistral AI's minimal official library for running Mistral open-weight models, with CLI demo and chat commands plus a Python API for text and multimodal inference.mistralai/mistral-inference ↗details →
PowerInfer9.8kA local inference engine that speeds up LLM serving on consumer GPUs by keeping frequently used neurons on the GPU.Tiiny-AI/PowerInfer ↗details →
kimi-k3-in-c8.7kA portable C99 inference engine for Kimi K3 that streams a 1.56 TB checkpoint off disk, so the 2.78-trillion-parameter model generates on an 8 GB CPU machine with no BLAS, framework or GPU.FareedKhan-dev/kimi-k3-in-c ↗details →
GenieX8.4kQualcomm's on-device inference runtime for Snapdragon devices: run GGUF models or precompiled AI Hub bundles on the Hexagon NPU, Adreno GPU, or CPU from a CLI, Python, Kotlin, Docker, or an OpenAI-compatible server.qualcomm/GenieX ↗details →
ODS6.8kAn installer that turns a personal Windows, macOS or Linux machine into a private AI server, wiring local model inference, a chat UI, a control dashboard, voice, agents, workflows, RAG and image generation into one Docker stack.Osmantic/ODS ↗details →
TurboFieldfare6.8kA Swift and Metal runtime that keeps a Mixture-of-Experts model's shared core in memory and streams its routed experts from SSD, running Gemma 4 26B-A4B in about 2 GB of RAM on Apple Silicon.drumih/turbo-fieldfare ↗details →
LiteRT-LM6.5kGoogle's LLM orchestration layer on top of LiteRT, running Gemma, Llama, Phi-4 and Qwen on Android, iOS, web, desktop and IoT with GPU/NPU acceleration, multimodal input and tool use.google-ai-edge/LiteRT-LM ↗details →
CactusFREEMIUM6.1kA four-layer on-device inference stack for phones, wearables and robots — its own rotation-based quantizer, ARM NEON kernels, a zero-copy compute graph and an OpenAI-compatible engine with automatic cloud handoff.cactus-compute/cactus ↗details →
Shimmy5.9kA single-binary, OpenAI-compatible inference server for GGUF models, written entirely in Rust on the Airframe WebGPU engine — no Python runtime and no C++ toolchain.Michael-A-Kuykendall/shimmy ↗details →
Lemonade5.8kLocal AI server that runs chat, coding, speech and image models on your own NPU and GPU behind OpenAI-, Anthropic- and Ollama-compatible APIs, with optimizations for AMD Ryzen AI and Radeon hardware.lemonade-sdk/lemonade ↗details →
ExLlamaV24.6kA fast local inference library for running quantized LLMs in the EXL2 format on consumer NVIDIA GPUs.turboderp-org/exllamav2 ↗details →
LiteRT3.4kGoogle's on-device runtime and successor to TensorFlow Lite, running .tflite and .litertlm models with GPU and NPU acceleration on Android, iOS, desktop, web and IoT.google-ai-edge/LiteRT ↗details →
Distributed Llama3.1kShards one model across several home devices over Ethernet using tensor parallelism, so a cluster of laptops or Raspberry Pis contributes both RAM and speed to a single inference run.b4rtaz/distributed-llama ↗details →
h3.c2.8kA C and Metal inference engine that runs the MiniMax H3 multimodal model natively on Apple Silicon, generating video and audio from a prompt in a single binary.antirez/h3.c ↗details →
ort2.5kA Rust binding for Microsoft's ONNX Runtime that loads ONNX models exported from PyTorch, TensorFlow and other frameworks and runs them with hardware acceleration through CUDA, TensorRT, CoreML and other execution providers.pykeio/ort ↗details →
MTPLX2.5kA native Mac app and CLI that runs local LLMs on Apple Silicon using the model's own multi-token prediction heads for exact speculative decoding, with an OpenAI- and Anthropic-compatible local server.youssofal/MTPLX ↗details →
WARP (SQLite AI)2.4kA dependency-free C inference engine (formerly WASTE) that keeps a model's dense trunk in RAM and streams routed Mixture-of-Experts weights from NVMe, so trillion-parameter models run on a laptop.sqliteai/warp ↗details →
edge02.1kA streaming Mixture-of-Experts inference framework that keeps expert weights on SSD and reads them on demand, pairing a trained prerouter that predicts routing a step ahead with Recover-LoRA adapters that recover 4-bit loss.Edge0-AI/Edge0 ↗details →
ExLlamaV31.5kThe next-generation ExLlama runtime that adds the EXL3 quantization format for faster local LLM inference on GPUs.turboderp-org/exllamav3 ↗details →
Nativ1.5kA native macOS app that bundles an mlx-vlm server: chat with MLX models from your Hugging Face cache, watch tokens per second, and serve OpenAI- and Anthropic-compatible endpoints locally.Blaizzy/nativ ↗details →
Swiftlet649A Swift and Metal runtime that streams Mixture-of-Experts weights from storage so 35B and 80B Qwen models run on Apple Silicon Macs and on iPhones with little RAM.leonickson1/Swiftlet ↗details →
Slotstream398A single Swift binary that runs Qwen3.8-Flash-Next on Apple Silicon by streaming MoE experts from SSD, serving the Ollama and OpenAI chat APIs on port 11434.carloslfu/slotstream ↗details →
Desert Ant CoreFREEMIUM352On-device SDKs for iOS, macOS, Android, web and Node that run a set of small single-purpose models — transcription, PII redaction, language ID, clipping — fully offline via Core ML, LiteRT and WebAssembly.Desert-Ant-Labs/desert-ant-core ↗details →
Open-source and commercial AI tools, grouped by what they do — filter by access (open source, freemium, commercial, enterprise) with the chips above. Open-source projects link to a detail page with a plain-English overview and a getting-started guide; their star counts are pulled live from GitHub and refreshed every few hours.