Local & Open Models · TRACK 04/04
Inference & Serving Engines
vLLM, batching, and the economics of serving your own GPUs.
// THE TRACK
01 · START HEREInference ServersYou will understand what separates a laptop chatbot from production LLM serving and what an inference server actually does.BEGINNERvLLMYou will understand how vLLM's PagedAttention and continuous batching squeeze dramatically more throughput from a single GPU than naive serving.INTERMEDIATEContinuous BatchingYou will understand the scheduling trick that lets a single GPU serve dozens of users at once without making anyone wait.INTERMEDIATEOllama vs vLLMKnow exactly when Ollama's simplicity wins, when vLLM's throughput wins, and how to migrate between them.INTERMEDIATESpeculative Decoding LocallyYou will understand how a tiny draft model lets a large model generate faster without changing its output, and how to turn it on locally.INTERMEDIATEThroughput vs LatencyYou will understand the core trade-off in LLM serving and which knobs to turn when your server is too slow for users or too underused to be cheap.INTERMEDIATESelf-Hosting Cost vs APIYou will be able to do the break-even math that decides whether self-hosting an open model actually saves money versus paying a per-token API.INTERMEDIATESGLangYou will understand what SGLang is, how RadixAttention prefix caching reuses shared prompt prefixes, and why it is a leading alternative to vLLM for serving.ADVANCEDHugging Face TGIYou will understand what Hugging Face's Text Generation Inference (TGI) is, what it was built to do, and why Hugging Face now steers new work toward vLLM and SGLang.INTERMEDIATETensorRT-LLMYou will understand what TensorRT-LLM is, how NVIDIA's compiled CUDA kernels target lowest-latency inference, and the trade-offs of building a model engine for it.ADVANCED