LLM Fundamentals · TRACK 03/06
Transformers & Attention
The architecture behind every modern model, without the math degree.
// THE TRACK
01 · START HERETransformer ModelsGet a working mental model of the transformer — the one architecture behind every model you'll ever call — without the math degree.BEGINNERThe Attention MechanismBuild an intuition for self-attention — queries, keys, and values — and understand the paper title everyone quotes.BEGINNERMixture-of-Experts (MoE)See how MoE models get huge capacity at a fraction of the compute by routing each token to a handful of specialist experts.ADVANCEDFlashAttentionLearn why attention's real bottleneck is memory movement, not math — and how FlashAttention exploits that to speed up every modern model.ADVANCEDPretraining vs Post-TrainingFollow a model from a pile of internet text to a polite assistant, through every stage of the modern training pipeline.INTERMEDIATEEncoder vs Decoder ModelsUnderstand the three transformer body plans and why chat models, embedding models, and translators are built differently.INTERMEDIATEPositional EncodingUnderstand why attention is 'order-blind' by default and how positional encoding tells a transformer which token comes first, second, and last.INTERMEDIATEGrouped-Query AttentionLearn how grouped-query attention shrinks memory and speeds up generation by sharing key/value heads, and why most modern models adopted it.ADVANCEDSelf vs Cross-AttentionClearly separate self-attention (a sequence looking at itself) from cross-attention (one sequence looking at another), and see where each appears in real architectures.ADVANCEDResiduals & Layer NormMeet the two unglamorous components, residual connections and layer normalization, that make it possible to stack 80+ transformer layers without the network falling apart.ADVANCED