Sebastian Raschka · 2026-11-15 · notable
Sebastian Raschka — understanding positional embeddings in modern LLMs
In short
Sebastian Raschka explains how LLMs encode token position, from GPT-2 learned embeddings and sinusoidal encodings to RoPE with PyTorch code, plus partial RoPE, NoPE and mixed RoPE/NoPE layers in 2026 open-weight models.
Sebastian Raschka · 2026-11-15 · 3 sources checked
Sebastian Raschka walks through how LLMs know where each token sits, with RoPE explained step by step in PyTorch.
What is it?
Positional embeddings are the subject of Sebastian Raschka's 15 November 2026 article. It covers learned absolute embeddings (GPT-2), fixed sinusoidal encodings from the original Transformer, and Rotary Position Embedding (RoPE), with a worked example.
How does it work?
RoPE rotates query and key vectors by an angle that depends on token position, so the attention score depends on the offset between two tokens. The article gives PyTorch code for the rotation tables, split-half and interleaved variants, and use in causal attention.
Why does it matter?
The article also surveys newer designs: partial RoPE, NoPE (no explicit position encoding), mixed RoPE/NoPE layers such as SmolLM3 skipping RoPE in every fourth layer, and learned relative bias in Inkling. RoPE is still the most common choice in 2026 open-weight models.
Who is it for?
Engineers and students learning LLM architecture