# Sebastian Raschka — understanding positional embeddings in modern LLMs

> Sebastian Raschka walks through how LLMs know where each token sits, with RoPE explained step by step in PyTorch.

- Org: Sebastian Raschka
- Date: 2026-11-15
- Category: article, tutorial
- Source: https://magazine.sebastianraschka.com/p/understanding-positional-embeddings
- Canonical page: https://ai-tldr.dev/releases/sebastian-raschka-positional-embeddings-nov15/

## In short

Sebastian Raschka explains how LLMs encode token position, from GPT-2 learned embeddings and sinusoidal encodings to RoPE with PyTorch code, plus partial RoPE, NoPE and mixed RoPE/NoPE layers in 2026 open-weight models.

## What it is

Positional embeddings are the subject of Sebastian Raschka's 15 November 2026 article. It covers learned absolute embeddings (GPT-2), fixed sinusoidal encodings from the original Transformer, and Rotary Position Embedding (RoPE), with a worked example.

## How it works

RoPE rotates query and key vectors by an angle that depends on token position, so the attention score depends on the offset between two tokens. The article gives PyTorch code for the rotation tables, split-half and interleaved variants, and use in causal attention.

## Why it matters

The article also surveys newer designs: partial RoPE, NoPE (no explicit position encoding), mixed RoPE/NoPE layers such as SmolLM3 skipping RoPE in every fourth layer, and learned relative bias in Inkling. RoPE is still the most common choice in 2026 open-weight models.

## Who it's for

Engineers and students learning LLM architecture

## Links

- [Full Article](https://magazine.sebastianraschka.com/p/understanding-positional-embeddings)
- [Hacker News](https://news.ycombinator.com/item?id=50506376)
- [LLMs From Scratch](https://github.com/rasbt/LLMs-from-scratch)