LLM Fundamentals · TRACK 02/06
Tokens & Tokenization
The unit everything is priced, limited, and measured in.
// THE TRACK
01 · START HERETokensLearn the unit every model reads, every API prices, and every context limit is counted in.BEGINNERTokens vs Words vs CharsGet reliable rules of thumb for converting between tokens, words, and characters — and know when those rules lie.BEGINNERByte-Pair EncodingWalk through byte-pair encoding step by step and see exactly how raw text becomes the token IDs a model consumes.INTERMEDIATEThe Strawberry ProblemUse the famous strawberry test to understand a whole family of LLM failures that trace back to one cause: the tokenizer.INTERMEDIATECounting TokensCount tokens accurately with real tools before you send a prompt, so context limits and invoices never surprise you.BEGINNERChat Templates & Special TokensSee how role-tagged messages actually become one token stream — and why using the wrong template silently ruins a local model's answers.INTERMEDIATETokenizing Other LanguagesUnderstand why a sentence in Thai or Arabic can cost several times more tokens than the same sentence in English, and what that means for global users.INTERMEDIATEVocabulary SizeLearn what a tokenizer's 'vocabulary' is, why it's a fixed list of around 100,000 tokens, and how its size trades off speed against efficiency.INTERMEDIATEUnknown & Rare WordsSee why modern LLMs almost never face a truly 'unknown word' and how typos, brand names, and made-up terms get broken into familiar subword pieces.INTERMEDIATE