Hugging Face Tokenizers

// Overview

Tokenizers is Hugging Face's implementation of the tokenization algorithms in common use, written in Rust with bindings for Python, Node.js and Rust itself (a community Ruby binding lives in a separate repository). It is the layer that turns text into the integer ids a model consumes, and the piece the wider Transformers ecosystem loads when you call a fast tokenizer.

It does two jobs. The first is training: point it at a corpus and it will fit a new vocabulary using Byte-Pair Encoding, WordPiece or Unigram, with configurable normalizers, pre-tokenizers and special tokens. The second is inference-time encoding, and here the library handles the whole pre-processing chain — normalization, pre-tokenization, truncation, padding and the special tokens a given model expects.

The design detail that matters in practice is alignment tracking: normalization keeps a mapping back to the original string, so for any token you can recover the exact span of input text it came from. That is what makes token-level tasks — named-entity extraction, span highlighting, offset-based post-processing — possible without re-deriving positions by hand. The project is Apache-2.0 and states a target of tokenizing a gigabyte of text in under twenty seconds on a server CPU.

// What it does

• Train new vocabularies with Byte-Pair Encoding, WordPiece or Unigram from your own corpus
• Rust core with official Python, Node.js and Rust bindings
• Alignment tracking through normalization — map any token back to its span in the original text
• Full pre-processing chain built in: normalizers, pre-tokenizers, truncation, padding and special tokens
• The fast-tokenizer implementation the Transformers ecosystem loads, so trained tokenizers travel with models
• Apache-2.0, installable as a wheel with `pip install tokenizers`

// Getting started

The Python binding is the usual entry point. These snippets come from the project README's quick example.

pip install tokenizers

# or from source
pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.pre_tokenizers import Whitespace

tokenizer = Tokenizer(BPE())
tokenizer.pre_tokenizer = Whitespace()

from tokenizers.trainers import BpeTrainer

trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])
tokenizer.train(files=["wiki.train.raw", "wiki.valid.raw", "wiki.test.raw"], trainer=trainer)

output = tokenizer.encode("Hello, y'all! How are you 😁 ?")
print(output.tokens)
# ["Hello", ",", "y", "'", "all", "!", "How", "are", "you", "[UNK]", "?"]

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Train a domain-specific vocabulary before pretraining or continued-pretraining a model
• Encode text in a production serving path with truncation, padding and special tokens handled for you
• Recover the exact source span behind a token for entity extraction or span highlighting
• Use the same tokenizer definition across Python services, Node.js front ends and Rust binaries

// How Hugging Face Tokenizers compares

Hugging Face Tokenizers alongside other open-source tokenizers tools AI/TLDR tracks, ranked by GitHub stars.