GigaToken

// Overview

GigaToken is a tokenizer for language-model data pipelines, written in Rust by Marcel Rød and published to PyPI as `gigatoken`. Its single claim is throughput: where Hugging Face Tokenizers and tiktoken measure encoding in megabytes per second, GigaToken measures it in gigabytes per second, and the project publishes per-tokenizer benchmark tables for several CPUs so you can check the number for the vocabulary you actually use.

On the repository's 11.9 GB OpenWebText benchmark run on a dual-socket AMD EPYC 9565 (144 cores), GigaToken reports 24.53 GB/s on the GPT-2 vocabulary against 24.8 MB/s for Hugging Face Tokenizers and 36.0 MB/s for tiktoken — a 989× and 681× ratio as published. Other vocabularies land lower: 22.16 GB/s for Qwen 3, 19.69 GB/s for the DeepSeek V3/R1/V4 vocabulary, and single-digit GB/s for the SentencePiece-style Gemma and Llama 2 families. The same tables are given for an Apple M4 Max and a Ryzen 7 9800X3D. Both baselines it is measured against are themselves multithreaded Rust.

There are two ways to use it. Compatibility mode wraps an existing Hugging Face or tiktoken tokenizer object and returns something usable in the same call sites, with the project stating that considerable effort went into matching Hugging Face output exactly — at a real cost to speed, so you get a large speed-up rather than the headline one. The native GigaToken API is the fast path: you hand it a model name and a file source, and the Rust side reads the data itself, skipping Python overhead and parallelising freely.

// What it does

• Encodes at gigabytes per second on commodity CPUs, with published per-tokenizer, per-CPU benchmark tables in the README
• Compatibility wrappers — `.as_hf()` and `.as_tiktoken()` — so it slots into code written against either library
• Native API that reads text files directly from Rust for maximum parallelism, bypassing Python data-structure overhead
• Covers the vocabularies in common use, including GPT-2, GPT-OSS, Llama 3 and 4, Qwen 2 through 3.6, DeepSeek V3/R1/V4, GLM 4 and 5, Kimi K2, Gemma, Phi-4, OLMo, Nemotron 3 and ModernBERT
• Accepts Hugging Face model names directly, so `gt.Tokenizer("Qwen/Qwen3-8B")` is the whole setup
• Installs as a single wheel: `pip install gigatoken`, MIT licensed

// Getting started

GigaToken ships as a Python package with the Rust engine compiled in. The snippets below are the two usage modes from the project README.

pip install gigatoken

import gigatoken as gt

hf_tokenizer = ...
tokenizer = gt.Tokenizer(hf_tokenizer).as_hf()

tokens = tokenizer.encode_batch(["This is a test string", "And here is another"])

import gigatoken as gt

tokenizer = gt.Tokenizer("Qwen/Qwen3-8B")  # accepts HF model names
file_source = gt.TextFileSource(["owt_train.txt"], separator=b"<|endoftext|>")
tokens = tokenizer.encode_files(file_source)

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Pre-tokenising a multi-terabyte pretraining corpus, where tokenization time is a real line item in the schedule
• Speeding up a data-loading pipeline that currently spends its time inside HF Tokenizers or tiktoken
• Repeated full-corpus re-tokenisation while iterating on data mixes or vocabularies
• Dropping into an existing codebase without a rewrite, via the Hugging Face or tiktoken compatibility wrappers

// How GigaToken compares

GigaToken alongside other open-source tokenizers tools AI/TLDR tracks, ranked by GitHub stars.