Marcel Rød · 2026-07-21 · major
GigaToken v0.9.0 — a Rust tokenizer that runs ~1000× faster than HuggingFace
GigaToken is a Rust tokenizer from Stanford PhD student Marcel Rød that hits 24.5 GB/s on EPYC 9565 and drops into HuggingFace and tiktoken code paths. v0.9.0 ships MIT with SIMD encoders for GPT-2, Llama, Qwen, DeepSeek, GLM, and Gemma.
GigaToken hits ~24.5 GB/s on a 144-core EPYC and drops straight into HuggingFace and tiktoken code paths.
Key specs
| GitHub stars | 1,011 |
|---|---|
| Peak throughput | 24.53 GB/s (EPYC 9565) |
| Speedup vs hf | ~1000x |
Quick facts
| Maker | Marcel Rød (Stanford PhD student, CS336) |
|---|---|
| Version | 0.9.0 (2026-07-21) |
| License | MIT |
| Language | Rust with Python bindings |
| Install | pip install gigatoken |
| Supports | GPT-2, Llama 3/4, Qwen 2–3.6, DeepSeek, GLM 4/5, Gemma, Mistral |
| Traction | 1,011 GitHub stars; HN front page (253 points) |
What is it?
GigaToken is a new Rust tokenization library from Stanford PhD student Marcel Rød that reads text at gigabytes per second while staying API-compatible with the two tools every ML team already uses — HuggingFace tokenizers and OpenAI's tiktoken. Version 0.9.0 hit PyPI on July 21 with a matching MIT-licensed source release on GitHub.
How does it work?
The core is SIMD-optimized Rust with per-tokenizer fast paths and intelligent caching. On an AMD EPYC 9565 with 144 cores it reports 24.53 GB/s on GPT-2 encoding versus 24.8 MB/s from HuggingFace and 36.0 MB/s from tiktoken. On a 16-core Apple M4 Max it still delivers 8.79 GB/s. Python users install with `pip install gigatoken` and swap in `gigatoken.Tokenizer` for their existing HF or tiktoken calls.
Why does it matter?
Tokenization is a real bottleneck when you're preparing hundreds of GB of pretraining data or streaming logs through a classifier. Cutting that step from hours to minutes changes what a single laptop or one CI job can turn around, and the drop-in HF/tiktoken shims mean teams can adopt it without rewriting their pipelines.
Who is it for?
ML engineers building pretraining pipelines, data-engineering teams processing large text corpora, and researchers running high-throughput evaluation loops.
Frequently asked questions
- Is GigaToken really a drop-in replacement for HuggingFace tokenizers?
- GigaToken exposes the same Python API surface as HuggingFace's `tokenizers` package and OpenAI's `tiktoken`, so most call sites work without changes. It supports the tokenizer families ML teams already use — GPT-2, Llama 3/3.1/3.2/3.3/4, Qwen 2/2.5/3/3.5/3.6, DeepSeek, GLM 4/5, Nemotron 3, Gemma, Mistral, and CodeLlama — under an MIT license.
- How much faster is GigaToken in practice?
- On an AMD EPYC 9565 (144 cores), GigaToken encodes GPT-2 text at 24.53 GB/s versus 24.8 MB/s from HuggingFace and 36.0 MB/s from tiktoken — roughly a 1000x speedup on the same hardware. On an Apple M4 Max with 16 cores the number is 8.79 GB/s, still about three orders of magnitude ahead of HuggingFace.
- Who built GigaToken and what is it built on?
- GigaToken is written by Marcel Rød, a third-year PhD student at Stanford who teaches CS336 (language modeling from scratch). The library is Rust with Python bindings, uses SIMD instructions and per-tokenizer optimizations, and is released under the MIT license on GitHub with 1,011 stars on launch day.
- Where does GigaToken run and what's the install command?
- GigaToken runs anywhere CPython 3 does — Linux, macOS, and Windows on both x86 and ARM. Install with `pip install gigatoken`, then import as `gigatoken as gt` and call `gt.Tokenizer("Qwen/Qwen3-8B")` (or any supported tokenizer id). File-source and streaming APIs are exposed for pretraining-scale workloads.
Try it
pip install gigatoken