Hugging Face · 2026-09-09 · major
Transformers v5.17.0 — seven new architectures and a vision RoPE cleanup
Transformers v5.17.0 adds seven model architectures, including Tencent's 780B HYV4 mixture-of-experts and Moonshot's Kimi Linear attention. Vision 2D/3D rotary embeddings move into one shared module, which breaks custom vision models.
Seven architectures land at once, and every custom vision model that hand-rolled its rotary embeddings has to move.
Quick facts
| Maker | Hugging Face |
|---|---|
| Version | v5.17.0 |
| Released | September 9, 2026 |
| License | Apache-2.0 |
| New architectures | HYV4, VibeVoice, NeoMME, Fun-ASR-Nano, KimiLinear, Canary, NeuCodec |
| Breaking change | Vision 2D/3D RoPE centralized in modeling_rope_utils.py |
| Stars | 165,115 |
What is it?
Seven model families join Transformers v5.17.0 in a single release. The headline entry is HYV4 — Hy4-Preview, a 780B mixture-of-experts that activates 49B parameters per token across a 1M-token context. Alongside it come VibeVoice for long-form multi-speaker speech, NeoMME multimodal encoders at 260M and 800M, Alibaba's 800M Fun-ASR-Nano speech recogniser, Moonshot's KimiLinear, plus Canary and NeuCodec for speech.
How does it work?
HYV4 stacks four ideas: Multi-head Latent Attention compresses keys and values into a low-rank latent, DeepSeek Sparse Attention picks a top-k of keys per query with a light indexer, gated MLA adds learnable attention sinks, and Independent Hyper-Connections replace the plain residual path with several parallel streams. KimiLinear centres on Kimi Delta Attention, a refinement of Gated DeltaNet with per-channel forget gates. Each MoE layer in HYV4 holds 256 routed experts plus one shared expert and sends each token to eight of them.
Why does it matter?
The upgrade is free for anyone loading stock checkpoints, but the vision rotary embedding change is a real migration. Custom vision models that relied on attention-layer-level or model-specific RoPE grid interleaving must switch to the centralized `modeling_rope_utils.py` implementation or they will not load. Generation also stops synchronizing the accelerator on every decode step, which trims per-step overhead without a config change.
Who is it for?
ML engineers and model authors
Frequently asked questions
- What breaks when upgrading to Transformers v5.17.0?
- Transformers v5.17.0 standardizes vision 2D and 3D rotary embeddings into one shared frequency module. Custom vision models that computed RoPE grids at the attention-layer level, or used model-specific interleaving logic, have to migrate to the centralized `modeling_rope_utils.py` implementation. Stock checkpoints from the Hub are unaffected.
- How large is the HYV4 model added in this release?
- HYV4, released as Hy4-Preview, is a 780B-parameter mixture-of-experts model that activates 49B parameters per token. Each MoE layer contains 256 routed experts plus one always-active shared expert, routing every token to eight of them, and the context window is 1M tokens.
- Does Transformers v5.17.0 run HYV4's multi-token prediction layers?
- No. The Transformers implementation of HYV4 does not execute the multi-token prediction layers. Released checkpoints still carry those weights so other runtimes can use them for speculative decoding, but Transformers ignores them at load time.
- What speech models does Transformers v5.17.0 add?
- Transformers v5.17.0 adds four speech-related entries. Fun-ASR-Nano is an 800M speech recogniser from Alibaba DAMO Academy covering Chinese, English and Japanese, with seven Chinese dialects, hotword customization and native punctuation. VibeVoice synthesizes long multi-speaker audio, while Canary handles recognition and translation and NeuCodec covers audio coding.
Try it
pip install -U transformers