K2 Horizon MoVA 36B-A4B

// Overview

K2 Horizon MoVA 36B-A4B is the sparse model in K2 Horizon, the six-model fleet the Institute of Foundation Models released on 3 September 2026 alongside the 0.9B, 3.7B, 7B, 32B and 375B-A23B sizes. It stores 36 billion parameters and runs about 4 billion of them per token, and takes text input across a native 524,288-token context window carried from the midtraining stages onward.

Its distinguishing piece is MoVA, IFM's Mixture-of-Value Attention. Conventional Mixture-of-Experts designs apply sparsity to the feed-forward layers: many experts exist, but a router picks a small subset per token. MoVA extends that routing into multi-head attention itself, which IFM describes as opening another dimension for scaling capacity beyond the feed-forward network. IFM positions the 36B-A4B and the dense 32B as a matched pair — trained under the same conditions on the same data — so researchers can compare dense and sparse architectures directly.

On IFM's published table, the model scores 80.8 on GPQA Diamond, 66.3 on AA-LCR long-context reasoning, 58.6 on Terminal-Bench 2.1, 38.9 on SciCode, 26.8 on tau3-Banking agentic tool use and 25.2 on Humanity's Last Exam without tools. IFM compares it against open-weight models up to 15 times its total size: it leads Nemotron 3 Ultra (550B) on Terminal-Bench 2.1 and tau3-Banking while trailing it on GPQA Diamond and Humanity's Last Exam. IFM also states that under the same training conditions the model performs only slightly below the dense Horizon 32B while activating far fewer parameters.

Like the rest of the fleet it ships under Apache-2.0 with more than the weights. IFM publishes training data or detailed construction recipes, training code, model configurations, intermediate checkpoints, fine-grained training logs and evaluation results across the whole fleet; the 36B-A4B card notes that its final checkpoint is out and that intermediate checkpoints, data and training code follow. Each model in the fleet was pretrained on roughly 20 trillion tokens, about 10 trillion of them synthetic, with nearly 17% of the pre-training corpus made up of explicit problem-solving trajectories.

Serving recipes are published for vLLM and SGLang, the latter validated on 2× H200. Both use the `k2_horizon` reasoning and tool-call parsers, and IFM asks for `reasoning_effort="high"`, `temperature=1.0` and `top_p=0.95` — the settings all of its reported results were produced with. IFM is a research institute the Mohamed bin Zayed University of Artificial Intelligence launched in May 2025, continuing the fully-open line of work it began with the 2023 LLM360 paper.

// Benchmarks

This model's scores

1. GPQA Diamond (graduate-level science QA)80.8%
2. AA-LCR (long-context reasoning)66.3%
3. Terminal-Bench 2.1 (agentic terminal use)58.6%
4. SciCode (scientific coding)38.9%
5. tau3-Banking (agentic tool use)26.8%
6. Humanity's Last Exam (without tools)25.2%
7. AA-Omniscience Non-Hallucination69.2%
8. AA-Omniscience Accuracy (factual accuracy)18.8%
9. CritPt (frontier physics reasoning)2.1%

Scores on a 0–100 scale (25-point gridlines); higher is better. Each benchmark links to its published source.

// Strengths

• Frontier-adjacent scores at roughly 4B active parameters per token
• 58.6 on Terminal-Bench 2.1 and 26.8 on tau3-Banking, ahead of every open-weight peer in IFM's table including the 550B Nemotron 3 Ultra
• 512K-token native context window
• Apache-2.0 weights, with training data recipes, code, logs and intermediate checkpoints published across the fleet
• Runs on 2 GPUs under the validated SGLang and vLLM recipes
• Shares architecture, vocabulary, tooling and evaluation setup with the other five Horizon sizes, so moving between them takes no rewriting

// Best for

• Reach for it when you want agentic and reasoning quality close to a 32B dense model but with a fraction of the per-token compute
• Reach for it for local-workstation and efficient-serving deployments that still need tool use and terminal work
• Reach for it for long-document and repo-scale work that needs the 512K-token window
• Reach for it for research comparing sparse against dense training, since IFM trained it and the 32B under matched conditions
• Look at K2 Horizon 375B-A23B instead when quality matters more than serving cost, or at the 7B and smaller sizes for on-device work

// How to access

// K2 Horizon — every version

The full lineage of the K2 Horizon line, newest first. Every version has its own page — click any to compare specs, benchmarks and pricing.