Allen Institute for AI · 2026-05-08 · notable
Ai2 EMO — 14B MoE That Keeps Near-Full Accuracy When Run on Just 12.5% of Its Experts
Ai2 trained a 1B-active, 14B-total MoE so that document-level routing makes expert subsets specialise into real semantic domains. Drop to 16 experts (12.5%) and quality only falls ~3%; standard MoEs collapse.
An MoE where you can keep just 16 of 128 experts and still have a working model.
Key specs
| Parameters | 14B |
|---|---|
| Active params | 1B |
| Experts | 128 routed + 1 shared, top-8 |
| Train tokens | 1T + 50B EMO anneal |
| Drop at12 5pct experts | ~3% |
| Drop at25pct experts | ~1% |
What is it?
EMO is an open Mixture-of-Experts language model from Allen AI's OLMo team. Total parameters are 14B but only 1B are active per token. The novelty is the training objective: every token in the same document must route through the same shared pool of experts, so different documents end up using different pools.
How does it work?
During pretraining, EMO sets a per-document expert pool by averaging router preferences across the document's tokens, then constrains routing into that pool while a global load-balancing loss keeps experts evenly used. Document pool sizes are randomised so the model learns to work at many subset sizes. At inference you pick a domain-relevant subset of experts and run only that subset; the rest is left unloaded.
Why does it matter?
Standard MoEs degrade sharply when you try to drop experts at inference: their experts mostly specialise on surface features like punctuation, so any subset is broken. EMO's experts cluster around semantic domains (Health, Politics, Code), so you can ship a smaller domain-specialised slice without retraining. Ai2 also released a matched standard-MoE baseline trained on the same OLMoE mix for direct comparison.
Who is it for?
open-source MoE researchers and teams that want domain-specialised slices of a base model
Try it
pip install transformers && from transformers import AutoModelForCausalLM; AutoModelForCausalLM.from_pretrained('allenai/Emo_1b14b_1T', trust_remote_code=True)