Google · 2026-06-03 · major
Gemma 4 12B — Google's First Unified, Encoder-Free Multimodal Decoder Streams Audio, Video, and Image Patches Straight Into an 11.95B Apache-2.0 LLM With 256K Context
An 11.95B decoder-only multimodal model that drops the vision and audio encoders, projects raw 48x48 patches and 40ms audio frames straight into the LLM, runs on a 16GB laptop, and ships open-weight under Apache 2.0.
A 12B open multimodal model that lets raw audio and video patches skip the encoder and hit the LLM directly.
Key specs
| Parameters | 11.95B |
|---|---|
| Context window | 256K |
| MMLU | 77.2% |
| GPQA | 78.8% |
| Aime2026 | 77.5% |
| Live code bench v6 | 72.0% |
| Mmmu pro vision | 69.1% |
| Math vision | 79.7% |
| Codeforces elo | 1,659 |
| Min vram | 16GB |
What is it?
Gemma 4 12B is Google DeepMind's first medium-sized model in the Gemma 4 line that ingests text, image, audio, and video without separate vision or audio encoders. Weights and an instruction-tuned variant are on Hugging Face and Kaggle under Apache 2.0, designed to fit a 16GB laptop GPU or unified-memory Mac.
How does it work?
Instead of routing pixels through a vision tower and audio through a speech encoder, the model uses a 35M-parameter vision embedder that projects raw 48x48 RGB patches and 40ms 16kHz audio frames straight into the LLM's hidden dimension. The backbone is a 48-layer dense transformer with a 262K-token vocabulary and a 1024-token sliding window, trained to do speculative-style multi-token prediction for faster decode.
Why does it matter?
Removing the encoders cuts memory and latency for multimodal pipelines and lets a single set of weights handle ASR, video understanding, and agentic reasoning on commodity hardware. Posted scores include 77.2 MMLU-Pro, 77.5 AIME 2026, 72.0 LiveCodeBench v6, 78.8 GPQA Diamond, and 1659 Codeforces Elo — close to the larger Gemma 4 26B MoE at less than half the memory.
Who is it for?
On-device app builders, agent developers, and labs that want a permissively licensed multimodal model.
Try it
huggingface.co/google/gemma-4-12B-it