Overview
NCP-ArchPreview is a pretraining experiment from the NCP team at Shanghai AI Lab and LUMIA Lab, Shanghai Jiao Tong University, with its technical report posted on 9 September 2026. The idea it tests is Next Concept Prediction: on top of ordinary next-token prediction, the model also predicts discrete "concepts" that span several tokens, drawn from a product-quantised concept vocabulary and produced by a dedicated concept module trained jointly with the rest of the network.
Structurally the model is three stacks rather than one: a 16-layer token encoder, an 8-layer concept module, and a 16-layer token decoder, with a hidden size of 4,096 and 32 attention heads. Token-level attention uses a 4,096-token local window with full attention every fourth layer. The published checkpoints total roughly 8.94B parameters, of which the vector-quantisation module is about 17M.
The headline claim is efficiency during pretraining. The report states that the model reaches OLMo-3-7B's final Stage 1 pretraining loss using 51.3% of the training tokens, and approaches an 8.9B baseline's performance with 85% of the standard computation. On downstream evaluation at the step-1,300,000 Stage 1 checkpoint, the overall average is 49.04 against OLMo-3-7B's 46.59, with the largest gains on PIQA (+8.60) and GSM8K (+5.99).
This is a research artefact, not a product: the weights are intermediate Stage 1 checkpoints trained on the Dolma 3 Mix (5.73T tokens for the full run), the maximum training context is 8,192 tokens, and there is no instruction-tuned or chat variant. They are published through ArchSpace, described as an open experiment that puts community architecture hypotheses through traceable, reproducible training and evaluation — forty checkpoints from step 400,000 upward are available, so the training curve itself can be inspected.
| Released | 2026-09-09 |
|---|---|
| License | Apache-2.0 |
| Weights | Open weights |
| Parameters | ≈8.94B, including a 17M-parameter vector-quantisation module |
| Context | 8,192 tokens (maximum training context) |
| Architecture | Latent-space language model: 16-layer token encoder, 8-layer concept module with product quantisation (32 codebooks × 128 codewords), 16-layer token decoder; hidden size 4,096, 32 attention heads, 4,096-token local attention window with full attention every fourth layer |
| Modalities | Text |
| Status | Research preview — intermediate Stage 1 pretraining checkpoints published under Apache-2.0; no instruction-tuned or chat release |
Benchmarks
NCP-ArchPreview Stage 1 against the OLMo-3-7B Stage 1 baseline, as published on the model card
| Benchmark | NCP-ArchPreview 8.9B (Stage 1) | OLMo-3-7B (Stage 1) |
|---|---|---|
| Overall average | 49.04 | 46.59 |
| MMLU | 64.8 | 62.22 |
| GSM8K | 45.26 | 39.27 |
| MATH-500 | 14.48 | 12.52 |
| HumanEval | 31.38 | 27.1 |
| MBPP | 35.91 | 34.53 |
| ARC-Challenge | 81.57 | 77.99 |
| PIQA | 80.85 | 72.25 |
| Mathematics domain average | 24.54 | 20.79 |
| Code domain average | 27.79 | 25.15 |
| Likelihood BPB average (lower is better) | 0.811 | 0.824 |
This model's scores
- ARC-Challenge81.57%
- PIQA80.85%
- MMLU64.8%
- Overall average49.04%
- GSM8K45.26%
- MBPP35.91%
- HumanEval31.38%
- MATH-50014.48%
Scores on a 0–100 scale (25-point gridlines); higher is better. Each benchmark links to its published source.
Strengths
- Reaches OLMo-3-7B's final Stage 1 pretraining loss on 51.3% of the training tokens, per the technical report
- Beats the OLMo-3-7B Stage 1 baseline on every published downstream row: 49.04 overall average against 46.59
- Lower bits-per-byte on the likelihood average (0.811 against 0.824), so the gain is not only on multiple-choice tasks
- Apache-2.0 weights with forty intermediate checkpoints published, making the training trajectory reproducible and inspectable
- Loads through standard Hugging Face Transformers with trust_remote_code, and the card documents vLLM and SGLang deployment
Best for
- Studying latent-space and multi-token prediction objectives against a matched dense baseline
- Reproducing the token-efficiency claim by comparing checkpoints along the published Stage 1 curve
- Architecture research that needs an open 8.9B model with a non-standard concept module rather than another vanilla transformer
- Teaching or evaluation work on how product-quantised concept vocabularies affect downstream scores
FAQ
What is Next Concept Prediction?
A second training objective layered on next-token prediction. A concept module predicts discrete concepts that span multiple tokens, taken from a product-quantised concept vocabulary (32 codebooks of 128 codewords each in this model), and the whole stack — token encoder, concept module, token decoder — is trained jointly end to end.
Can I chat with NCP-ArchPreview?
No. The published weights are intermediate Stage 1 pretraining checkpoints with no instruction tuning or chat template, and the maximum training context is 8,192 tokens. The card's own example is plain text completion via AutoModelForCausalLM with trust_remote_code=True in BF16.
What does the token-efficiency claim mean in practice?
The report says the model reaches OLMo-3-7B's final Stage 1 pretraining loss after 51.3% of the tokens, and approaches an 8.9B baseline's quality at about 85% of the standard computation. The downstream table backs this at the step-1,300,000 checkpoint: 49.04 overall average against 46.59, and a lower bits-per-byte average of 0.811 against 0.824.
Who built it, and what is ArchSpace?
The model card credits the NCP Team at Shanghai AI Lab and LUMIA Lab, Shanghai Jiao Tong University. ArchSpace is the Hugging Face organisation it is published under, described there as an open experiment for LLM architecture innovation that puts community architecture hypotheses into transparent, traceable and reproducible training and evaluation workflows.