Shanghai AI Lab · 2026-09-09 · major
NCP-ArchPreview — an 8.9B model that predicts concepts, not just tokens
NCP-ArchPreview is an 8.9B open-weight language model from Shanghai AI Lab that predicts multi-token "concepts" alongside normal next-token prediction. It reaches OLMo-3-7B's final pretraining loss using 51.3% of the training tokens.
An 8.9B open-weight model that learns to predict the next idea as well as the next token.
Key specs
| Tokens needed to match olmo 3 7 b loss | 51.3% |
|---|
Quick facts
| Maker | Shanghai AI Lab + LUMIA Lab, Shanghai Jiao Tong University |
|---|---|
| Parameters | 8.94B |
| License | Apache-2.0 |
| Training data | 5.73T tokens (Dolma 3 Mix) |
| Context window | 8,192 tokens |
| Architecture | 16-layer token encoder, 8-layer concept module, 16-layer token decoder |
| Availability | Open weights on Hugging Face |
Benchmarks
What is it?
Next Concept Prediction adds a second training target to an ordinary language model: alongside each next token, NCP-ArchPreview also predicts a discrete "concept" that spans a short group of tokens. The NCP Team at Shanghai AI Lab and the LUMIA Lab at Shanghai Jiao Tong University trained the 8.9B result on 5.73T tokens of Dolma 3 and put the weights on Hugging Face under Apache 2.0. The team describes it as the largest latent-space language model shown so far.
How does it work?
A product-quantized codebook turns the model's own hidden states into a concept vocabulary. Three parts split the job: a 16-layer token encoder, an 8-layer concept module holding 32 codebooks, and a 16-layer token decoder. The concept module predicts the next concept, and that prediction is fed back down to the token level to guide what gets generated next. Both objectives train together, end to end.
Why does it matter?
Training cost is the headline for NCP-ArchPreview. It reaches OLMo-3-7B's final pretraining loss after only 51.3% of the tokens, then finishes 2.45 points ahead on the downstream macro-average once fully trained. The learned space keeps paying off after pretraining: updating just the 17M-parameter quantizer is a cheap way to adapt the model to a new domain.
Who is it for?
pretraining researchers
Frequently asked questions
- Is NCP-ArchPreview open source?
- NCP-ArchPreview releases its weights under the Apache 2.0 license on Hugging Face, in the ArchSpace-Collection organisation, so commercial use is allowed. The model card lists training code as coming soon and points to evaluation code that is already available. Multiple checkpoints from both training stages are published, not only the final one.
- How does NCP-ArchPreview compare to OLMo-3-7B?
- NCP-ArchPreview scores 49.04 on the downstream macro-average against OLMo-3-7B's 46.59. The biggest single jump is GSM8K, 45.26 versus 39.27. MMLU moves from 62.22 to 64.80 and HumanEval from 27.10 to 31.38. It also matches OLMo-3-7B's final pretraining loss after using only 51.3% of the training tokens.
- Can NCP-ArchPreview be used as a chat assistant?
- NCP-ArchPreview is a base model with no conversational alignment, so it is not ready to drop in as a chat assistant. Its model card gives the intended use as research on continued pretraining, latent-space modelling and benchmark evaluation. Teams who want a chat model would have to run their own instruction tuning on top of these weights.
- What else is the concept space good for?
- The latent space inside NCP-ArchPreview stays useful after pretraining ends. Updating only the 17M-parameter vector-quantization module gives a lightweight route to domain adaptation. Feeding concept representations into a DFlash2 drafter raised the mean accepted length in speculative decoding by 4.17%, which the team reports came with negligible extra overhead.
Try it
AutoModelForCausalLM.from_pretrained('ArchSpace-Collection/NCP_ArchPreview_dolma3_8.9B_Stage1_Step1300000', trust_remote_code=True)