# Weight-Space Programming — GPT-6 Astra reads and edits network weights

> A frontier LLM looks at a network's weights, works out what they compute, and rewrites them without any training loop.

- Org: MIT CSAIL
- Date: 2026-11-02
- Category: paper, algorithm
- Source: https://rikiyat.github.io/blog/weight-space-programming/
- Canonical page: https://ai-tldr.dev/releases/mit-csail-weight-space-programming/

## In short

MIT CSAIL researchers report that GPT-6 Astra can read raw neural network weights and edit them by hand. One edit to 0.016% of Qwen3.5-0.8B's weights lifted MATH-500 from 7.9% to 48.1%. Full logs are public.

## What it is

Weight-Space Programming is the name MIT CSAIL researchers give to an LLM reasoning about what each part of a neural network computes and then editing the weights directly. In eight experiments, OpenAI's GPT-6 Astra named which Atari-style game a small network was trained on from its weights alone, told which shape a diffusion model draws, wrote game-playing networks from scratch, and fixed Qwen3.5-0.8B's habit of looping on math problems.

## How it works

The model gets the weights as text, or a sandboxed Python environment where it can load the tensors with numpy or PyTorch but cannot run the network. It may spend a small budget of forward passes (0, 10, 25 or 50) to watch the network's behaviour between edits. For Qwen, Astra traced the looping to one direction written by the MLPs of layers 15 and 19 and subtracted it from 64 output columns in each, changing 131,072 weights.

## Why it matters

The study suggests a new route for interpretability and model repair: auditing a model or removing a fault without data and without gradient training. The authors also show limits. Astra could not find a sycophantic model from weights alone, and it could not improve Gemma 3 270M on MATH-500, which suggests an edit can remove an obstacle but not add a missing skill.

## Who it's for

interpretability and ML researchers

## Try it

git clone https://github.com/RikiyaT/lm_understand-logs

## Quick facts

- Authors: Rikiya Takehi, Jayden Teoh, Alex Zhang, Omar Khattab (MIT CSAIL)
- Model used: OpenAI GPT-6 Astra via Codex CLI, built-in tools off
- Experiments: 8 (games, diffusion, small LMs, Go, Qwen math)
- Qwen3.5-0.8B GSM8K: 13.8% → 67.1%
- Qwen3.5-0.8B IFEval: 21.0% → 62.3%
- Release: Blog post + full logs (MIT scripts, CC BY 4.0 data)

## FAQ

### Which model did the MIT CSAIL team use for Weight-Space Programming?

The MIT CSAIL team used OpenAI's GPT-6 Astra, called through the Codex CLI with every built-in tool switched off, including shell, file editing and web search. The released logs say all 1,125 turns were tool-free. In four experiments, Astra's Python ran in the authors' own sandbox with no network and a time limit, and the printed output came back as the next message.

### Can I reproduce the weight-space programming experiments?

The RikiyaT/lm_understand-logs repository holds the prompts, Astra's replies and reasoning summaries, the code it ran with its output, the evaluations behind each number, and the networks it read or wrote, for experiments 1 to 8. Scripts are MIT-licensed and logs, data and networks are CC BY 4.0. The 17 GB Gemma 3 270M fine-tunes are not included, but data, scripts and hashes to rebuild them are.

### Where did GPT-6 Astra fail at reading weights?

GPT-6 Astra could not tell which of several Gemma 3 270M fine-tunes was sycophantic from the weights alone; it only succeeded after it could inspect attention patterns on texts it wrote. It also could not improve Gemma 3 270M on MATH-500, which moved from 6.8 to at most 7.7. The authors call current abilities exciting but still limited by context length and harness overhead.

### How did a weight edit beat PPO on Breakout?

In one experiment GPT-6 Astra wrote a MinAtar game network from scratch, as text in one reply, with no training. On Breakout it scored 517.4, against 15.8 for a PPO agent trained for 100M steps and 25.3 at 500M steps. On Seaquest (53.1 vs 88.9) and Asterix (53.2 vs 55.2) the PPO agents trained for 500M steps stayed ahead.

### What do the Weight-Space Programming authors plan next?

The MIT CSAIL authors say they are building a benchmark and a harness to measure how well LLMs do Weight-Space Programming. They also list auditing without inference, such as finding backdoors from weights, zero-data capability steering and unlearning, one-shot model initialization, recursive self-improvement, and grafting modules between different models as directions to explore.

## Links

- [Blog Post](https://rikiyat.github.io/blog/weight-space-programming/)
- [Experiment Logs](https://github.com/RikiyaT/lm_understand-logs)
- [HN Discussion](https://news.ycombinator.com/item?id=50341174)