Google Cloud AI Research · 2026-08-20 · major
EnvHarness — Google's wrapper layer makes static agent benchmarks adapt
EnvHarness wraps an existing agent benchmark so its behaviour can be reshaped without editing the benchmark itself. The paper reports up to a 9.0-point gain on held-out tasks with 9.8% fewer steps. Code is Apache-2.0.
EnvHarness reshapes an existing agent benchmark into a training environment without touching the benchmark's own code.
Key specs
| Best held out gain | +9.0 points |
|---|---|
| Execution steps saved | 9.8% |
Quick facts
| Maker | Google Cloud AI Research |
|---|---|
| Also credited | Washington University in St. Louis, UNC Chapel Hill |
| Parts | EnvHarness wrapper layer + EnvRigger |
| License | Apache-2.0 (code) |
| Language | Python |
| Evaluated on | 5 benchmarks across 4 domains |
| Paper | arXiv 2608.19880, 20 August 2026 |
Benchmarks
What is it?
EnvHarness turns a fixed agent benchmark into an environment you can edit. Google Cloud AI Research built it with Washington University in St. Louis and the University of North Carolina at Chapel Hill, and released the code under Apache-2.0. A second part, EnvRigger, watches an agent's trajectories and writes new environment components aimed at the weak spots it finds.
How does it work?
Composable wrappers do the work. Customisation in EnvHarness happens through the standard reset() and step() calls alone, so the interface contract and the original benchmark's verifiers both stay intact. EnvRigger then reads agent trajectories and synthesises targeted components from them, which lets the environment and the policy keep changing together instead of the environment staying frozen.
Why does it matter?
Agent training environments are hand-built and static, so they stop teaching once an agent gets good at them. The EnvHarness paper reports up to a 9.0-point gain on held-out instances with 9.8% fewer execution steps across five benchmarks in four domains. Teams that already run WebArena, SWE-bench Verified or ALFWorld can reuse those environments for training rather than building new ones.
Who is it for?
agent and RL researchers
Frequently asked questions
- Is EnvHarness open source?
- Yes. EnvHarness ships as Apache-2.0 Python code at github.com/google-research/envharness, published alongside the arXiv paper on 20 August 2026. The repository sits under the google-research organisation. A project page at envharness.com hosts a browser playground, so the framework can be inspected and tried without an API key or a paid account.
- Which benchmarks did EnvHarness improve, and by how much?
- The EnvHarness paper reports three headline results. On WebArena the agent moves from 38.7 to 41.6. On SWE-bench Verified it moves from 47.7 to 52.6. On ALFWorld it moves from 61.7 to 68.3, the largest jump at 5.9 points. Across five benchmarks in four domains the best held-out gain reaches 9.0 points.
- Does EnvHarness require rewriting an existing benchmark?
- No. EnvHarness customises an environment through the standard reset() and step() interface alone, using composable wrapper components. The interface contract and the original benchmark's verifiers both stay intact, so scores remain comparable to the unmodified benchmark and maintainers do not have to fork or patch the environment's own code.
- How does EnvHarness compare to training an agent in the real environment?
- The paper includes that comparison as a separate column. Learning in real environments scores 38.5 on WebArena, 49.9 on SWE-bench Verified and 62.4 on ALFWorld. EnvHarness environments beat all three, at 41.6, 52.6 and 68.3. On WebArena, real-environment training actually scored slightly below the base agent's 38.7.
- Can I try EnvHarness without running any code?
- Yes. The EnvHarness project page at envharness.com includes a playable demo built on the 24 game, where four cards are combined with plus, minus, times and divide, two at a time, until one number is left. Visitors can edit environment components live and watch how the policy's observations diverge from the real environment state.
Try it
git clone https://github.com/google-research/envharness