# Forge

> Rescue, validate and retry tool calls so smaller models stay reliable

- Category: Evaluation & Production Ops / Guardrails & Security
- Language: Python
- Licence: MIT
- Repository: https://github.com/antoinezambelli/forge
- Canonical page: https://ai-tldr.dev/tools/forge-guardrails/
- Updated: 2026-05-19

## In short

Forge is a reliability layer for LLM tool-calling, aimed squarely at self-hosted models. It sits in the Guardrails & Security category of the Evaluation & Production Ops tools and is open-source, written in Python and licensed MIT. To get started: install the standalone proxy.

## Overview

Forge is a reliability layer for LLM tool-calling, aimed squarely at self-hosted models. Small open-weight models can reason well enough to pick the right tool but still emit the call in the wrong shape — a Mistral-style payload, an XML wrapper, arguments that do not match the schema — and a single malformed call derails a multi-step workflow. Forge sits in the middle and fixes that class of failure before it reaches your code.

Three mechanisms do the work. Rescue parsing extracts a malformed tool call from whatever format the model produced and normalises it to the canonical OpenAI schema. Response validation checks the call against the tools you actually declared. A retry loop — three attempts by default — nudges the model when validation fails instead of returning a broken call. On top of that you can declare workflow structure: required steps, prerequisites, and a terminal tool that ends the run. The project reports lifting an 8B local model from single digits to 84% on a 26-scenario evaluation suite, and Claude Sonnet from 85% to 98%.

You can adopt it at whichever level fits: run it as a proxy server and point an existing OpenAI-compatible client at `http://localhost:8081/v1`, use the `WorkflowRunner` to drive a declared workflow, or import the guardrails as middleware inside your own agent loop. Backends include llama-server, Ollama, vLLM, Llamafile and the Anthropic API.

## What it does

- Rescue parsing that normalises malformed tool calls (Mistral format, XML wrappers) into the canonical OpenAI schema
- Response validation against your declared tools before a call is returned to your code
- Automatic retry loop, three attempts by default, when validation fails
- Workflow structure: optional required steps, prerequisites and a terminal tool
- Three usage modes — proxy server, `WorkflowRunner`, or guardrails middleware in your own loop
- Context management strategies such as `TieredCompact` under an explicit token budget
- Works with llama-server, Ollama, vLLM, Llamafile and the Anthropic API

## Getting started

Forge needs Python 3.12+ and a running LLM backend. Install the standalone proxy if you want a drop-in endpoint, or the library if you are building the loop yourself.

1. Install the standalone proxy

   Linux and macOS. `init` writes a config, `check` verifies the backend is reachable.

```bash
curl -fsSL https://raw.githubusercontent.com/antoinezambelli/forge/main/install.sh | sh
forge-proxy init
forge-proxy check
```

2. Or install the Python library

   The package is published as `forge-guardrails`; the Anthropic extra adds that backend.

```bash
pip install forge-guardrails
pip install "forge-guardrails[anthropic]"
```

3. Run it as a proxy

   Start the proxy against your backend, then point any OpenAI-compatible client at http://localhost:8081/v1 — no application changes needed.

```bash
python -m forge.proxy --backend llamaserver --gguf path/to/model.gguf --port 8081
```

4. Or declare a workflow in code

   Define your tools with a Pydantic parameter model, declare the workflow, and let WorkflowRunner drive it with a context manager and a token budget.

```python
import asyncio
from pydantic import BaseModel, Field
from forge import (
    Workflow, ToolDef, ToolSpec,
    WorkflowRunner, LlamafileClient,
    ContextManager, TieredCompact,
)

def get_weather(city: str) -> str:
    return f"72°F and sunny in {city}"

class GetWeatherParams(BaseModel):
    city: str = Field(description="City name")

workflow = Workflow(
    name="weather",
    description="Look up weather for a city.",
    tools={
        "get_weather": ToolDef(
            spec=ToolSpec(
                name="get_weather",
                description="Get current weather",
                parameters=GetWeatherParams,
            ),
            callable=get_weather,
        ),
    },
    required_steps=[],
    terminal_tool="get_weather",
    system_prompt_template="You are a helpful assistant.",
)

async def main():
    client = LlamafileClient(
        gguf_path="path/to/model.gguf",
        mode="native",
        recommended_sampling=True,
    )
    ctx = ContextManager(strategy=TieredCompact(keep_recent=2), budget_tokens=8192)
    runner = WorkflowRunner(client=client, context_manager=ctx)
    await runner.run(workflow, "What's the weather in Paris?")

asyncio.run(main())
```

## When to use it

- Make an 8B self-hosted model usable for multi-step tool-calling workflows that would otherwise break on malformed calls
- Add validation and retries to an existing agent without changing it — put the proxy in front of the model endpoint
- Enforce that a workflow visits required steps and finishes on a specific terminal tool
- Keep a long tool-calling conversation inside a token budget with a compaction strategy

## Version history

- 2026-05-19: Forge reached the Hacker News front page with an evaluation of its rescue parsing, retry nudges and step enforcement on multi-step agentic tool-calling workflows against self-hosted models. An ACM CAIS '26 demo accompanies the eval suite.