Forge

// Overview

Forge is a reliability layer for LLM tool-calling, aimed squarely at self-hosted models. Small open-weight models can reason well enough to pick the right tool but still emit the call in the wrong shape — a Mistral-style payload, an XML wrapper, arguments that do not match the schema — and a single malformed call derails a multi-step workflow. Forge sits in the middle and fixes that class of failure before it reaches your code.

Three mechanisms do the work. Rescue parsing extracts a malformed tool call from whatever format the model produced and normalises it to the canonical OpenAI schema. Response validation checks the call against the tools you actually declared. A retry loop — three attempts by default — nudges the model when validation fails instead of returning a broken call. On top of that you can declare workflow structure: required steps, prerequisites, and a terminal tool that ends the run. The project reports lifting an 8B local model from single digits to 84% on a 26-scenario evaluation suite, and Claude Sonnet from 85% to 98%.

You can adopt it at whichever level fits: run it as a proxy server and point an existing OpenAI-compatible client at `http://localhost:8081/v1`, use the `WorkflowRunner` to drive a declared workflow, or import the guardrails as middleware inside your own agent loop. Backends include llama-server, Ollama, vLLM, Llamafile and the Anthropic API.

// What it does

• Rescue parsing that normalises malformed tool calls (Mistral format, XML wrappers) into the canonical OpenAI schema
• Response validation against your declared tools before a call is returned to your code
• Automatic retry loop, three attempts by default, when validation fails
• Workflow structure: optional required steps, prerequisites and a terminal tool
• Three usage modes — proxy server, `WorkflowRunner`, or guardrails middleware in your own loop
• Context management strategies such as `TieredCompact` under an explicit token budget
• Works with llama-server, Ollama, vLLM, Llamafile and the Anthropic API

// Getting started

Forge needs Python 3.12+ and a running LLM backend. Install the standalone proxy if you want a drop-in endpoint, or the library if you are building the loop yourself.

curl -fsSL https://raw.githubusercontent.com/antoinezambelli/forge/main/install.sh | sh
forge-proxy init
forge-proxy check

pip install forge-guardrails
pip install "forge-guardrails[anthropic]"

python -m forge.proxy --backend llamaserver --gguf path/to/model.gguf --port 8081

import asyncio
from pydantic import BaseModel, Field
from forge import (
    Workflow, ToolDef, ToolSpec,
    WorkflowRunner, LlamafileClient,
    ContextManager, TieredCompact,
)

def get_weather(city: str) -> str:
    return f"72°F and sunny in {city}"

class GetWeatherParams(BaseModel):
    city: str = Field(description="City name")

workflow = Workflow(
    name="weather",
    description="Look up weather for a city.",
    tools={
        "get_weather": ToolDef(
            spec=ToolSpec(
                name="get_weather",
                description="Get current weather",
                parameters=GetWeatherParams,
            ),
            callable=get_weather,
        ),
    },
    required_steps=[],
    terminal_tool="get_weather",
    system_prompt_template="You are a helpful assistant.",
)

async def main():
    client = LlamafileClient(
        gguf_path="path/to/model.gguf",
        mode="native",
        recommended_sampling=True,
    )
    ctx = ContextManager(strategy=TieredCompact(keep_recent=2), budget_tokens=8192)
    runner = WorkflowRunner(client=client, context_manager=ctx)
    await runner.run(workflow, "What's the weather in Paris?")

asyncio.run(main())

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Make an 8B self-hosted model usable for multi-step tool-calling workflows that would otherwise break on malformed calls
• Add validation and retries to an existing agent without changing it — put the proxy in front of the model endpoint
• Enforce that a workflow visits required steps and finishes on a specific terminal tool
• Keep a long tool-calling conversation inside a token budget with a compaction strategy

// How Forge compares

Forge alongside other open-source guardrails & security tools AI/TLDR tracks, ranked by GitHub stars.