PromptTools

// Overview

PromptTools, from Hegel AI, is a set of self-hostable tools for experimenting with prompts, models and vector databases. Its premise is that prompt evaluation should happen in the places engineers already work — code, notebooks and a local UI — rather than in a hosted dashboard you have to send your data to. Everything runs on your machine, against your own API keys.

The core abstraction is an *experiment*: you give it the axes you want to vary — a list of message sets, a list of models, a list of temperatures — and it runs the full cross-product, collects the responses, and renders them as a table you can eyeball or score. `openai_experiment.run()` followed by `.visualize()` is the whole loop in a notebook. Because the axes are just Python lists, a regression check across two models and a dozen prompts is a handful of lines rather than a bespoke script.

The same treatment extends to retrieval. PromptTools ships experiments for vector databases — Chroma, Weaviate, Qdrant, LanceDB and Pinecone among them — so you can measure whether a change to chunking or an embedding model actually improved what comes back, instead of inferring it from downstream answer quality. On the model side the supported list spans OpenAI (completion, chat and fine-tuned models), Anthropic, Google Gemini and Vertex AI, Mistral AI, Azure OpenAI, Replicate, HuggingFace Hub and Inference Endpoints, and LLaMA.cpp for local models.

For people who would rather click than write code, a Streamlit playground exposes three modes — Instruction, Prompt Template and Model Comparison — with fields for each provider's API key, so a non-engineer can run the same side-by-side comparison. It runs locally with a single `streamlit run`, and Hegel AI also hosts a version on the Streamlit Community Cloud (without LlamaCpp support).

// What it does

• Experiment objects that sweep the cross-product of prompts, models and parameters in a few lines, then render the results as a table
• Broad provider coverage in one harness: OpenAI, Anthropic, Google Gemini and Vertex AI, Mistral AI, Azure OpenAI, Replicate, HuggingFace, and LLaMA.cpp for local models
• Vector-database experiments that measure retrieval accuracy directly — Chroma, Weaviate, Qdrant, LanceDB and Pinecone
• A local Streamlit playground with Instruction, Prompt Template and Model Comparison modes for click-driven comparisons
• Notebook-first workflow: `run()` then `visualize()`, with example notebooks in the repo and a Google Colab starting point
• Entirely self-hosted — prompts, responses and API keys stay on your machine

// Getting started

PromptTools installs from PyPI. The fastest path is a notebook; the playground is a second command once the package is installed.

pip install prompttools

from prompttools.experiment import OpenAIChatExperiment

messages = [
    [{"role": "user", "content": "Tell me a joke."},],
    [{"role": "user", "content": "Is 17077 a prime number?"},],
]

models = ["gpt-3.5-turbo", "gpt-4"]
temperatures = [0.0]
openai_experiment = OpenAIChatExperiment(models, messages, temperature=temperatures)
openai_experiment.run()
openai_experiment.visualize()

pip install notebook
git clone https://github.com/hegelai/prompttools.git
cd prompttools && jupyter notebook examples/notebooks/OpenAIChatExperiment.ipynb

git clone https://github.com/hegelai/prompttools.git
cd prompttools && streamlit run prompttools/playground/playground.py

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Reach for it when a prompt change needs a before/after across several models, and 'it looked better in the chat window' is not evidence
• Reach for it when picking between providers for one task and you want the same grid run against all of them
• Reach for it to measure whether a chunking or embedding change actually improved retrieval, rather than judging it by the final answer
• Reach for it when the data cannot leave your machine, so a hosted prompt-evaluation service is not an option

// How PromptTools compares

PromptTools alongside other open-source evaluation & red-teaming tools AI/TLDR tracks, ranked by GitHub stars.