Ferret

// Overview

Ferret is a declarative-first, expression-oriented embedded language and runtime for data automation. You write the query — FQL — and the runtime works out how to execute it, across browsers, HTTP APIs, databases, documents and whatever custom sources the host application exposes. The original use case was web scraping that stays readable: rather than a script of imperative navigate-click-wait steps, a page extraction is a query that describes the shape of the data you want.

The runtime is a Go library, not a standalone binary, and that is the point. An application embeds the engine and decides exactly which functions, modules, data and external operations a given program may touch — a capability-based model where host values expose application objects and resources directly to FQL. Programs compile to bytecode and can be reused across isolated execution sessions, so the same plan runs repeatedly without recompilation. A separate CLI, test runner (`lab`) and web worker live in sibling repositories.

The repository's main branch carries Ferret v2, an alpha with a new architecture and public API; v1 remains available on its own branch. v2 adds domain-oriented orchestration and constrained mutable state on top of the declarative core, for automation that cannot be expressed as a pure data transformation, plus event-driven synchronisation and managed lifecycles for files, connections, cursors and streams. Existing v1 integrations migrate in two stages via a `compat` module and a `ferret migrate` command. Ferret is Apache-2.0 licensed.

// What it does

• FQL, a purpose-built declarative language for querying, transforming, synchronising and automating structured data — deliberately focused rather than general-purpose
• One query model over browsers, APIs, databases, documents and custom sources, so the source changes without the query shape changing
• Embeddable Go runtime with reusable compiled plans and isolated execution sessions
• Capability-based host values: the embedding application decides exactly which functions, modules, data and external operations a program can reach
• Bytecode VM with portable, precompiled programs for efficient repeated execution
• Event-driven synchronisation and dispatch, plus managed resource lifecycles for files, connections, cursors and streams

// Getting started

Ferret is consumed as a Go module. The v2 flow is engine → compile query → create session → run.

go get github.com/MontFerret/ferret/v2@latest

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/MontFerret/ferret/v2"
)

func main() {
	ctx := context.Background()

	eng, err := ferret.New()
	if err != nil {
		log.Fatal(err)
	}
	defer eng.Close()

	plan, err := eng.Compile(ctx, ferret.NewAnonymousSource(`return 1 + 1`))
	if err != nil {
		log.Fatal(err)
	}
	defer plan.Close()

	session, err := plan.NewSession(ctx)
	if err != nil {
		log.Fatal(err)
	}
	defer session.Close()

	output, err := session.Run(ctx)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(string(output.Content))
}

ferret migrate --dry-run   # list the files that would change
ferret migrate --print     # print a unified diff, change nothing
ferret migrate             # apply

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Extract structured records from JavaScript-heavy pages with a query that describes the data rather than a script of browser steps
• Embed a sandboxed automation language in a Go application and expose only the operations that application is willing to allow
• Run the same compiled extraction plan repeatedly across isolated sessions in a crawling or monitoring service
• Join data from a browser, an HTTP API and a database in one query instead of three connectors and glue code

// How Ferret compares

Ferret alongside other open-source web scraping & crawling tools AI/TLDR tracks, ranked by GitHub stars.