Overview
Framedex turns a scattered media archive into a queryable plain-text knowledge base. For every clip or photo it writes a `.description.md` sidecar next to the original containing technical metadata, GPS coordinates resolved to place names, a transcript with speaker diarization, face embeddings, and a scene description from a vision model with a keep / review / cull rating.
The design is deliberately non-destructive and local. Originals are never modified — the sidecars live alongside them — and each drive carries its own self-contained set, so an archive spread across several SSDs still works. A central face database ties identities together across drives, which is what makes cross-drive queries like "every clip with this person, rated keep, in California" possible. Because the output is Markdown, the corpus is greppable, diffable and directly usable as context for an LLM.
The pipeline chains established tools: ffprobe, exiftool and ffmpeg for media and metadata, WhisperX for word-aligned transcription, pyannote for diarization, insightface for 512-dimensional face embeddings, and Nominatim for reverse geocoding. Scene description has three vision backends — a Claude CLI mode, an Anthropic API mode, and a `local` mode via LM Studio for fully offline operation. Ratings and keywords can be exported as `.xmp` sidecars for Lightroom, and an Apple Photos library can be indexed in place without exporting it first.
What it does
- Per-file `.description.md` sidecars with metadata, GPS place names, transcript, faces and an AI scene description — originals untouched
- Speaker-diarized transcripts via WhisperX and pyannote, with translation to English for non-English audio
- Face detection and 512-dimensional embeddings (insightface) in a central database, so people can be queried across drives
- Multi-drive layout: each SSD keeps self-contained sidecars
- Three vision backends — Claude CLI, Anthropic API, or a local LM Studio model for offline use
- `fdx-query` filters by person, place, rating, keyword, language and time of day
- `fdx-xmp` exports ratings and keywords to Lightroom `.xmp`; `fdx-photos` indexes an Apple Photos library in place
Getting started
Framedex is a Python project installed with uv. It needs the ffmpeg, ffprobe and exiftool binaries on your system, a Hugging Face token for diarization, and an Anthropic API key only if you use the `api` vision backend.
Install it
Clone the repository and install with all extras, then run the setup check.
git clone git@github.com:Simbastack-hq/framedex.git ~/.claude/skills/framedex
cd ~/.claude/skills/framedex
# Install with all dependencies
uv pip install -e '.[all]'
# Verify setup
python3 scripts/setup.pySet your keys
The Hugging Face token gates the diarization models; the Anthropic key is only needed for the `api` vision backend.
export HF_TOKEN=hf_yourTokenHere
export ANTHROPIC_API_KEY=sk-ant-...Index a drive
Start with a handful of files to check the output, then run the whole drive and build the summary and master indexes.
fdx /Volumes/SSD-2024 --max-files 5
fdx /Volumes/SSD-2024
fdx-summary /Volumes/SSD-2024
fdx-master /Volumes/SSD-2024Query and export
Filter the corpus, or push ratings and keywords out to Lightroom.
fdx-query /Volumes/SSD-2024 --rating keep --place-contains California
fdx-xmp /Volumes/SSD-2024Commands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Make years of unlabelled footage findable — search by who is in a clip, where it was shot, or what was said
- Triage a shoot before editing, using the keep / review / cull ratings and exporting them to Lightroom
- Build a plain-text corpus about your own media that an LLM or agent can read directly
- Index an archive on a machine with no cloud access, using the local LM Studio vision backend
How Framedex compares
Framedex alongside other open-source parsing & ingestion tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| MarkItDown | ★ 183k | A Microsoft Python utility that converts many file types, including Office docs and PDFs, into Markdown for LLMs. |
| MinerU | ★ 79.7k | A document extraction tool that converts PDFs and Office files into clean Markdown or JSON, with strong handling of complex layouts and CJK content. |
| Docling | ★ 66.3k | An IBM-originated document conversion pipeline that turns PDF, DOCX, PPTX, HTML, and more into structured, LLM-ready Markdown or JSON. |
| Marker | ★ 39.7k | A fast pipeline that converts PDFs and other documents to Markdown, JSON, or HTML while preserving tables, equations, and formatting. |
| OfficeCLI | ★ 30.5k | A single-binary command-line Office suite built for AI agents: create, read and edit Word, Excel and PowerPoint files, render them to HTML or PNG, and drive it all over MCP. |
| OpenDataLoader PDF | ★ 29.1k | OpenDataLoader PDF turns any PDF into structured Markdown, JSON, or HTML with bounding boxes, and auto-tags untagged files into screen-reader-ready Tagged PDFs. |
| Repomix | ★ 28.3k | Repomix packs an entire repository into one file that is easy to feed to AI tools like Claude, ChatGPT, and Gemini. |
| Framedex | ★ 378 | Turn a video and photo archive into Markdown you can query |