Knowhere

// Overview

Knowhere is a document parsing and retrieval system for agent workloads. Its premise is that conventional OCR and document-intelligence pipelines make perfect element-by-element extraction a prerequisite for retrieval — and on dirty PDFs and slide decks, errors in reading order, layout, tables or hidden text layers accumulate into unreliable context. Knowhere instead parses along two tracks: a Text Track that preserves native structure where it is reliable, and a Vision Track where frontier vision models understand a PDF page or a PowerPoint slide as a whole.

Both tracks converge on the same chunk and metadata schema, so everything downstream — storage, hierarchy, graph construction, retrieval — is format-independent. What a run produces is not a pile of chunks but navigable memory: section nodes that keep document paths, page ranges, summaries, entities and linked assets, plus a cross-document graph built from page-derived typed entities and keywords. Images and tables are extracted, summarised with a VLM and linked back to the section they came from, and results retain the source document, section path, page number and rendered visual evidence for citation.

Retrieval is deliberately agent-neutral. Knowhere exposes one corpus schema and a set of hierarchy-aware tools — document outlines, structural filters, exact search, fuzzy recall, full reading, assets and cross-document relationships — and lets the agent decide which to call and how deep to traverse, rather than forcing every query through a fixed pipeline. The same contract is available to built-in agents and to external ones over MCP, and classic top-K retrieval remains available when a deterministic lookup is what you want. The project is model-agnostic, with the API, worker, dashboard, Docker stack and Python/Node SDKs all published under Apache-2.0.

// What it does

• Dual-track parsing — a Text Track for reliable native structure, a Vision Track for complex PDFs and .pptx understood page-by-page by a vision model
• One hierarchy-native memory schema shared by both tracks: section paths, page ranges, summaries, entities, linked assets
• Cross-document memory graph built from page-derived typed entities and keywords across a namespace
• Agent-native retrieval: outlines, structural filters, exact search, fuzzy recall, full reading and relationship tools, callable over MCP
• Page-grounded citations that resolve back to documents, sections, page numbers and rendered visual evidence
• VLM-assisted image and table handling, linked back to their source section nodes
• Model-agnostic across OpenAI, Qwen, GLM, DeepSeek, Volcengine and other compatible providers
• Self-hostable stack (API, worker, dashboard, Docker Compose) plus official Python and Node SDKs for the hosted API

// Getting started

Running the backend locally needs Python 3.11+, uv, and Docker with docker compose. Supported inputs are .pdf and .pptx (Vision Page via the V2 Jobs API), .doc/.docx/.xls/.xlsx, images, and .md/.txt/.html/.json.

uv sync --all-packages

cp apps/api/.env.example apps/api/.env
cp apps/worker/.env.example apps/worker/.env

./deploy/local-dev/start-dev.sh

cd apps/api && uv run main.py
cd apps/worker && uv run worker.py

# when the schema changed
cd apps/api && uv run alembic upgrade heads

cd apps/api
uv run scripts/init_user.py --email you@example.com

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Reach for it when your corpus is PDFs and slide decks that ordinary parsers mangle — scans, atlases, drawing collections, dense layouts
• Reach for it when answers must be citable back to a page, not just to a chunk
• Reach for it to give an agent a document graph it can navigate like a reader instead of a flat vector lookup
• Reach for it when several agents or frameworks need to share one corpus contract over MCP

// How Knowhere compares

Knowhere alongside other open-source parsing & ingestion tools AI/TLDR tracks, ranked by GitHub stars.