open-parse

// Overview

Open Parse is a Python library for chunking documents before they go into a RAG system. Instead of flattening a file into raw text and slicing it, it looks at the visual layout of a document, headings, sections, tables and images, and groups related content together so each chunk stays coherent.

It is aimed at developers building retrieval pipelines who find that naive text splitting loses the structure of their source files. It works on PDFs out of the box using pdfminer.six, with optional table detection and OCR support, and an optional semantic pipeline that clusters nodes by embedding similarity.

Within the parsing and ingestion space, Open Parse sits between plain text splitters and heavier ML layout parsers: it handles layout, tables and markdown in one library, and uses pydantic models so you can serialize parsed results to dicts or JSON.

// What it does

• Visually analyzes document layout instead of splitting on raw text, so chunks follow the document's real structure
• Basic markdown support for headings, bold and italics
• High-precision table extraction into clean Markdown, with optional deep-learning models (Table Transformer, unitable)
• Optional semantic ingestion pipeline that embeds and clusters related nodes (e.g. via OpenAI embeddings)
• Built on pydantic, so parsed results serialize to a dict or JSON
• Extensible post-processing and optional OCR via Tesseract

// Getting started

Install the core library with pip, then parse a PDF into nodes. ML table detection and a semantic pipeline are optional add-ons.

pip install openparse

import openparse

basic_doc_path = "./sample-docs/mobile-home-manual.pdf"
parser = openparse.DocumentParser()
parsed_basic_doc = parser.parse(basic_doc_path)

for node in parsed_basic_doc.nodes:
    print(node)

parsed_content.dict()

# or to convert to a valid json dict
parsed_content.json()

pip install "openparse[ml]"
openparse-download

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Preparing PDFs for a RAG pipeline so retrieval returns coherent, structurally-grouped passages instead of arbitrary text slices
• Extracting tables from reports or manuals into clean Markdown for downstream LLM input
• Building a semantic ingestion step that clusters related document sections by embedding similarity
• Keeping document parsing in-house instead of sending data to a commercial document-AI vendor

// How open-parse compares

open-parse alongside other open-source parsing & ingestion tools AI/TLDR tracks, ranked by GitHub stars.