How to Extract Tables and Structured Data from Images

// In plain English

You have a screenshot of a pricing table, a scanned invoice, or a photo of a spreadsheet. The numbers are right there on the screen, but they're trapped in pixels — your code can't loop over them, sum a column, or load them into a database. You need to turn that picture back into real data: rows, columns, and cells you can use.

A vision-language model can do this. You send it the image plus an instruction like "return this table as JSON," and it reads the layout, recognizes the text, and hands back structured data. Unlike a traditional OCR engine, which dumps a flat stream of characters, a vision model understands that this is a table — it knows which cell belongs under which header and which row goes with which label.

Think of the difference between a photocopier and a careful assistant. A photocopier reproduces marks on a page without understanding any of it. The assistant looks at the same table and tells you, "Row three, the 'Quantity' column, says 12." This article is about getting that second behavior reliably — not just can the model read the text, but how do I get clean, validated, structured output I can trust.

// Why it matters

Enormous amounts of useful data exist only as images: scanned invoices, receipts, lab reports, bank statements, product spec sheets, screenshots from dashboards that have no export button, and PDFs that are really just pictures of pages. For decades, getting that data into a system meant either a person retyping it by hand or a brittle, rules-based OCR pipeline that broke the moment a layout changed.

Vision models change the economics. One model, prompted well, can handle invoices from a hundred different vendors with a hundred different layouts — no template per vendor, no fragile coordinate rules. That flexibility is exactly why builders care:

• Accounts payable. Read an invoice image and emit { vendor, invoice_number, line_items, total } straight into your accounting system.
• Data migration. A client hands you 5,000 scanned forms. A vision model turns them into a database table overnight instead of a temp agency over a month.
• Analytics from screenshots. A metric lives only on a vendor's dashboard with no API. Screenshot it, extract the table, track it over time.
• Document processing at scale. Receipts, shipping manifests, medical results — anywhere the source of truth is a picture of a grid.

But there's a catch that makes this harder than it looks, and it's the whole reason this article goes deeper than "can it read text." A vision model is a generative model: when it's unsure what a smudged cell says, it doesn't error out — it writes down a plausible guess. A traditional OCR engine that can't read a digit might give you a blank or a garbage character you can spot. A vision model can silently emit 1,240 where the real value was 1,248, and the output looks perfectly clean. Catching those silent errors is a core skill, and we cover it below.

// How it works

Under the hood, a vision model splits your image into patches, converts those patches into tokens, and processes them in the same sequence as your text prompt — the same machinery covered in how vision models see. From the model's point of view, the image and your instruction are one continuous input. That's why prompting matters so much here: the picture and the request are read together.

The end-to-end flow for table extraction has four stages:

The two stages that decide whether you get usable data are stage two (telling the model the exact shape you want) and stage four (verifying what came back). Most teams that struggle have skipped one or both.

Step 1 — Give the model a target schema, not a vague ask

The single biggest quality lever is telling the model the precise output structure before it answers. If you ask "extract the table," the model invents its own field names, drops columns it thinks are unimportant, and formats numbers inconsistently across calls. If you hand it a schema, you constrain it to fill in your shape.

Extract the table in this image as JSON matching exactly this shape:

{
  "rows": [
    { "item": string, "qty": integer, "unit_price": number, "total": number }
  ]
}

Rules:
- Use null for any cell that is blank or unreadable. Never guess.
- Numbers must be plain digits: no currency symbols, no thousands separators.
- Return ONLY the JSON object, nothing else.

Step 2 — Use the API's structured-output feature when it exists

Asking for JSON in prose works, but the model can still wrap it in commentary or a code fence you have to strip. Major APIs offer a stronger guarantee: tool use (also called function calling) or a JSON-schema mode, where you declare the schema as a formal spec and the model is forced to return data that fits it. The model literally cannot return a field you didn't define or omit a required one.

import base64, json
from anthropic import Anthropic

client = Anthropic()

with open("invoice.png", "rb") as f:
    img_b64 = base64.standard_b64encode(f.read()).decode()

# Declare the exact shape as a tool the model must call.
table_tool = {
    "name": "record_table",
    "description": "Record every row of the table in the image.",
    "input_schema": {
        "type": "object",
        "properties": {
            "rows": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "item": {"type": "string"},
                        "qty": {"type": ["integer", "null"]},
                        "unit_price": {"type": ["number", "null"]},
                        "total": {"type": ["number", "null"]},
                    },
                    "required": ["item"],
                },
            }
        },
        "required": ["rows"],
    },
}

msg = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=2000,
    tools=[table_tool],
    tool_choice={"type": "tool", "name": "record_table"},
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {
                "type": "base64", "media_type": "image/png", "data": img_b64}},
            {"type": "text", "text":
                "Record the table. Use null for any unreadable cell; never guess."},
        ],
    }],
)

# The result is already a Python dict in your schema — no string parsing.
table = next(b.input for b in msg.content if b.type == "tool_use")
print(json.dumps(table, indent=2))

For the mechanics of attaching the image bytes — base64 vs. URL, media types, size limits — see send images to an LLM API. For multi-page documents, send PDFs to an LLM API covers passing whole files at once.

// Verifying the output (the step everyone skips)

Getting back valid JSON is not the same as getting back correct data. The JSON can parse perfectly and still contain a hallucinated cell. Because the model never throws an error on uncertainty, you have to be the error detector. The good news: most silent mistakes are catchable with cheap, deterministic checks in your own code — no second AI call needed.

Cheap checks that catch most errors

• Row count. Count the visible rows yourself (or ask the model for a count separately) and compare. A dropped or duplicated row is the most common failure.
• Type and range checks. A qty of 99999 or a negative unit_price is almost certainly a misread digit. Flag anything outside sane bounds for review.
• Arithmetic cross-checks. If the table has qty, unit_price, and total, verify qty * unit_price == total for each row. A mismatch points straight at the misread cell. Same for a totals row that should equal the sum of its column.
• Required fields. If item is empty on a row, the model probably split or merged something wrong.
• Null rate. A sudden spike in null values (with the never-guess prompt) tells you the image quality dropped — a useful signal, not a failure.

When the stakes are high

For critical pipelines, add a confirmation layer. Ask the model to return a per-cell confidence flag, or run the same image twice (or through two different models) and compare — cells that disagree go to a human. This costs more, but routing only the uncertain rows to a person is far cheaper than checking every row, and far safer than checking none.

// Handling tricky layouts

Clean, simple grids are easy. Real-world tables fight back. Here's how the common hard cases behave and what to do about each.

The throughline: the cleaner and more explicit the input, the better the output. A tightly cropped, upright, readable image of a single table beats a wide-angle photo of a whole page every time. Preprocessing the image is often a bigger win than tweaking the prompt.

// Why free-text extraction drifts (and schema fixes it)

It's worth understanding why a target schema helps so much, because it explains a class of bugs that confuse beginners. When you ask for output in free text, two runs on the same image can disagree:

A generative model picks the next token from a probability distribution, so an open-ended request leaves many equally-likely ways to format the answer. Each run can take a different path. A schema collapses that freedom: there's exactly one valid place for each value, so the model spends its effort reading the cells instead of deciding how to present them. You convert an unstable creative task into a stable fill-in-the-blanks task — which is what you actually wanted.

// Going deeper

Once the basic extract-and-validate loop works, a few directions are worth knowing as you move toward production.

Vision model vs. dedicated OCR vs. hybrid. Vision models shine at understanding — messy layouts, mixed languages, inferring which header a value belongs to. Classic OCR engines (and document-understanding models) can be cheaper and more deterministic for clean, high-volume forms. A strong production pattern is hybrid: run fast OCR to get raw text and coordinates, then let a vision model assemble the structure and resolve ambiguity. The broader tradeoff is covered in LLM OCR and document understanding.

Cost and resolution. Bigger images mean more tokens and more money per call, but shrinking the image too far makes small digits unreadable. There's a sweet spot per document type — find it by measuring accuracy at a few resolutions, not by guessing. Cropping to just the table region both cuts cost and raises accuracy, since the model isn't spending tokens on margins and logos.

Batching and idempotency. For thousands of documents, process them through a queue, store the raw image alongside the extracted JSON, and keep the model id and prompt version with each result. When you improve the prompt, you can re-run only the low-confidence items. Treat extraction as reproducible: same image plus same prompt plus same model should give you a record you can audit later.

The honest limits. No vision model reads a smudged, low-resolution, badly-scanned table at 100% accuracy — neither does a human. The realistic goal is not zero errors but zero silent errors: every cell the model is unsure about should be flagged, not invented. Get that right with the never-guess prompt plus arithmetic and row-count checks, and you can safely automate the easy 95% while routing the hard 5% to a person. That blend — automate the confident cases, escalate the uncertain ones — is how reliable document pipelines are actually built.