Vision Model Limitations: Where VLMs Quietly Fail

// In plain English

A modern vision-language model can look at a photo and describe it, read a screenshot, or answer questions about a chart. It feels like the model truly sees. But a VLM doesn't perceive an image the way your eye does. It chops the picture into a grid of patches, turns each patch into a handful of numbers (tokens), and reasons over that compressed summary. A lot of fine detail never survives that step.

Think of describing a crowded street scene to a friend over a bad phone line. You get the gist across — "busy market, lots of stalls, sunny" — but if they ask exactly how many oranges are in the third stall, or what the tiny price tag says, you're guessing. The model is in the same spot. It has a vivid impression of the image, not a pixel-perfect copy it can re-inspect.

So the failures in this guide are not random bugs. They are predictable consequences of how images get squeezed into tokens. Once you know where the line is, you can design around it instead of being surprised by it.

// Why it matters

The dangerous thing about VLM failures is that they are confident. The model rarely says "that text is too small for me to read." It returns a clean, fluent, authoritative answer — that happens to be wrong. A wrong-but-confident count or a misread invoice number looks exactly like a correct one until someone checks.

That matters most when the image carries a decision:

• Document and data extraction. Misreading a single digit on an invoice, a meter, or a lab result turns an automation into a liability.
• Inventory, safety, and compliance. "How many people are wearing hard hats?" or "count the pills in this blister pack" sounds simple and is exactly the kind of counting VLMs get wrong.
• Accessibility and UI automation. A model driving a screen has to know which button is where; small spatial mistakes cascade into clicking the wrong thing.
• Anything you bill or ship on. If the model's reading feeds a refund, a diagnosis, or a legal record, an unflagged error is expensive.

Builders who treat a VLM as a perfect eye ship brittle systems. Builders who know its blind spots add a crop here, a second pass there, an "are you sure?" check, and a human in the loop on the high-stakes 1%. The difference between those two products is entirely this knowledge.

// Why these failures happen: the tokenization bottleneck

Almost every recurring weakness traces back to one design choice. A VLM cannot feed millions of raw pixels into the language model — that would be impossibly expensive. Instead an image encoder downscales the picture to a fixed working resolution, splits it into a grid of patches, and emits a bounded number of visual tokens. The language model then reasons over those tokens, never the original pixels. (For the full picture, see how vision models see.)

Two facts about that pipeline explain most of the failures:

• Resolution is capped. A huge image is shrunk to fit a maximum size before tokenizing. Fine structures — tiny fonts, thin lines, distant objects — can blur into a few pixels or vanish entirely. The model literally cannot read what the encoder threw away.
• Tokens summarize, they don't store. Each patch becomes a feature vector capturing what's roughly there, not an exact pixel record. The model has a strong sense of texture and gist but a weak sense of precise position, exact quantity, and small high-frequency detail.

Add to that how these models were trained. They learned mostly from image–caption pairs scraped from the web. Captions describe scenes ("a dog catching a frisbee in a park") far more than they enumerate ("four dogs, two on the left") or transcribe tiny text. So the model is brilliant at gist and weak at precision — exactly because that's what its training data rewarded.

// The failure catalogue

Here are the failures you'll meet most, what triggers each, and the practical fix. None requires retraining the model — they're all about how you frame the task.

Notice the pattern: every fix either gives the model more pixels where it matters (cropping, tiling, higher resolution) or lets the model admit uncertainty (allowing "can't tell", asking it to list before counting). Those two levers solve the large majority of real-world VLM problems.

// A worked example: making counting reliable

Counting is the textbook VLM failure, so it's a perfect case study. Suppose you ask a model to count items in a busy shelf photo. The naive prompt — "How many bottles are in this image?" — invites a single confident guess with no way to check it.

Three changes make the same model far more reliable: force it to enumerate before it counts, let it flag uncertainty, and crop when the scene is dense. Enumerating works because listing each item one by one pushes the reasoning into the text the model is actually good at, instead of a single holistic glance.

from anthropic import Anthropic

client = Anthropic()

PROMPT = (
    "Count the bottles on the shelf.\n"
    "Do NOT answer with a single number first.\n"
    "1) List each bottle you can clearly see, one per line, "
    "   with its rough location (e.g. 'top row, 2nd from left').\n"
    "2) Mark any item you are unsure about as (uncertain).\n"
    "3) Then give the total count of CLEARLY visible bottles, "
    "   and a separate count of uncertain ones.\n"
    "If the image is too low-resolution to be sure, say so."
)

msg = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=800,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {
                "type": "base64",
                "media_type": "image/jpeg",
                "data": image_b64,  # a tight crop of the shelf
            }},
            {"type": "text", "text": PROMPT},
        ],
    }],
)
print(msg.content[0].text)

The separate "uncertain" count is the real win. Now your downstream code can branch: if uncertainty is zero, trust the number; if it's high, route the image to a human or a dedicated object-detection model. You've turned a silent error into a visible signal. (For the request format itself, see sending images to an LLM API.)

// When not to use a VLM (and what to use instead)

Sometimes the right call is to not hand the job to a general vision model at all. A quick decision guide:

A common and strong pattern is hybrid: let a specialist tool do the precise part and the VLM do the reasoning part. For document work, run a dedicated OCR engine to extract exact text, then feed that text plus the image to the VLM so it can interpret the layout and answer questions — see LLM OCR and document understanding. You get the detector's precision and the model's reasoning, instead of forcing one tool to do both jobs badly.

There's also a cost angle. High-resolution images and tiling both spend more tokens. If you're tiling a page into nine crops to read small text, you're paying roughly nine images' worth of image tokens. Sometimes a cheaper dedicated OCR call is both more accurate and less expensive than brute-forcing resolution through a VLM.

// Going deeper

Once the basic failure modes click, a few subtler points separate a robust system from a demo.

Tiling has its own blind spot. Splitting a big image into tiles fixes resolution, but each tile is encoded in isolation. An object straddling two tiles can get counted twice or missed, and the model loses the global context that told it what it was looking at. Overlap your tiles slightly, and re-stitch results carefully rather than trusting per-tile counts to simply add up.

"Lost in the middle" applies to vision too. Just as text models recall facts at the edges of a long prompt better than the middle, VLMs attend unevenly across an image. Salient, central, high-contrast regions get more reliable attention than faint detail in a corner. If the critical content is a small mark in the periphery, crop it to the center.

Hallucination is worse where information is scarce. A VLM fills gaps with priors learned from training. Blurry regions, blank areas, and ambiguous shapes are exactly where it invents plausible content — a price on an unreadable tag, a face in noise. The fix is structural: always offer the model an explicit escape hatch like "reply 'not visible' if you cannot clearly read it," or it will feel pressured to produce something.

Evaluate on the hard cases, not the easy ones. A model that nails clean stock photos can collapse on your real, messy, low-light, skewed inputs. Build a small test set from genuinely difficult examples — tiny text, dense scenes, edge crops — and measure there. "It worked on three nice screenshots" is not evidence it works.

The frontier keeps moving — the failure shape doesn't. Newer models read smaller text and count better than older ones, and resolution limits keep rising. But the causes here are baked into how images become tokens, so the same weaknesses keep reappearing in milder form. Design for them as permanent properties to manage, not bugs that the next release will fully erase. To ground all of this in the bigger picture, revisit what a multimodal AI is.