What Is a Vision Language Model (VLM)? LLMs That Can See

// In plain English

A plain large language model only reads and writes text. Show it a photo, a screenshot, or a chart and it's blind — you'd have to describe the picture in words before it could say anything useful. A vision language model, or VLM, removes that wall. You can hand it an image and a question in the same prompt — "what's wrong with this error screen?" — and it answers in words, just like a normal chatbot.

The trick is that a VLM has two senses wired into one brain. It still understands and produces language like any LLM, but it has also learned to see — to turn the pixels of an image into the same kind of internal representation it uses for words. Because pictures and text now live in one shared mental space, the model can reason across both: read the label on a bottle, compare two diagrams, summarize a slide, or describe what a person in a photo is doing.

Think of a brilliant friend who, until last year, you could only talk to over the phone. You'd describe a graph and they'd reason about it sight-unseen. A VLM is that same friend finally sitting next to you, looking at the screen. You stop narrating and just point: "this one — what does it mean?" Nothing about their intelligence changed. They simply gained eyes.

// Why it matters

An enormous share of the world's information is visual, and text-only models simply couldn't touch it. Scanned invoices, dashboards, product photos, app screenshots, medical scans, handwritten notes, slide decks, charts in a PDF — none of it is plain text. VLMs unlock that entire pile.

• Document and screenshot understanding. Pull totals off a receipt, extract fields from a form, read a chart that has no underlying data, or answer questions about a dense slide. This is the single biggest commercial use of VLMs.
• Accessibility and description. Generate alt text for images, describe a scene for a blind user, or caption a photo library so it becomes searchable.
• Visual debugging and support. A user pastes a screenshot of a broken UI or a stack trace as an image, and the model reads it and explains the fix — no copy-paste of text required.
• Agents that use a computer. An AI agent can take a screenshot of the screen, see the button it needs, and click it. Sight is what lets agents operate real software built for humans.

Who should care? Anyone whose data isn't already clean text. Before VLMs, reading an image meant stitching together brittle tools — an OCR engine to grab text, a separate object detector to find things, a classifier for categories, each needing its own training data. A VLM collapses that pipeline into one model you talk to in plain English. You don't train a custom chart-reader; you ask it to read the chart. That generality — describe the task in words, get the answer — is why VLMs replaced a whole shelf of narrow computer-vision tools for everyday work.

// How it works

A VLM is built from three parts bolted together: a vision encoder that turns an image into numbers, a small projector that translates those numbers into the model's language, and the language model itself that does the reasoning and writes the answer. The clever part is that the LLM stays almost unchanged — you teach it to accept a new kind of input rather than rebuilding it from scratch.

Step 1: the vision encoder turns pixels into features

Raw pixels mean nothing to a language model. So the image first passes through a vision encoder — usually a Vision Transformer (ViT) — which slices the picture into a grid of small patches (say 16x16 pixels each) and converts each patch into a vector that captures what's there: an edge, a face, the letter "R", a red region. The encoder most VLMs build on is CLIP, a model trained on hundreds of millions of image-caption pairs so that pictures and the words that describe them land near each other in the same numeric space.

Step 2: the projector makes images speak the model's language

The encoder's vectors aren't yet in a format the language model understands. A small projector (often just a couple of neural-network layers) maps each image patch into an image token — a vector shaped exactly like the token embeddings the LLM already uses for words. Now an image is, to the model, just a sequence of "tokens" it can read alongside text. A single image typically becomes hundreds or even thousands of these tokens, which is why images eat a real chunk of your context window.

Step 3: the language model reasons over everything at once

Your text prompt is tokenized normally and the image tokens are slotted right in beside it. From here the transformer does what it always does — its attention mechanism lets every token look at every other token, so words can attend to image patches and vice versa. The model reads "what color is the car?" and the patches showing the car, and generates the answer word by word. To the LLM, vision is just more tokens in the prompt.

Getting here takes training in two stages. First, alignment: freeze the vision encoder and the LLM and train only the projector on millions of image-caption pairs, so the model learns to map pictures onto words. Then visual instruction tuning: fine-tune on examples of images paired with questions and good answers, so the model learns to follow instructions about images, not just caption them. The influential LLaVA project showed this two-stage recipe could turn a text-only LLM into a capable VLM surprisingly cheaply.

// What VLMs can and can't do

VLMs are genuinely impressive, but they have sharp, predictable edges. Knowing where they're strong versus shaky saves you from trusting an answer you shouldn't.

The pattern: VLMs are great at the gist and at language-flavored visual tasks, weaker at tasks demanding pixel-perfect precision. Counting is a classic failure — ask "how many people are in this photo?" of a crowd and you'll often get a confident wrong number. They can also hallucinate about images: describe an object that isn't there, or misread a number, with the same fluent confidence a text model uses when it makes things up. And resolution matters enormously — most models down-scale large images, so small text or fine detail can simply be lost before the model ever sees it.

// Try it in code

Using a VLM through an API is barely different from a text chat — you just add an image block to your message alongside the text. Here's a screenshot-Q&A request to Claude in Python. You can pass an image either as a public URL or as base64-encoded bytes; the model returns plain text.

from anthropic import Anthropic

client = Anthropic(api_key="sk-ant-...")

message = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=400,
    messages=[
        {
            "role": "user",
            "content": [
                # The image block — here we point at a public URL.
                {
                    "type": "image",
                    "source": {
                        "type": "url",
                        "url": "https://example.com/error-screenshot.png",
                    },
                },
                # The text block — your question about the image.
                {
                    "type": "text",
                    "text": "What error is this screen showing, and how do I fix it?",
                },
            ],
        }
    ],
)

print(message.content[0].text)

That's the whole pattern: a list of content blocks, mixing image and text in one message. You can include several images to compare them, and follow up with normal text turns — the model keeps the picture in context. The same shape works across providers, with minor differences in how the image field is named, so it's worth skimming the official provider guides for the exact format.

// Going deeper

Once the basics click, the interesting questions are about resolution, tokens, and how vision plugs into bigger systems. A few directions worth knowing.

High-resolution and the token budget. A vision encoder built for 336x336 pixels would turn a 4K screenshot into mush, so modern VLMs use tricks like tiling ("AnyRes"): split a big image into a grid of tiles, encode each at full resolution, and feed them all in. This dramatically improves reading small text and dense documents — but it multiplies the image tokens, and tokens are what you pay for in cost and latency. Resolution, accuracy, and price are a constant three-way trade.

Multimodal RAG. You can embed images into the same vector space as text and retrieve them by meaning, then feed the retrieved pictures to a VLM. Approaches like ColPali index document pages as images rather than extracted text, which preserves layout, tables, and figures that plain-text extraction destroys. If you know RAG, this is RAG with eyes — and it's a fast-moving corner of multimodal AI.

Vision as a tool for agents. Computer-use agents loop: screenshot the screen, decide an action, click or type, screenshot again. The VLM's grounding — pointing at the right pixel coordinates for a button — is the hard part, and it's an active research frontier. This is where vision stops being a party trick and becomes the sensory layer for software that operates other software, often combined with tool use and function calling.

Beyond still images. Video is just images over time, so VLMs extend to it by sampling frames — though doing this efficiently without drowning in tokens is unsolved. The same architecture also stretches to audio and other modalities, the road toward truly "any-to-any" models. And the field still wrestles with honest open problems: VLMs remain weak at precise spatial reasoning and counting, they inherit and can amplify biases in their image-text training data, and they open new attack surfaces — a prompt injection can be hidden inside an image as text the model dutifully reads and obeys. Treat any image from an untrusted source as untrusted input, not just decoration.