What Is LLaVA? The Original Open Vision-Language Model

// In plain English

A plain large language model reads and writes text. It has never seen anything — show it a photo of a chart, a screenshot, or a handwritten note, and it has no way to make sense of the pixels. LLaVA (Large Language and Vision Assistant) was one of the first open models to fix that by teaching a text-only LLM to look at images and talk about them.

The trick behind LLaVA is almost embarrassingly practical: instead of building a brand-new model that understands both pictures and words from scratch, the authors glued together two parts that already worked well. They took an existing image encoder (a network already good at turning a photo into a compact numeric summary) and an existing chat LLM (already good at conversation), then trained a small piece in the middle to translate between them. That small translator is the whole insight.

Think of a brilliant blind professor who can reason about anything you describe, paired with a sighted assistant whose only job is to whisper what the picture actually shows in words the professor already understands. The professor does all the thinking; the assistant just converts sight into the professor's language. LLaVA's middle layer is that assistant — it converts image features into something the LLM can read as if they were words.

// Why it matters

LLaVA matters less as a model you would deploy today and more as the blueprint that almost every open vision-language model still follows. Before it, connecting sight to language usually meant expensive end-to-end training of a huge custom model. LLaVA showed you could get a capable visual chatbot by reusing two strong off-the-shelf components and training only a thin bridge — cheaply, openly, and reproducibly.

Three ideas it popularized are now standard practice.

• Reuse, don't rebuild. Freeze a pretrained vision encoder, freeze (or lightly tune) a pretrained LLM, and learn only a small connector between them. You inherit years of progress in both fields for the price of training one little layer.
• Instruction data can be synthetic. The team used a strong text-only model to generate image-based question-and-answer conversations from existing image captions and bounding boxes. That gave them a large visual-instruction dataset without armies of human labelers — a recipe the whole field adopted.
• Open and reproducible. Weights, data-generation method, and training code were public, so anyone could rebuild, extend, or specialize it. That openness is a big reason the architecture spread so fast.

Who should care? Anyone trying to understand how modern VLMs work under the hood, anyone fine-tuning an open vision model for a niche domain (medical scans, receipts, UI screenshots), and anyone choosing between open vision models — because they are mostly variations on the LLaVA recipe. Knowing the original makes every newer model easier to read.

// How it works

LLaVA has three parts in a row: a vision encoder that looks at the image, a projection layer that translates what it saw into the LLM's vocabulary of vectors, and the LLM itself that reads those translated image vectors right alongside your typed question.

Step 1 — the vision encoder turns pixels into features

The image first goes through a pretrained vision encoder (LLaVA used a CLIP-style encoder, a network trained on huge numbers of image–caption pairs). It chops the picture into a grid of patches and produces one feature vector per patch — a numeric summary capturing what is in that region. The encoder is usually frozen: its weights don't change during LLaVA training, because it is already good at seeing.

Step 2 — the projection layer translates features into tokens

Here is the heart of LLaVA. The vision encoder speaks one numeric language; the LLM expects another (the same vector space its word tokens live in). The projection layer is a small trained network that maps each image feature into that token space. The output is a handful of image tokens — vectors the LLM can consume exactly as if they were words it had read. In the first LLaVA this projector was a single linear layer; a later version upgraded it to a small two-layer network for a quality bump. Either way, it is tiny compared to the encoder and the LLM.

Step 3 — the LLM reads image tokens and text together

The projected image tokens are placed into the prompt right next to your typed question, forming one mixed sequence: [image tokens] + "What is unusual about this picture?". The LLM processes the whole thing in one pass and generates a normal text answer. From the model's point of view, the image is just an unusual stretch of tokens at the start of the conversation.

Training: two stages, mostly cheap

Because the encoder and LLM are already trained, LLaVA's own training is light. It happens in two stages.

Stage one teaches the projector to speak the LLM's language using simple image-and-caption pairs. Stage two teaches the assembled model to follow instructions about images using the synthetic question-and-answer data — the step that turns a captioner into a conversational visual assistant. That two-stage "visual instruction tuning" is the recipe LLaVA is famous for.

// The LLaVA recipe in pseudocode

You don't need the real training code to grasp the idea — the forward pass fits in a few lines. The key move is concatenating projected image tokens with text tokens before the LLM ever runs.

# Three components, two of them pretrained and (mostly) frozen.
vision_encoder = load_clip_encoder()      # frozen: already good at 'seeing'
projector      = LinearOrMLP()            # the ONLY new part we train hard
llm            = load_chat_llm()           # pretrained chat model

def answer(image, question):
    # 1) Encode the image into per-patch feature vectors.
    image_features = vision_encoder(image)         # shape: (num_patches, vis_dim)

    # 2) Project those features into the LLM's token space.
    #    Now each image patch is a vector the LLM can read like a word.
    image_tokens = projector(image_features)        # shape: (num_patches, llm_dim)

    # 3) Turn the text question into normal text tokens.
    text_tokens = llm.embed(question)               # shape: (num_words, llm_dim)

    # 4) Concatenate: image tokens FIRST, then the question.
    sequence = concat([image_tokens, text_tokens])

    # 5) The LLM reads the mixed sequence and writes a text answer.
    return llm.generate(sequence)

print(answer(my_photo, "What is written on the sign?"))

// LLaVA vs modern VLMs

Newer open vision models keep LLaVA's skeleton and improve almost every part of it. Understanding what changed makes the whole VLM landscape legible.

The pattern is clear: the shape is LLaVA's, the scale and polish are new. A model like Qwen-VL handles much higher resolution, far more training data, and tasks like dense OCR and document understanding that early LLaVA struggled with. But if you opened it up, you would still find an image encoder, a connector, and an LLM in a row. That is why people call LLaVA the ancestor: you deploy its descendants, but you can read all of them once you understand the original.

// Common pitfalls and limits

Treating LLaVA as a current production model, rather than the foundational design it is, leads to a few predictable disappointments. These limits also explain why newer models added what they did.

• Small text and fine detail. With a low fixed input resolution, the original LLaVA misses tiny text, dense tables, and crowded charts. If you need real OCR or document understanding, reach for a higher-resolution descendant.
• It still hallucinates about images. Asked about an object that isn't there, it may confidently describe one. Grounding in pixels reduces fabrication but does not remove it — the same caution that applies to any vision model's limitations applies here.
• Spatial precision is rough. Early LLaVA is good at what is in an image, weaker at exact where — precise coordinates, counting many similar objects, or reading a layout pixel-perfectly.
• The backbone caps the reasoning. LLaVA can only reason as well as the LLM it was built on. An older backbone means weaker reasoning, no matter how good the vision part is — which is exactly why newer VLMs swap in stronger LLMs.
• 'LLaVA' is a family, not one thing. Several versions and community forks exist with different encoders, connectors, and backbones. Always check which variant you are running before comparing results.

// Going deeper

Once the encoder-projector-LLM picture is solid, the interesting questions are all about how the bridge is built and how much of the model you let move during training. A few directions worth knowing.

The connector is where the design choices live. LLaVA's projector is the simplest possible bridge — a linear map, later a small MLP. Other VLM families use a resampler or a set of learned query tokens that compress hundreds of image patches into a fixed, smaller number of tokens. Fewer image tokens means cheaper inference and lets you fit higher-resolution images into the same budget — directly relevant to image token cost. The connector is a real engineering tradeoff, not an afterthought.

What you freeze decides cost and quality. Freeze everything but the projector and training is cheap but capped; also fine-tune the LLM (often with a lightweight method like LoRA) and you get better visual reasoning for more compute. Unfreezing the vision encoder too can help on unusual image domains but risks damaging its general sight. Most LLaVA-style recipes are a deliberate point on this freeze-versus-tune spectrum.

Resolution is the quiet bottleneck. Many later gains over the original LLaVA come not from a cleverer brain but from seeing more pixels — splitting a large image into tiles, encoding each, and feeding all their tokens to the LLM. That single change unlocks small text, charts, and full documents, which is why high-resolution handling became a headline VLM feature.

Where to go next. To see how all of this generalizes, read what a vision-language model is in general and how vision models actually 'see'. The durable lesson from LLaVA is architectural humility: you rarely need a new model from scratch. Connect two strong components with a small, well-trained bridge and train it on the right instruction data, and a great deal becomes possible. Nearly every open VLM you will meet is a more polished retelling of that one idea.