Visual Instruction Tuning

Authors: Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee
Published: NeurIPS 2023 (Oral); arXiv v2, 11 December 2023
Primary source: Paper abstract · Full paper (PDF)
Reading note: Results and limitations below refer to this 2023 paper and its LLaVA setup, not later LLaVA versions.

Takeaway

LLaVA shows a practical way to turn a language model into a visual assistant: connect it to an image encoder, then train it on image-based instructions rather than captions alone. The clever part is the data pipeline. A text-only GPT-4 never sees the original image; it receives captions and object locations as a textual stand-in and writes conversations, descriptions, and reasoning examples. Those examples teach the new model how people ask about images. Paper, Sections 3–4

Three things to remember

1. The main contribution is instruction data, not just a new connector. The authors turn existing image annotations into 158,000 instruction-following examples.
2. A simple bridge makes vision usable by a language model. A linear projection converts CLIP image features into token-like vectors that Vicuna can process.
3. The evidence is promising but partly circular. LLaVA scores well on visual chat and ScienceQA, yet GPT-4 helps create the data and also judges a central benchmark.

Problem

Captions are not conversations

Before this work, many vision-language models learned from image–caption pairs. That can teach what is in an image, but not necessarily how to follow varied requests such as “explain what is unusual,” “describe the scene in detail,” or “reason about what may happen next.” Text assistants had already benefited from instruction tuning; comparable image-and-language instruction data were scarce and expensive to collect. Paper, Introduction and Section 3

The paper's change

The authors reformat image understanding as assistant behavior. For each COCO image, they give a text-only teacher captions and bounding boxes, which carry object names and locations. GPT-4 then produces three kinds of supervised examples: multi-turn conversation, detailed description, and complex reasoning. The resulting LLaVA-Instruct dataset contains 58,000 conversation, 23,000 description, and 77,000 reasoning examples—158,000 in total. Paper, Section 3

The teacher receives symbolic text about an image, not the image itself. This makes the pipeline scalable, but its answers can only be as grounded as those captions and boxes.

Method

A bridge between two pretrained models

LLaVA combines a frozen CLIP ViT-L/14 vision encoder with Vicuna, a conversational language model. The image encoder produces a grid of visual features. A trainable linear projection maps those features into the same vector space as Vicuna's word embeddings. The projected visual tokens and the user's text instruction then enter Vicuna together, and Vicuna generates the answer. Paper, Section 4.1

Plain-language analogy: the projection is a small translator. It does not interpret the whole scene by itself; it changes the image encoder's output into a form the language model can use.

Two-stage training

Training first aligns the modalities, then teaches assistant behavior. In stage 1, the authors use 595,000 filtered CC3M image–caption pairs. Both CLIP and Vicuna stay frozen; only the projection learns, effectively becoming a “visual tokenizer” for the language model. In stage 2, CLIP remains frozen while the projection and Vicuna are updated on the 158,000 visual-instruction examples. For the general chatbot, the three response types are sampled uniformly. Paper, Section 4.2

Separating alignment from instruction tuning preserves the pretrained image encoder while giving the language model a chance to adapt to visual dialogue.

Evidence and limits

What the experiments show

The strongest causal evidence is the data ablation on LLaVA-Bench (COCO). On 90 generated questions over 30 COCO images, the model without instruction tuning scored 21.5 relative to the GPT-4 reference; the full instruction dataset scored 85.1. Conversation-only training reached 73.8, while adding description and reasoning examples improved the overall result. This supports the paper's central claim that the kind of training data matters, although the benchmark is small and generated through the same broad pipeline being tested. Paper, Table 4

ScienceQA adds a more conventional quantitative test: 21,000 multimodal multiple-choice questions across natural science, social science, and language science. LLaVA alone achieved 90.92%. When text-only GPT-4 chose between its answer and LLaVA's answer when they disagreed, the ensemble reached 92.53%. The paper also reports that skipping alignment pretraining reduced LLaVA from 90.92% to 85.81%, a 5.11-point drop. Paper, Tables 7–8

What remains uncertain

The visual-chat scores are not independent human judgments: GPT-4 generated much of the instruction data and graded model answers against a GPT-4 reference that received ground-truth captions and boxes. This does not invalidate the comparison, but it can favor GPT-4-like phrasing and reasoning. LLaVA-Bench (COCO) is also only 90 questions, so it cannot establish broad real-world reliability. ScienceQA is stronger evidence for a bounded multiple-choice task, but the best 92.53% result belongs to an ensemble with GPT-4, not LLaVA alone.

The authors explicitly warn about hallucinated details, inherited bias from CLIP and Vicuna/LLaMA, harmful inputs, energy costs at larger scale, and the difficulty of evaluating fine-grained visual grounding. They describe the work as an initial step focused mainly on real-life tasks, not proof of safe general-purpose visual understanding. Paper, Broader Impact Paper, Conclusion

Practical meaning

Why this mattered

The reusable idea is a data recipe: if a capable model cannot directly consume a modality, structured descriptions of that modality can still help it create supervision for a student that can. Combined with a lightweight projection layer, this offered researchers a relatively direct path from pretrained components to an interactive multimodal assistant. That is an interpretation of the method's product value, not a claim that the paper proved deployment readiness.

How to use the lesson carefully

For builders, separate three questions. First, does the connector preserve enough visual detail? Second, does instruction data cover the requests users will actually make? Third, is evaluation independent of the teacher that created the data? The paper answers the first two with useful early evidence; the third remains only partly answered. A production system would still need human evaluation, tests for hallucinated visual details, subgroup and safety checks, and benchmarks drawn from the intended setting.

Verification questions for a close read

• Would the gains remain under human judges who do not know which model produced each answer?
• How much quality comes from GPT-4's instruction data versus the model architecture or base models?
• What errors arise when captions or bounding boxes omit the detail needed for a generated answer?
• Does performance hold on dense documents, medical images, diagrams, or other domains far from COCO?