An Image is Worth 16x16 WordsTransformers for Image Recognition at Scale

Authors: Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby
Published: ICLR 2021; arXiv version 2, 3 June 2021
Primary source: arXiv:2010.11929 · Full paper

Takeaway

Vision Transformer (ViT) asks a wonderfully simple question: what if an image classifier used an ordinary Transformer, with pieces of an image standing in for words? The answer in this paper is yes—but only once pretraining becomes very large. ViT removes most image-specific machinery, scales efficiently, and transfers well; on smaller training sets, its lack of built-in visual assumptions becomes a weakness. [Paper: abstract; §§1, 4.3]

Three things to remember

1. An image becomes a sentence of patches. ViT cuts the image into fixed-size squares, turns each square into a vector, adds position information, and processes the sequence with a standard Transformer encoder. [Paper: §3.1, Figure 1]
2. Scale replaces some built-in visual rules. Comparable ResNets do better when pretraining data is limited; ViT catches up and passes them as the dataset grows. [Paper: §§1, 4.3, Figures 3–4]
3. The win is a transfer-learning result, not “Transformers always beat CNNs.” The strongest model was pretrained on the private JFT-300M dataset, then fine-tuned on smaller benchmarks. [Paper: §§4.1–4.3]

Visual 1. The core mechanism. The round token represents the learned classification token; the paper uses its final state as the image representation. Original teaching visual based on §3.1 and Figure 1.

Problem

Why not feed pixels directly to attention?

Self-attention compares every item with every other item, so its cost grows roughly with the square of the sequence length. A normal image contains far too many pixels for naïve all-to-all attention. Earlier vision systems therefore used convolutional networks, mixed convolution with attention, or restricted attention to local or sparse patterns. Those choices worked, but specialized attention patterns could be harder to run efficiently on contemporary accelerators. [Paper: §§1–2]

What CNNs give you for free

A CNN begins with useful assumptions: nearby pixels matter together, the two-dimensional neighborhood is important, and the same detector can work after an object shifts position. These are called inductive biases—rules built into the architecture before it sees data. ViT keeps very little of that structure beyond cutting the image into patches and adjusting positions when image resolution changes. It must learn most spatial relationships from examples. [Paper: §§3.1–3.2]

Visual 2. The central tradeoff: stronger built-in visual structure helps a CNN with less data; ViT leans more heavily on large-scale learning. This is a conceptual explanation, not a measured chart. Original teaching visual based on §§3.1 and 4.3.

Method

Turn patches into tokens

For an image of height H, width W, and patch size P × P, ViT creates N = HW/P² patches. Each flattened patch is mapped through a learned linear projection to a fixed-length patch embedding. Smaller patches preserve finer detail but make the sequence longer and attention more expensive. Names such as ViT-L/16 mean the Large model using 16×16-pixel input patches. [Paper: §§3.1, 4.1; Table 1]

Add position and a classification token

Because a plain Transformer does not know where a patch came from, ViT adds a learned position embedding to every patch embedding. It also places one learned classification token at the front of the sequence. After the encoder, that token's state becomes the whole-image representation used by the classifier. The paper found no meaningful gain from more elaborate two-dimensional position embeddings in its tests. [Paper: §3.1; Appendix D.4]

Pretrain, then adapt

The resulting sequence passes through the standard Transformer encoder: alternating multi-head self-attention and small feed-forward networks, with normalization and residual connections. The model is first trained on a large labelled image collection. For a downstream task, the original prediction head is replaced and the whole model is fine-tuned. Higher-resolution fine-tuning keeps the patch size fixed and interpolates the learned position embeddings onto the larger patch grid. [Paper: §§3.1–3.2]

Visual 3. ViT's practical lifecycle—and its main cost. The evidence in this paper comes from supervised pretraining followed by task-specific fine-tuning. Original teaching visual based on §§3.2 and 4.1.

Evidence and limits

What the experiments show

The study pretrained models on ImageNet (1.3 million images), ImageNet-21k (14 million), and the private JFT dataset (303 million). It then evaluated transfer on ImageNet, ImageNet-ReaL, CIFAR-10/100, Oxford-IIIT Pets, Oxford Flowers-102, and VTAB's 19 tasks; each VTAB task used 1,000 training examples. [Paper: §4.1]

The headline JFT-pretrained ViT-H/14 reached 88.55% ImageNet accuracy, 90.72% on ImageNet-ReaL, 94.55% on CIFAR-100, and 77.63% average accuracy across VTAB. In the paper's comparison, it used 2,500 TPUv3-core-days of pretraining, versus 9,900 for the BiT-L convolutional baseline. These figures are averages over three fine-tuning runs where the paper reports them, and they combine architecture choices with training choices. [Paper: Table 2; §4.2]

The more revealing result is the crossover. On ImageNet-scale pretraining, ResNets beat ViT and larger ViTs could overfit. On random JFT subsets, ViT-B/32 was worse than ResNet-50 at 9 million images but better at 90 million or more. In the controlled JFT scaling study, ViTs offered the better performance-versus-compute tradeoff; hybrid CNN–Transformer models helped at small model sizes, but the gap faded at larger sizes. [Paper: §§4.3–4.4; Figures 3–5]

What remains uncertain

• Huge data is part of the method. The strongest result relies on JFT-300M, which is private, so reproducing the headline setup is difficult. The public ImageNet-21k route worked well, but the paper says it still required about 30 days on an eight-core cloud TPUv3. [Paper: §§4.1–4.2]
• This is mainly image classification. The paper does not establish comparable results for detection or segmentation; it names those as future work. [Paper: §5]
• Self-supervision was preliminary. Masked-patch pretraining gave ViT-B/16 79.9% ImageNet accuracy—2 percentage points above training from scratch, yet 4 points below supervised pretraining. [Paper: §4.6]
• Attention pictures are not full explanations. The analysis shows that some early heads attend globally and learned positions recover row-and-column structure, but it does not prove why each prediction is made or guarantee robustness. [Paper: §4.5; interpretation]
• The efficiency comparison has caveats. The authors note that optimizer, schedule, weight decay, and other training choices can affect pretraining efficiency, not just architecture. [Paper: §4.2]

Practical meaning

When this idea is useful

ViT is most compelling when an organization can pretrain once—or reuse a pretrained checkpoint—and adapt the representation to many recognition tasks. Its simple, standard Transformer backbone also makes improvements in Transformer tooling easier to carry over from language to vision. The durable idea is not merely “use attention”; it is represent an image as tokens, learn broadly at scale, then transfer. [Paper: §§1, 3, 5; interpretation]

Questions to ask before adopting it

1. Do you have enough relevant data and compute, or a suitable pretrained model?
2. Is the job image classification, or are you assuming the result transfers to detection, segmentation, video, or another setting?
3. Are comparisons controlled for pretraining data, training recipe, resolution, and compute?
4. Does a smaller CNN or hybrid meet the same product need more cheaply?
5. What happens on distribution shifts, rare classes, and real-world failure cases not covered by the paper's benchmarks?

Source notes and terms

• Primary evidence: Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, arXiv:2010.11929v2 / ICLR 2021. Section and table labels above refer to this version.
• Transformer: a neural network that lets sequence items exchange information through self-attention.
• Patch embedding: a learned vector representation of one image patch.
• Fine-tuning: further training a pretrained model for a smaller, specific task.
• Uncertainty: the paper reports benchmark accuracy and pretraining compute, but does not provide a full accounting of data creation, energy, inference cost, or deployment behavior.