Deep Residual Learning for Image Recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2015

Takeaway

Three things to remember

1. More layers were not automatically better. The authors showed a degradation problem: a 34-layer plain network had higher training error than an otherwise comparable 18-layer network. This was an optimization failure, not simply overfitting. Paper, §3.1 and Figure 4
2. Residual blocks change what each stack of layers must learn. Instead of directly learning a complete mapping from input to output, the stacked layers learn a change, while a shortcut carries the original input forward. The block adds the two paths together. Paper, §3.2
3. That simple change made much deeper image classifiers practical. A 152-layer residual network reached 5.71% top-5 validation error on ImageNet as a single model, and the residual-network ensemble reached 3.57% top-5 test error in the ILSVRC 2015 classification task. Paper, Tables 3 and 4

The paper’s lasting contribution is not merely “a deeper network.” It is a better optimization target: let a block preserve its input by default and learn only the useful correction.

Problem

The degradation problem

Deep networks should, in principle, be able to match shallower ones. Extra layers could simply copy the existing representation. Yet the authors found that adding layers to a plain network could raise both training and validation error. On ImageNet, the 34-layer plain model had 28.54% top-1 validation error, worse than the 18-layer plain model’s 27.94%. Because the deeper model also fit the training set less well, ordinary overfitting did not explain the result. Paper, §3.1, Figure 4, and Table 2

Visual 1 — The puzzle. Depth adds representational capacity, but the optimizer may fail to find even the copy-the-shallow-network solution. This is an original teaching diagram, not a reproduced paper figure.

Why existing fixes were not enough

Better initialization and batch normalization had already made vanishing and exploding gradients less severe. The authors argue that degradation remained: even when signals could travel through the network, the solver still struggled to use the added layers productively. Their controlled comparison therefore asks a sharper question: can the architecture make the easy identity solution easier to reach? Paper, §2 and §3.1

Method

Learn the change, not the whole answer

Suppose the desired mapping of a block is H(x). A plain stack tries to learn H(x) directly. A residual block instead asks its stacked layers to learn F(x) = H(x) − x, then returns F(x) + x. If preserving the input is already useful, the learned path only needs to move toward zero rather than recreate the identity mapping through several nonlinear layers. The paper presents this as an optimization hypothesis, supported by experiments rather than as a proof that every residual function is inherently easy to learn. Paper, §3.2 and §4.4

Visual 2 — The mechanism. The shortcut carries x unchanged; the learned path produces F(x); addition gives F(x) + x.

Identity shortcuts

The shortcut usually adds no parameters and almost no computation. When input and output dimensions match, it is simply the identity. When dimensions change, the paper tests either zero-padding or a learned projection so the two paths can be added. The authors found that all shortcut variants worked, with projection shortcuts giving a modest gain at extra cost. Paper, §3.2, §3.3, and Table 3

From basic blocks to bottlenecks

For 18- and 34-layer networks, each residual function uses two 3×3 convolutions. For the 50-, 101-, and 152-layer models, the authors use a three-layer “bottleneck”: 1×1, 3×3, then 1×1 convolutions. The 1×1 layers reduce and then restore channel width, keeping the more expensive 3×3 operation narrow. This lets the 152-layer model remain less computationally complex than the paper’s VGG-16 reference. Paper, §3.3 and Table 1

Visual 3 — Scaling the idea. Both designs keep the shortcut; the bottleneck controls computation as the stack becomes much deeper.

Evidence and limits

The cleanest test of the idea

The most persuasive result is the matched 18-versus-34-layer experiment. Architecture and training setup were held as comparable as possible; the key change was whether blocks were plain or residual.

Source: Paper, Table 2. Lower is better. The 34-layer residual model improves by 3.51 percentage points over the 34-layer plain model and by 2.85 points over ResNet-18; these differences are calculated directly from the reported values.

Results, scope, and uncertainty

The depth trend continued: single-model ImageNet top-5 validation error fell from 7.76% for ResNet-34 to 6.71% for ResNet-50, 6.05% for ResNet-101, and 5.71% for ResNet-152. The authors also report gains on ImageNet detection and localization, COCO detection and segmentation, and CIFAR-10, suggesting the representation transferred beyond one classification benchmark. Paper, Tables 3–7 and §4.2–§4.4

But the paper does not prove that residual learning always makes optimization easy, nor that increasing depth indefinitely keeps helping. On CIFAR-10, the 1,202-layer model reached 7.93% test error, worse than the 110-layer model’s 6.43%; the authors point to overfitting and note that they did not use stronger regularization such as dropout. Paper, §4.2 and Table 6 The experiments also come from the 2015 vision setting: supervised convolutional networks, specific training recipes, and benchmark datasets. Later uses of residual connections are important historically, but they are outside the evidence established by this paper.

Practical meaning

What the architecture buys you

Residual connections give every block a safe baseline: pass the current representation through, then modify it only when the learned path helps. For a practitioner, that means depth can be explored without forcing every new group of layers to relearn an identity transformation. This is an interpretation of the mechanism and results—not a guarantee that any deep model will train well. Optimization settings, normalization, data, compute, and the exact shortcut design still matter.

A useful design question

When extending a model, ask: can the new component easily behave like “do nothing”? If yes, optimization can begin from a function close to the existing system and learn an incremental correction. ResNet is the paper’s concrete realization of that idea through addition and identity shortcuts.

Terms and source notes

• Identity mapping: returns its input unchanged.
• Residual function: the change F(x) learned by the stacked layers inside a block.
• Projection shortcut: a learned transformation used when shortcut dimensions must change.
• Top-1 / top-5 error: the correct class is absent from the model’s first guess / five highest-scoring guesses.
• Primary source: He et al., “Deep Residual Learning for Image Recognition”, arXiv:1512.03385; published at CVPR 2016. Numerical claims above point to the relevant section, figure, or table in the paper PDF.

Questions worth checking when applying the idea elsewhere: Does the shortcut truly preserve compatible information? Is addition the right merge operation? Do gains come from residual structure, extra capacity, or the training recipe? And does validation performance still improve when depth, compute, and regularization are compared fairly?