Denoising Diffusion Probabilistic Models

Jonathan Ho, Ajay Jain, Pieter Abbeel · NeurIPS 2020 · Paper · PDF

Takeaway

Three things to remember

1. Generation becomes a denoising problem. The model learns to undo small amounts of Gaussian noise, then creates an image by repeating that learned step from pure noise.
2. Predicting the added noise is the useful simplification. A U-Net receives a noisy image and its noise level; training asks it to recover the exact noise that was added. The paper connects this objective to denoising score matching and a Langevin-like sampling process.
3. The quality result was strong, but generation was slow. On unconditional CIFAR-10, the best model reports FID 3.17 and Inception Score 9.46. Sampling uses 1,000 sequential network evaluations, so the paper establishes image quality more clearly than practical speed.

The paper’s central change is simple to say: instead of asking one network pass to invent an image, train a network on many levels of corruption and let it build an image through many small corrections. [Source: abstract; Sections 1, 3.2, and 4]

Problem

Why image generation needed another route

By 2020, GANs could make sharp images but trained through a competition between two networks; autoregressive models generated one ordered piece at a time; and likelihood-based approaches often traded sample quality against tractable probability estimates. Diffusion models already had a clean probabilistic definition, but the authors say they had not yet been shown to produce high-quality samples. Their question was whether this slow, step-by-step family could become a serious image generator. [Source: Section 1]

Turn destruction into supervision

The forward process is fixed rather than learned: start with a real image and add a small amount of Gaussian noise at each step until almost no signal remains. Because the corruption rule is known, training can create unlimited pairs of a noisy image and the exact noise used to make it. The hard problem—creating structure from randomness—is approached by learning the reverse of an easy, controlled destruction process. [Source: Section 2, Equations 2 and 4]

Method

Learn the noise at a random time

For each training example, the algorithm picks a real image, samples a time step uniformly, samples Gaussian noise, and directly constructs the image at that noise level. The U-Net sees the noisy image plus the time step and predicts the sampled noise. Training minimizes squared error between the true and predicted noise. Because any time step can be sampled directly, training does not need to run through all earlier corruption steps first. [Source: Equation 4, Algorithm 1, and Equation 14]

Reverse the chain to generate

At generation time, the model starts from standard Gaussian noise. It predicts a noise component, removes the appropriate amount, adds the variance required by the reverse transition, and repeats from step 1,000 down to step 1. The final step returns an image. The authors show that this parameterization resembles denoising score matching across noise levels and a finite Langevin-like sampling chain. [Source: Section 3.2, Equation 11, and Algorithm 2]

Why the simplified loss matters

The exact variational bound gives different weights to different noise levels. The paper’s best-sampling model drops those weights and uses an unweighted noise-prediction loss, called Lsimple. The authors interpret this as shifting attention away from nearly clean images and toward harder denoising tasks. In their ablation, this choice—not diffusion alone—drives the large quality improvement. [Source: Section 3.4 and Table 2]

Evidence and limits

What the experiments establish

On unconditional CIFAR-10, Lsimple reaches FID 3.17 and Inception Score 9.46 ± 0.11. The corresponding FID against the test set is 5.24; the headline 3.17 is computed against the training set, following the convention used by the paper. On 256×256 LSUN, the reported FIDs are 7.89 for Church and 4.90 for Bedroom, with sample quality described as similar to ProgressiveGAN. [Source: Table 1, Section 4.1, Figures 3–4]

The ablation makes the mechanism more credible. With fixed variance and the full variational objective, noise prediction gives FID 13.51; pairing noise prediction with Lsimple improves it to 3.17. Learned diagonal variance was unstable in the tested settings, and predicting the reverse mean with an unweighted squared-error objective produced poor samples. [Source: Table 2 and Section 4.2]

What remains costly or uncertain

The method needs 1,000 neural-network evaluations per sample in these experiments. On the reported TPU v3-8 setup, a batch of 256 CIFAR-10 images took 17 seconds, while a batch of 128 images at 256×256 took 300 seconds. The paper does not test text conditioning, controllable editing, human preference, robustness outside its datasets, or deployment-scale latency. Its lossless likelihood also trails stronger likelihood-based models: the best-sampling CIFAR-10 model has an upper-bound NLL of 3.75 bits/dimension. [Source: Table 1; Section 4; Appendix B]

The progressive compression result is an interpretation, not a ready codec. The appendix states that its assumed random-coding procedure is not tractable in high dimensions. The authors also warn that generative models can ease the creation of deceptive media and reproduce or reinforce biases present in training data. [Source: Appendix note before Table 4; Broader Impact]

Practical meaning

A reusable design pattern

The durable idea is to replace one difficult prediction with a curriculum of controlled corruption levels and a shared correction model. That pattern is attractive when corruption is easy to simulate, the correction target is known, and repeated refinement is acceptable. For a product team, the paper suggests separating three questions: can the model learn a reliable local correction, does repeated correction produce globally coherent outputs, and is the number of steps affordable?

Coarse structure arrives before detail

The reverse process behaves like progressive decoding: large-scale image structure appears early and fine detail appears late. On CIFAR-10, the highest-quality model’s variational bound is split into a 1.78 bits/dimension rate and 1.97 bits/dimension distortion, corresponding to 0.95 RMSE on a 0–255 scale. The authors argue that more than half of the lossless code length is spent on visually imperceptible differences. This helps explain how a model can have excellent-looking samples without leading likelihood scores. [Source: Section 4.3 and Table 4]

Reading notes and verification questions

• Gaussian noise: random pixel-level perturbation drawn from a normal distribution.
• Noise schedule: the preset amount of noise added at each forward step; the experiments increase it linearly from 0.0001 to 0.02.
• Score matching: learning a direction in image space that points toward more likely data; the paper shows how its noise predictor corresponds to this idea across noise levels.
• FID: a distance between feature distributions of real and generated image sets; lower is better, but the reference split matters.
• Check before reusing the result: Which dataset split defines FID? How many denoising steps are used? Is the goal visual quality, likelihood, compression, or latency? Does a claimed codec include a tractable coding method?

These notes keep author claims separate from later product interpretation. All numerical claims above come from the paper’s v2 PDF; practical implications are explicitly framed as interpretation.