Auto-Encoding Variational Bayes

Diederik P. Kingma and Max Welling · arXiv:1312.6114 · submitted 20 December 2013, revised version available on arXiv
Primary source: arXiv abstract · paper PDF

Takeaway

The paper makes learning with hidden, continuous variables practical by turning a troublesome random sampling step into an ordinary differentiable computation. Its Auto-Encoding Variational Bayes (AEVB) algorithm then trains two models together: a recognition model that quickly proposes likely hidden explanations for an observation, and a generative model that tries to recreate observations from those explanations. [Source: abstract; §1, p.1]

Three points to remember:

1. Move the randomness. Draw parameter-free noise first, then transform it using learned parameters. Gradients can pass through that transform. [Source: §2.3–2.4, pp.3–4]
2. Learn how to infer. One recognition network replaces a fresh, iterative inference procedure for every datapoint. This is now commonly called amortized inference. [Source: §1 and §2.1, pp.1–2; “amortized” is later terminology]
3. Optimize one useful compromise. The variational lower bound rewards reconstruction while keeping the approximate hidden-code distribution close to the prior. [Source: Eq. 3 and Eq. 7, pp.3–4]

Problem

Hidden causes are hard to recover

A directed generative model says that an unseen variable $z$ helps produce an observation $x$. Learning such a model requires reasoning backward—from $x$ to plausible values of $z$. But the exact posterior $p_\theta(z\mid x)$, the marginal likelihood $p_\theta(x)$, and even the expectations required by standard mean-field variational methods may all be intractable when the likelihood is nonlinear, such as a neural network. [Source: §2.1, p.2]

A fresh optimization or sampling loop for every observation is expensive. The paper instead learns a shared shortcut from observations to distributions over latent codes. This is a new teaching diagram, not a paper figure.

Earlier options do not scale cleanly

The authors focus on large datasets where full-batch optimization is too costly. Monte Carlo EM can require an expensive sampling loop per datapoint, while a naïve Monte Carlo gradient for the recognition model has very high variance. Wake-sleep offers an efficient recognition model, but its updates do not all optimize the variational lower bound used here. [Source: §2.1–2.2, pp.2–3; §4, pp.5–6]

Method

Turn sampling into a differentiable path

Suppose the recognition model outputs a Gaussian with mean $\mu$ and scale $\sigma$. Sampling $z$ directly appears to put randomness between those outputs and the training loss. The paper rewrites the same sample as

$$\epsilon \sim \mathcal N(0,I), \qquad z = \mu + \sigma \odot \epsilon.$$

Now the randomness lives in $\epsilon$, whose distribution does not depend on the learned parameters. For a fixed noise draw, $z$ is a differentiable function of $\mu$ and $\sigma$, so ordinary backpropagation can estimate gradients. The paper calls the resulting estimator Stochastic Gradient Variational Bayes (SGVB). [Source: Eq. 4–6 and §2.4, pp.3–4; Eq. 10, p.5]

Reparameterization does not remove randomness. It expresses randomness as an input to a deterministic transform, opening a path for gradients. New teaching diagram based on §2.4.

Learn an encoder and decoder together

The recognition model $q_\phi(z\mid x)$ acts as a probabilistic encoder: it maps an observation to a distribution over plausible codes. The generative model $p_\theta(x\mid z)$ acts as a probabilistic decoder: it maps a code to a distribution over possible observations. AEVB updates both parameter sets jointly using randomly drawn minibatches. In the reported experiments, the authors used minibatches of 100 and one latent sample per datapoint. [Source: §2.1, pp.2–3; Algorithm 1 and Eq. 8, p.4; §5, p.7]

Balance reconstruction with an organized latent space

For the common case where the prior and approximate posterior make the KL divergence tractable, the objective can be read as

$$\mathcal L = \underbrace{\mathbb E_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)]}_{\text{reconstruct the observation}} - \underbrace{D_{KL}\!\left(q_\phi(z\mid x)\,\|\,p_\theta(z)\right)}_{\text{stay near the prior}}.$$

The first term rewards codes that help the decoder explain the data. The second discourages each observation’s code distribution from drifting arbitrarily far from the prior. Maximizing this lower bound therefore trains inference and generation through one objective. [Source: Eq. 3 and Eq. 7, pp.3–4]

The objective balances reconstruction against regularization toward the prior; it does not promise perfect reconstruction or a perfectly matched posterior. New teaching illustration based on Eq. 3.

Evidence and limits

What the experiments show

The authors trained image models on MNIST and Frey Face and compared AEVB with wake-sleep. Across the latent dimensions shown in Figure 2, they report that AEVB converged considerably faster and reached a better variational lower bound in every comparison. The reported variance of the bound estimator was below 1 and omitted from the plots. Their CPU implementation processed one million training samples in roughly 20–40 minutes on an Intel Xeon delivering an effective 40 GFLOPS. [Source: §5 and Figure 2 caption, pp.6–7]

For a three-dimensional MNIST latent space, Figure 3 also compares estimated marginal likelihood against wake-sleep and Monte Carlo EM. The main scaling result is qualitative: Monte Carlo EM was not an online method and could not be applied efficiently to the full 50,000-example MNIST training set, whereas AEVB and wake-sleep could. The paper also demonstrates two-dimensional latent visualizations for MNIST and Frey Face. [Source: §5 and Figure 3, pp.7–8; Appendix A, pp.9–10]

What remains uncertain

The experiments are narrow by modern standards: two image datasets, shallow multilayer perceptrons, and limited baselines. Higher-dimensional marginal-likelihood estimates became unreliable, so that comparison used only three latent variables. The paper leaves variational inference over global parameters, deeper or convolutional hierarchies, time-series models, and supervised latent-variable models to future work. It does not establish performance on text, high-resolution images, very deep architectures, or today’s large-scale workloads. [Source: §2, p.2; §5, p.7; §7, p.8]

The lower bound is also not the true log likelihood unless the approximate posterior exactly matches the true posterior. A simple diagonal Gaussian recognition model can therefore be a limiting approximation. The paper proves a practical gradient estimator under differentiability and reparameterizability conditions; it does not claim that every latent distribution admits the same simple treatment. [Source: Eq. 1–3, p.3; §2.4, pp.4–5]

Practical meaning

The durable design pattern

The lasting idea is broader than an autoencoder-shaped network: make approximate inference a learned, reusable computation and rewrite stochastic nodes so they can be trained end to end. That pattern makes latent-variable models compatible with minibatches, backpropagation, and flexible neural likelihoods. It is a sound starting point when a system needs a compact probabilistic representation, fast inference for new examples, and a generative path back to data. This is an interpretation of the mechanism, not a claim tested beyond the paper’s experiments.

When to be cautious

Use the method’s objective as a tradeoff, not a guarantee. Good reconstruction does not prove that latent factors are meaningful, independent, fair, or causally correct; a high lower bound is still only a lower bound. For a new application, separately check held-out likelihood or task performance, reconstruction quality, sample quality, inference speed, posterior approximation error where measurable, and sensitivity to the chosen prior and encoder family.

Reader’s verification guide

• Core derivation: Eq. 1–8 and §2.4, pp.3–5.
• Concrete variational autoencoder: §3, p.5 and Appendix B, p.10.
• Baselines and settings: §4–5, pp.5–7.
• Results: Figures 2–3, pp.7–8, plus latent-space examples in Appendix A.
• Author-stated next steps: §7, p.8.

Terminology. Posterior means beliefs about hidden variables after seeing data. Variational lower bound—now often called the evidence lower bound or ELBO—is an optimizable lower bound on log marginal likelihood. Recognition model is the learned approximation from data to a posterior distribution. Reparameterization expresses a random draw as fixed-distribution noise passed through a differentiable transform.