Diffusion models: restoring Dada's old photograph

In a narrow lane off Anarkali, there is a small studio where an old man restores photographs. Families bring him faded, scratched, water-stained pictures of grandparents, wedding portraits from the 1960s, shop openings from before Partition.

He doesn't repaint them in one go. He works in small passes. First he recovers the rough shapes: a face here, a turban there. Then he sharpens the eyes. Then the folds of the clothes. Each pass removes a little more damage, guided by fifty years of knowing what faces and fabrics should look like.

Now imagine handing him a photo so damaged it is nothing but grey speckles, and saying: "This was a picture of a bride in red, standing in a garden." Using only his experience, pass by pass, he could produce a convincing picture that never existed.

That is how diffusion models create images.

Step 1: learn by destroying photos

Training a diffusion model starts with a strange exercise. Take millions of real images. For each one, add a little random noise, like static on an old TV. Then a little more. And more. After enough steps, the image is pure noise: no trace of the original.

This is the forward process, and it's easy. Anyone can ruin a photo.

Step 2: learn to undo one step

The model is then trained on the hard direction. It is shown a noisy image and asked: what noise was added in the last step? If it can predict the noise, it can subtract it and make the image slightly cleaner.

Just like the restorer, it doesn't fix everything at once. It learns to remove one layer of damage at a time, at every level of noise, from "slightly grainy" to "pure static". To do this well across millions of pictures, it has to learn what the world looks like: how faces are shaped, how light falls, how fabric folds.

This approach became popular with a 2020 paper on denoising diffusion probabilistic models (DDPM) by Jonathan Ho and colleagues.

Step 3: create something new

Once trained, you can start from pure random noise, an image of nothing at all, and ask the model to clean it up, step by step. With each step, shapes emerge, then details, until there's a crisp picture. Because the starting noise is random, every result is different.

Steering with words

On its own, the model would produce some realistic image. To get the one you want, the denoising is guided by a text description. The text is turned into embeddings (Part 5), and at every step the model is nudged towards an image that matches it.

"A bride in a red dupatta standing in a Lahore garden, golden evening light, 1960s photograph"

This is the "Here's what the photo was of" instruction we gave the restorer. Systems like Stable Diffusion also save effort by working in a compressed version of the image (a "latent space"), which makes generation much faster.

The maths, briefly

In DDPM, each forward step mixes in a little Gaussian noise, controlled by a schedule $\beta_t$:

$$q(x_t \mid x_{t-1}) = \mathcal{N}\!\left(x_t;\ \sqrt{1-\beta_t}\, x_{t-1},\ \beta_t \mathbf{I}\right)$$

A useful result lets you jump straight to any step. With $\bar{\alpha}_t = \prod_{s=1}^{t} (1-\beta_s)$ and random noise $\epsilon \sim \mathcal{N}(0, \mathbf{I})$:

$$x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1-\bar{\alpha}_t}\, \epsilon$$

The network $\epsilon_\theta$ is trained to predict that noise, with a simple squared-error loss:

$$L = \mathbb{E}_{x_0,\, \epsilon,\, t} \Big[ \lVert \epsilon - \epsilon_\theta(x_t, t) \rVert^2 \Big]$$

Use it responsibly

Diffusion models can produce realistic images of things that never happened. That's wonderful for design, marketing mock-ups and creative work, and risky for fake news and fake documents. Good practice is to label AI images clearly and never use them to impersonate real people or create false records.

What to remember

• Diffusion models are trained by adding noise to images and learning to remove it.
• To generate, they start from pure noise and denoise step by step.
• Text prompts guide each step towards the image you described.
• The same idea works for audio, video and more.

The whole series, in one paragraph

A neural network (Part 1) learns by adjusting weights to reduce its mistakes, walking downhill with gradient descent (Part 2), while avoiding overfitting (Part 3). CNNs (Part 4) apply this to images. Embeddings (Part 5) and tokens (Part 6) turn language into numbers, and Transformers (Part 7) let every token pay attention to every other. Scale that up and you get LLMs (Part 8); connect them to your documents with RAG (Part 9); and run the learning in reverse on noise and you get diffusion (Part 10).

If you have a business problem that one of these ideas could solve, we would like to hear about it.