Anderson Mendoza

Software engineer Cartagena, Colombia

All posts

Explainer

How diffusion models make images

· 5 min read · Updated

Four discs of dots, from scattered static on the left to a globe of the Americas on the right

Type a sentence into an image generator and a picture comes back a few seconds later. What happens in between is easier to follow than its reputation suggests, and it starts from an odd place: the model was trained on ruined pictures.

Learning from ruined pictures

Take a photograph and add a little random noise, like static on an old television. Add more, and more again. After enough rounds nothing of the photograph is left.

Ruining a picture this way takes no skill. The training is in going back. The model is shown an enormous number of pictures at every stage of ruin, each with the caption that describes it, and is asked the same question every time: what noise was added here? Its answer is compared with the truth, and the model is adjusted.

To answer well it has to learn what pictures look like: that a cat has two ears, that shadows fall away from a lamp, that a horizon is level. None of that is written down for it. It is simply what a model needs to know to tell a picture from the noise on top of it.

From static to a picture

To make a new picture, the model is handed pure static and treats it as a picture with noise on it. It estimates the noise and removes some. The result is slightly less random, so it estimates again and removes a little more. After a few dozen rounds a picture is there.

The same castle generated with 1, 2, 3, 5, 8, 10, 15, 20, 30 and 40 steps: static at first, a blur by step 3, a finished picture from about step 15
One prompt and one seed, given from 1 to 40 steps. With too few, the picture never settles. Image: Benlisquare, CC BY-SA 4.0, resized.

Things appear in the same order every time. The first steps settle the composition: where the large shapes are, what is light and what is dark. The middle steps turn shapes into things. The last ones add only texture: leaves, hair, the grain of a wall. That is why a picture stopped early looks like a blurred version of the right scene, and not like a sharp picture of the wrong one.

The static it starts from is chosen by a number called the seed. The same seed, prompt and settings give the same picture, which is how people reproduce a result and refine it. A different seed gives a different picture for the same words.

Where the prompt comes in

A second model reads the prompt first and turns it into a list of numbers that stands for its meaning. At every step, the model removing the noise is given those numbers along with the noisy picture, so the question it answers becomes: what noise is on this picture of a lighthouse at dusk?

On its own that pull is weak, and there is a trick to strengthen it. At each step the model makes two estimates, one with the prompt and one without, and then moves further in the direction in which they differ. How much further is the setting most tools call guidance scale. Set it low and the picture drifts from what you asked for. Set it high and the picture follows the words closely but turns harsh and oversaturated.

The model that reads the prompt matters more than it sounds. Early generators used a small one that caught the gist and lost the detail, which is why they ignored half of a long prompt and could not spell. Newer ones read with much larger language models, and long prompts and legible lettering largely came with them.

Why it is fast enough to use

A picture of 1024 by 1024 pixels is about three million numbers, and removing noise from all of them dozens of times would be slow. So most generators don’t work on pixels.

A separate model, trained beforehand, can squeeze a picture into a much smaller grid of numbers and unpack it again with little visible loss. All the noise removal happens on the small grid, and the unpacking is done once, at the end. In the first Stable Diffusion a 512-pixel picture became a grid of 64 by 64, with 48 times fewer numbers to work on. This design is called latent diffusion, and it is the reason these models run on an ordinary graphics card.

What has changed since

The idea is the same as in 2022, and most of the parts have been replaced. The network that estimates the noise is now usually a transformer, the design used in language models. Many current models are trained on a slightly different task called flow matching: they learn the direction that leads straight from static to picture, which takes fewer steps to follow. Some are then distilled to need only a handful of steps, which is how a picture can appear while you are still typing.

The same recipe is used for video, where the model removes noise from all the frames together so that they agree with one another, and for sound.

What it is not doing

It is not searching a library of pictures and stitching pieces together. Nothing is stored in the model but its adjusted numbers, and there are far too few of them to hold the pictures it was trained on. A picture that appeared many times in training can still come back out nearly intact, which is one reason the question of what these models were trained on is still argued over.

It also has nothing to check its work against. It knows what hands usually look like, and it does not know that a hand has five fingers. That is why counting, lettering and anything that has to be exactly right were the last things to improve, and are still where it slips.

Up next

  • The design system under this page

    · 6 min read

    This site is drawn with Mereframe Components, the design system I wrote for my own products. Its six rules, how a surface takes its colours from one number, why state is written in ARIA, and what this page ships of it.

  • The globe on this page

    · 5 min read

    The globe on my home page is 2,405 dots on a plain canvas, drawn from a map of one kilobyte. How it is built, and why it is not a library.

For work, or about anything I’ve built: hello@mereframe.com