Diffusion decoding in five minutes
An autoregressive model such as Llama writes left to right. Each forward pass produces one token, and an append-only KV cache makes the next pass cheap.
A masked diffusion model such as LLaDA writes the way you fill in a crossword.
It starts from a canvas of [MASK] tokens after the prompt and, at every step,
looks at the whole canvas at once, proposes a token for every blank, and
commits the few it is most confident about. The rest stay masked for the next
step.
prompt: "Explain speculative decoding" canvas (8 positions shown)
step 0 [M] [M] [M] [M] [M] [M] [M] [M]
step 1 [M] [M] [M] [M] [M] [M] [M] .
step 2 A [M] [M] [M] [M] [M] [M] .
step 3 A draft [M] [M] [M] [M] tokens .
...
step 8 A draft model guesses several tokens .
Each step runs the full transformer over prompt plus canvas. Attention is bidirectional: position 3 can read position 7, which a causal model forbids.
The loop LLaDA uses
mojo-dllm implements the reference sampler from the LLaDA repository
(generate.py), with its defaults:
- Split the generated region into blocks of
--block-lengthpositions and fill them left to right (semi-autoregressive decoding). - Give each block
steps / blockssteps and spread its masked positions over them evenly: with 32 masks and 8 steps, 4 tokens per step. - At each step, for every masked position in the block, take the argmax token and its softmax probability.
- Commit the k positions with the highest probability ("low-confidence remasking": the uncertain ones stay masked).
At temperature 0 the loop is deterministic. Above 0, LLaDA adds Gumbel noise to the logits before the argmax; mojo-dllm does the same.
Why the runtime looks different
- No KV cache by default. Any position can change between steps, so every
step recomputes keys and values for the whole canvas.
--dual-cache(LLaDA only) keeps a block-level cache and gives up exactness; see Command line. A step costs a full prompt pass, about 2 × 7 billion operations per token on the canvas in LLaDA-8B, plus 2 × 0.5 billion for each row whose logits are computed. - GEMMs, not GEMVs. A step multiplies weight matrices by a hundred or more activation rows at once. The CPU kernel is built for that shape; see The quantized GEMM.
- Few rows need logits. Only the masked positions the sampler can commit are candidates (the current block for LLaDA, every masked position for Dream), so mojo-dllm computes the output projection (126 464 columns for LLaDA, 152 064 for Dream) for those rows alone. The LLaDA benchmark ablation measures what that saves.
DiffusionGemma's loop
DiffusionGemma does not mask. Its canvas starts as 256 random token ids over the whole vocabulary and every step re-scores all of them:
- Take the temperature
t = t_min + (t_max - t_min) * c / S, withccounting down fromS(48) to 1: 0.8 on the first step, 0.4 + 0.4/48 on the last. - For each canvas position, find the argmax of its logits, the entropy of
softmax(logits / t)and one token drawn from that softmax. mojo-dllm reads each 262 144-wide row once for the argmax and the entropy. - Sort positions by entropy, lowest first, and accept each one while the entropies of the positions before it sum to at most 0.1. The lowest-entropy position is always accepted.
- Accepted positions keep their drawn token and every other position gets a fresh random id.
- The answer is the argmax canvas. The run stops once that canvas has held for one step and its mean entropy is below 0.005, or after 48 steps.
The six numbers are --eb-steps, --eb-t-min, --eb-t-max, --eb-bound,
--eb-stability and --eb-confidence (see Command line). A run
takes as many forward passes as steps, so the sampler decides the cost.