mojo-dllm logo: masked tokens resolving into a flame
Local inference for diffusion language models, written in Mojo.

Introduction

mojo-dllm runs two families of diffusion language models, LLaDA and Dream, from quantized GGUF files on your own CPU or on an NVIDIA GPU. LLaDA-8B and Dream-7B are measured; LLaDA-1.5, Dream-Coder and DiffuCoder are tested on the CPU (see Quick start). The whole inference path is Mojo: the GGUF reader, the tokenizer, the quantized matrix multiply, the GPU kernels, the transformer and the denoising loop. The build/mojo-dllm binary needs no Python at runtime, and the GPU is optional. The ./mojo-dllm launcher that builds and starts it is a Python script run by uv.

A diffusion language model writes differently from the models you know. An autoregressive model appends one token at a time. LLaDA starts from a canvas of [MASK] tokens and fills it in over a fixed number of steps, committing the tokens it is most sure about first. Diffusion decoding in five minutes walks through it with a picture.

What works today

  • mojo-dllm run generates text from LLaDA-8B-Instruct and Dream-v0-Instruct-7B (Q4_K_M) on x86 CPUs with AVX2, using AVX-VNNI when present.
  • --device gpu runs both models on NVIDIA GPUs with compute capability 8.0 or newer. It has been measured on an RTX 3050 Laptop GPU only.
  • The LLaDA and Dream tokenizers match llama.cpp id for id on a 36-case multilingual corpus, and DiffusionGemma's gemma4 tokenizer on 54 cases.
  • Logits match llama.cpp's implementation of both models; the measured agreement is in Correctness.
  • Benchmarks against llama.cpp and diffuse-cpp on the same machine, with every measurement table generated from a result file in the repository.

What does not work yet

  • The GPU path is measured on one laptop part, an RTX 3050 held to 30 W, against llama.cpp's CUDA build; Benchmarks has the numbers. It supports NVIDIA only.
  • Two architectures run: LLaDA and Dream. A diffusion-gemma file is read, inspect prints its config, tokenize prints the ids of its chat prompt, and logits runs it on the CPU: one unified pass over the prompt and the canvas, checked against a numpy forward pass on a tiny fixture (#64). The entropy-bounded denoiser is written too (#65), but run stops on this file with diffusion-gemma does not run on this model until the model is wired into the generation loop (#277), on the CPU and the GPU. Other architectures stop with unsupported model architecture: <name> (runs: llada, dream; read only: diffusion-gemma).
  • One quantization mix: Q4_K and Q6_K projection matrices, as in a Q4_K_M file, with an embedding table and norm vectors that the reference decoders read (F32, F16, BF16, Q5_0, Q8_0, Q4_K, Q5_K, Q6_K). A matrix of any other type stops the load with an error that names it. DiffusionGemma's logits also reads Q5_0 and Q8_0 matrices for the down projections.

Where to go next

Quick start

You need Linux on x86-64 with AVX2, glibc 2.34 or newer, about 6 GB of free memory, and uv. uv installs the pinned Mojo toolchain (Mojo 1.1.0, max-core 26.6.0) into the project's .venv; nothing goes system-wide.

1. Set up

curl -LsSf https://astral.sh/uv/install.sh | sh    # skip if you have uv
git clone https://github.com/dllm-labs/mojo-dllm
cd mojo-dllm
./mojo-dllm setup

The uv installer puts uv in ~/.local/bin. If your shell does not find it afterwards, open a new terminal or run source ~/.local/bin/env.

setup runs four steps and prints each one:

  1. Toolchain. It checks the platform, glibc and AVX2, stops on anything that cannot work, and runs uv sync --frozen.
  2. Build. It compiles build/mojo-dllm, or reports that the binary is newer than its inputs and was built for this Mojo version and GPU architecture.
  3. Model. It asks before downloading LLaDA-8B-Instruct Q4_K_M (4.9 GB) into the Hugging Face cache, and checks the file's sha256 against the one the hub publishes. --yes answers the question for you; --no-model skips the step; --model picks another model.
  4. Check. It generates 16 tokens to prove the binary, the model and the device work together. The check fails when the binary exits non-zero, prints nothing, or runs longer than 300 seconds. With --no-model the step is skipped and setup ends with Setup complete (no smoke test).

setup --offline (or HF_HUB_OFFLINE=1) uses the model cache only, as run and pull do. Without a terminal to ask on (a script, CI), setup does not download anything unless you pass --yes. It then exits 3 and prints the command that finishes the job.

./mojo-dllm doctor runs those checks without changing anything, and adds free memory, the Mojo version, the binary's age, the GPU and the model cache. Every line that is not ok says how to fix it, with the command where one exists; for a missing GPU it says that the CPU will be used.

2. Generate

./mojo-dllm run "Explain speculative decoding in two sentences."
./mojo-dllm run "Write a haiku about rain." --max-tokens 64 --steps 32 --verbose

The prompt is the first word that is not a flag; quote it. run rebuilds the binary if a source file, pyproject.toml, uv.lock or scripts/build.sh changed, or if the Mojo pin or the GPU architecture (MOJO_DLLM_GPU_ARCH, else the local GPU) differs from the one recorded in build/mojo-dllm.stamp, finds the model, picks the device and hands the rest of the flags to build/mojo-dllm run. Command line lists them all.

A flag and its value are two arguments: --steps 8, not --steps=8. The launcher refuses the = form and prints the two-argument form to use. A prompt that starts with - goes after -- (./mojo-dllm run -- "-h"); -h and --help after -- belong to the prompt and do not print the help.

--max-tokens sets the canvas size, --steps the number of denoising passes. Fewer steps run faster and commit more tokens per step, which costs quality. Each step runs the full model over prompt and canvas, so expect seconds per step on a laptop CPU. --verbose prints load time, per-step time and peak memory.

The prompt goes through the model's chat template. Pass --no-chat to feed raw text instead.

The device

--device auto is the default. It picks the first NVIDIA GPU with compute capability 8.0 or newer that nvidia-smi lists, and the CPU when there is none. The binary opens CUDA's default device, which can be another card than the first one nvidia-smi lists, so the launcher sets CUDA_VISIBLE_DEVICES to the UUID of the card it named (with CUDA_DEVICE_ORDER=PCI_BUS_ID) before it starts the binary. A CUDA_VISIBLE_DEVICES you already set (even an empty one) is kept: the launcher does not pin the card and prints a warning saying so. --device gpu runs the same check: it pins the first qualifying card, and prints a warning when nvidia-smi finds no card or only one below compute capability 8.0. Status lines go to stderr, so stdout holds only the model's text:

device: gpu (NVIDIA GeForce RTX 3050 6GB Laptop GPU, compute 8.6, 585 MiB free)

auto does not compare the card's free memory with the model's size. In that line another program held most of the card, so the run that followed would have said how much memory it needs and exited. Pass --device cpu to run anyway.

3. Other models

-m (or --model) takes one of three forms:

FormExampleWhat happens
a file-m ~/models/llada.ggufused as is
the alias-m llada-8bmradermacher/LLaDA-8B-Instruct-GGUF, Q4_K_M (the default)
a repo-m owner/Some-Model-GGUF:Q4_K_Mthe file matching that quant, downloaded once

Without :QUANT, a repo resolves to its Q4_K_M file, then Q4_K_S, then Q6_K, the quants the matrix kernels load. A repo with none of them, and a quant that matches nothing, fail with the list of files the repo has, and owner/repo:QUANT selects one. Split (sharded) GGUF files are not supported. A Q8_0 file downloads when you ask for it with :Q8_0 but does not load: the matrix kernels need Q4_K or Q6_K.

Tested Q4_K_M checkpoints are LLaDA-8B-Instruct, LLaDA-1.5, Dream-v0-Instruct-7B, Dream-Coder-v0-Instruct-7B and DiffuCoder-7B-cpGRPO. The added checkpoints were checked on CPU; pass --device cpu to reproduce those runs. See the smoke and parity evidence for the exact files and generation agreement. Other quantizations still need compatible tensor types. An unsupported architecture stops with an error naming it.

For DiffuCoder, obtain the file separately under its apple-amlr license and pass its local path with --model; it is not bundled with the project.

./mojo-dllm pull MODEL downloads without generating. Models live in the Hugging Face cache ($HF_HUB_CACHE, else $HF_HOME/hub, else $XDG_CACHE_HOME/huggingface/hub, else ~/.cache/huggingface/hub), shared with every other tool that uses it. A gated or private repo needs HF_TOKEN.

The sha256 check runs on download. A file already in the cache is checked against the hub's sha256 the first time it is used, and the launcher remembers the result in ~/.cache/mojo-dllm/verified.json (under $XDG_CACHE_HOME when that is set). A file that fails the check is deleted, and the next run downloads it again. With --offline the check cannot run, so an unchecked file is used after a warning. pull --offline and setup then print already downloaded (not verified) and exit with status 3, and doctor marks the model with !. Run ./mojo-dllm pull once online to check it.

4. Offline

./mojo-dllm pull                      # on a connected machine
./mojo-dllm run "Hello" --offline     # later, with no network

--offline (or HF_HUB_OFFLINE=1) uses the cache only and fails with the pull command to run when the model is missing.

5. Look inside a model

./mojo-dllm pull                             # prints the model file's path
./mojo-dllm inspect PATH.gguf | head -40

You get the metadata, a count of tensor types, and every tensor with its shape. The binary's own commands (inspect, tokenize, logits) go to build/mojo-dllm unchanged, after the same rebuild check. Any other word stops with an error, and names the closest command when one is near.

Without the launcher

The launcher is a convenience. The binary works on its own:

uv sync --frozen
scripts/build.sh
build/mojo-dllm run --model PATH.gguf --prompt "Hello" --device cpu

build/mojo-dllm defaults to --device cpu and takes only file paths.

Command line

mojo-dllm run      --model PATH --prompt TEXT [options]
mojo-dllm tokenize --model PATH --prompt TEXT [--no-chat]
mojo-dllm inspect  PATH
mojo-dllm logits   --model PATH --tokens IDS --rows POSITIONS --out FILE [--threads N] [--device D]
                   [--path unified|cached] [--prev FILE --prev-temp-inv X]

This page describes the binary, build/mojo-dllm. The ./mojo-dllm run launcher takes the prompt as its first word, accepts -m for --model, defaults --device to auto and passes the other flags on to the binary; see the Quick start. A CUDA_VISIBLE_DEVICES you set before the launcher starts is kept: the launcher then does not pin the card and prints one warning saying so.

mojo-dllm COMMAND --help (or -h) prints that command's usage and exits 0. Help wins over every other problem in the arguments, wherever it appears, except as the value of a flag (--prompt --help sets the prompt to --help). mojo-dllm, mojo-dllm help and mojo-dllm --help print the usage of every command. Each command declares its flags: a flag the command does not declare stops the run (see Errors).

mojo-dllm --version prints the package version on its first line and the commit the binary was built from on its second, then exits 0:

mojo-dllm 0.2.0
commit: abc123456789

The package version is the "version" key of run --json. The commit is the 12-character HEAD of the checkout the binary was built from, with a -dirty suffix when src/ had pending changes, or unknown when the build cannot read Git metadata. A binary keeps reporting the commit it was built from after HEAD moves; rebuild to stamp the new one.

run

FlagDefaultMeaning
--model PATHrequiredLLaDA or Dream GGUF file
--prompt TEXTrequiredthe user message
--max-tokens N128tokens to generate (the masked canvas). The prompt tokens plus N must not exceed the file's <arch>.context_length (4096 when the key is absent); the check runs after the header is read and before any weight is mapped
--steps N--max-tokensdenoising steps; a multiple of the block count
--block-length N32, or --max-tokens when that is smallerLLaDA's semi-autoregressive block size; must divide --max-tokens. A Dream run refuses it
--temperature T0LLaDA: 0 is greedy; above 0 adds Gumbel noise as LLaDA does. Dream supports only 0 and stops with an error above it
--remasking Slow_confidenceLLaDA: low_confidence or random. Dream: entropy (the default), maskgit_plus, topk_margin, origin, or llama.cpp-style random
--threshold P0 (off)A number from 0 to 1. LLaDA: each step also commits every masked token whose confidence is at least P, so easy blocks finish in fewer steps (Fast-dLLM's parallel decoding). A step still commits at least what the schedule says. Not exact: see Benchmarks for its measured effect on accuracy
--dual-cacheoffLLaDA: cache every position's keys and values and, inside a block, recompute only that block's rows (Fast-dLLM's DualCache). The cache is refreshed at the first step of each block. Not exact: the prompt and later blocks see keys from the block's first step; see Benchmarks for its measured effect on accuracy. Cannot combine with --full-logits
--seed N42seeds this program's PRNG (it does not reproduce a PyTorch run). It fixes DiffusionGemma's random canvas and draws as well
--eb-steps N48DiffusionGemma: most denoising steps, at least 1 (use it instead of --steps, which that model refuses) (no effect yet, see the paragraph after the table)
--eb-t-min T0.4DiffusionGemma: the temperature the schedule approaches, 0 or more; the last of --eb-steps steps runs at T + (t-max - T) / steps (no effect yet, see the paragraph after the table)
--eb-t-max T0.8DiffusionGemma: the temperature of the first step, above 0; it falls linearly from there (no effect yet, see the paragraph after the table)
--eb-bound B0.1DiffusionGemma: a position is accepted while the entropies of the lower-entropy positions before it sum to at most B, which must be at least 0 (no effect yet, see the paragraph after the table)
--eb-stability N1DiffusionGemma: steps the argmax canvas must hold before the run may stop; 0 or more, and 0 lets the first step stop it (no effect yet, see the paragraph after the table)
--eb-confidence C0.005DiffusionGemma: the run may stop once the mean entropy over the canvas is below C, which must be at least 0 (no effect yet, see the paragraph after the table)
--threads Nall logical CPUsworker threads of every parallel CPU loop, including the activation quantization and the weight repack at load (with --device gpu, only the sampler uses them)
--device Dcpucpu, or gpu for an NVIDIA GPU with compute capability 8.0 or newer
--no-chatoffskip the chat template
--full-logitsoffLLaDA: compute logits for every position, as a generic runtime would (for benchmarking). Dream refuses it
--no-suppress-maskoffallow the mask token as a sampled answer for parity experiments; by default it is excluded from argmax and the softmax normalizer
--no-early-stopoffrun every step even after the answer is final (the benchmark and parity harnesses use it)
--visualoffshow the canvas: masked positions as teal blocks, tokens committed in that step in bold orange when colour is enabled
--visual-interval N1redraw every N steps and at the final step; requires --visual and N at least 1
--verboseoffmodel, device and load time before the text; step timings and peak memory after it
--dump-step-statsoffone line per denoising step
--jsonoffone JSON object with timings, tokens and text. peak_rss_bytes is null when the operating system could not report it, never 0. remasking names the strategy the run used (for Dream, entropy when the flag says low_confidence); block_length and full_logits appear for LLaDA only, because Dream reads neither. steps_taken counts the denoising steps run and stop_reason says why the loop ended: complete, answer_fixed (early stop), or for DiffusionGemma, once run accepts it (#277), converged or max_steps

The flags --threshold, --dual-cache, --full-logits and --block-length are LLaDA's; a Dream run refuses them by name before the model loads (--threshold applies to llada only, not dream).

The printed answer ends at the first end-of-text or end-of-turn token in the canvas. Generation stops as soon as that answer is final: once the first position after the prompt that is still masked or holds such a token holds such a token, no later step can change the printed text, because steps only fill masked positions. The result is the same text with fewer steps, and --json reports "stopped_early": true. --no-early-stop runs every step.

Dream's origin follows its Python reference: each masked position commits independently with probability 1 - s/t, forced to 1 at the final step. Zero commits in an intermediate step is valid. This differs from llama.cpp's origin, which floors the transfer count before calculating the probability. Dream random follows llama.cpp: assign each candidate a uniform random confidence, then select the scheduled top-k positions. Both rules use --seed. The default remains entropy for Dream; select --remasking origin to use Dream's reference algorithm.

Mask suppression keeps the Gumbel RNG draw for the excluded token, so toggling it does not shift the random stream. On GPU, suppression uses full logits because the device row statistics include the mask. --no-suppress-mask retains the existing device reduction path when its other conditions hold.

The --eb-* flags belong to DiffusionGemma and every other model refuses them by name. The parser accepts them, but run stops on a diffusion-gemma file until the model is wired into the generation loop (see Introduction). It refuses the file once its header is read, before the tokenizer is built, with diffusion-gemma does not run on this model: `run` waits for the model to be wired into the generation loop (#277), so use `logits` (CPU) until then, so the --eb-* flags have no effect yet; run --help says so. Only logits runs that architecture. The forward pass (#64) and the entropy-bounded loop (#65) are written; #277 tracks joining them. Once it lands, DiffusionGemma's --json reports "stopped_early": true when the stop rule ends the run before --eb-steps steps.

Visual output

The final canvas always appears, including when generation stops early between redraws. Its step number is the last observed step; any remaining masks stay visible. A final frame that was already drawn is not repeated.

stdoutNO_COLORcolour codesclear + home (ESC[2J ESC[H)
terminalunset or emptyyesyes
terminalnon-emptynoyes
not a terminalanynono; frames separated by one blank line

On a terminal, --visual leaves the last canvas frame on screen and does not print the answer a second time. Without --visual, or when stdout is not a terminal, run prints the answer after generation.

Without colour, newly committed tokens have no highlight. Redirected frames contain no ANSI escapes. --visual, --verbose and --dump-step-stats print to stdout before the JSON object, so each of them is refused together with --json: the command stops with the usage error --json cannot be combined with --visual (or --verbose, --dump-step-stats) and exit status 2, before the model loads. With --json, stdout holds the one JSON object and nothing else.

The checks that need only the flags run before the model file is opened, and each exits with status 2 and names the flag: --threshold outside 0 to 1 or not finite, --device other than cpu or gpu, --remasking outside the six names above, and --threshold above 0 with --remasking random. LLaDA also needs a --block-length that divides --max-tokens and a --steps that is a multiple of the number of blocks (Dream reads neither flag and accepts any value); that check depends on the architecture, which only the file's header says, so it runs once the header is read, still before any weight is mapped, and exits with status 1, as does the check of the default block length (32, or --max-tokens when smaller): flag --max-tokens 100 is not a multiple of the default block length 32. The same header check refuses a --remasking value that LLaDA does not take (entropy, maskgit_plus, topk_margin and origin); Dream takes all six.

Invalid --visual-interval values and use without --visual exit with status 1 before model loading. The messages name --visual-interval: must be at least 1, needs --visual, or needs an integer.

tokenize

Prints the token ids of the chat-formatted prompt, the same ids run feeds the model. Useful to compare against llama-tokenize. For a diffusion-gemma file the prompt is the file's chat template: <bos>, a user turn and an open model turn, which gives one BOS token (id 2). It reads the tokenizer arrays of the file and none of its weights, so it runs on a file too big to load. Only llada, dream and diffusion-gemma files have a chat template here: for another architecture the command stops with no chat template for architecture 'qwen2' (llada, dream and diffusion-gemma have one); pass --no-chat to tokenize the text as it is, and --no-chat tokenizes the text unchanged. run applies the same rule to its prompt.

inspect

Prints GGUF metadata, a count per tensor type, and every tensor's type and shape. It needs no supported architecture, so it works on any GGUF file. For diffusion-gemma it also prints a config: block with what the reader took from the file (layer kinds, K/V heads and head size per kind, experts, canvas length), or config: refused: and the reason, such as the key that is wrong. For llada, dream and diffusion-gemma it also lists each metadata key that the file lacks and the reader replaced by a default, one per line under defaults (key absent from the file):, as llada.rope.freq_base = 10000 (default, key absent). A file that sets every such key prints no such block. run --verbose prints the same block after the device line. The keys are the KV head count, the context length, the rope base and the logit shift for LLaDA and Dream, and the context length and the final logit softcap for DiffusionGemma.

logits

Runs one forward pass over the given token ids and writes f32 logits for the requested positions to a file, [positions, vocab], row major. The parity tests use it; see Correctness.

The file is written to FILE.tmp, checked there to hold exactly positions * vocab * 4 bytes, and only then renamed to FILE; any failure (a full disk, a quota, a missing or unwritable directory) removes the temporary file, prints ERROR: cannot write FILE: ... with the system's reason, exits with status 1 and leaves any earlier FILE as it was. wrote N rows appears only after the rename.

The rename replaces a symlink named FILE (the link goes and its target stays as it was) and a read-only FILE. When FILE already exists and is not a regular file (/dev/null, /dev/stdout, a pipe), the bytes are written to it directly: there is no name to rename to and no size to check.

On a diffusion-gemma file the tokens are the prompt followed by the canvas, whose length the file declares (diffusion.canvas_length, 256 in the real model). The model scores only the canvas, so --rows must list exactly those positions, the last canvas-length ones, and the file holds [canvas, vocab]. It runs on the CPU only; --device gpu is refused. Three flags apply to this architecture alone and are refused on the others:

  • --path unified (the default) runs the prompt and the canvas in one pass. --path cached prefills the prompt, keeps its keys and values, and steps the canvas against them. The two agree bit for bit.
  • --prev FILE names the previous denoising step's canvas logits, [canvas, vocab] f32 as this command writes them, and turns self-conditioning on. --prev-temp-inv X, which must come with it, is 1 divided by the temperature of that previous step (1.25 for the first step's 0.8). It must be above 0 and its product with the model's final logit soft cap must not exceed 1e8 (about 3e6 with the cap of 30), because the softmax turns the logits to NaN beyond that, and --path must be unified or cached; the path is checked before any model is read, the bound when the pass starts.

--rows and the size of the --prev file are checked against the canvas length that the file's header declares, before a weight is mapped, so a wrong --rows costs no model load.

Errors

An error names the thing that is wrong. Usage errors exit with status 2 and change nothing; every other error exits with status 1. Every ERROR: message goes to stderr, so stdout stays free for the answer or the JSON object; help text and --version go to stdout.

MessageCause and next step
ERROR: unknown flag --stpes for 'run' (did you mean --steps?)The command does not declare the flag. The parenthesis appears when a declared flag is at most 2 edits away. The next line is see: mojo-dllm run --help
ERROR: missing required flag --model for 'run'A required flag is absent. The usage of the command follows
ERROR: missing model path for 'inspect'inspect takes the path as an argument. The usage follows
ERROR: unknown command: bogusThe first argument is not a command. The next line is see: mojo-dllm --help
ERROR: unexpected argument 'x' for 'run'run, tokenize and logits take flags only, and inspect takes one path
ERROR: unknown flag --model=x for 'run'Flags take their value as the next argument. The next line says flags take their value as the next argument: --model x
ERROR: flag --prompt needs a valueA value flag is the last argument
ERROR: flag --steps needs an integer, got 'abc'A numeric flag holds text, or a number too large for 64 bits. --temperature needs a finite number and --tokens and --rows need comma-separated integers
ERROR: flag --prompt needs a value, got the flag --verboseA value flag is followed by the name of a declared flag, which is almost always a forgotten value. --help stays a legal value
ERROR: flag --steps is given twiceA value flag appears twice; the command no longer keeps the last silently. A boolean flag repeated is harmless
ERROR: flag --rows needs comma-separated integers, got '0,,2'An empty entry in --tokens or --rows (a trailing comma stays legal)
ERROR: flag --steps must be at least 1, got 0--max-tokens, --steps, --block-length, --threads and --eb-steps need at least 1, and --eb-stability at least 0. --temperature, --eb-t-min, --eb-bound and --eb-confidence need a finite number of at least 0, --eb-t-max and --prev-temp-inv one above 0 (not only finite), and --threshold a number from 0 to 1. The check runs before the model is mapped
ERROR: prompt 10 + --max-tokens 4096 = 4106 exceeds the context length 4096 (llada.context_length)The canvas is longer than the file's context length. Lower --max-tokens. Exit status 1, before any weight is mapped
ERROR: cannot open model file PATH: No such file or directoryThe path is wrong. The reason is the operating system's. The next line is to download a model, run: ./mojo-dllm pull
ERROR: cannot open model file PATH: Permission deniedThe open failed for a reason other than a missing file, such as Not a directory. The operating system names the reason. No download hint follows, because pull would not fix it
ERROR: file is empty: PATHThe model file has no bytes. Download it again
ERROR: cannot get the size of PATH: Illegal seekThe path opens but is not a regular file (a pipe or a FIFO). The operating system names the reason
ERROR: mmap failed for PATH: No such deviceThe file cannot be mapped (a device or a pseudo file). The operating system names the reason; Cannot allocate memory means the address space is exhausted
ERROR: is a directory: PATHThe model path names a directory
ERROR: flag --prompt is not valid UTF-8: invalid byte at offset Nrun and tokenize refuse a prompt that is not UTF-8 (a Latin-1 text, for example) before the model loads and exit with status 2. N is the byte offset of the first bad byte. Save the text as UTF-8
ERROR: PATH is not a GGUF file (bad magic).A failed download leaves an HTML error page with a .gguf name. Delete the file and run ./mojo-dllm pull again

The remaining errors exit with status 1 and come from the model file itself:

ERROR: unsupported GGUF tensor type: IQ2_XXS (tensor blk.0.attn_q.weight)
ERROR: tensor blk.0.attn_q.weight is Q8_0: matrix kernels exist for Q4_K and Q6_K
ERROR: unsupported model architecture: qwen2 (runs: llada, dream; read only: diffusion-gemma)
ERROR: required metadata key missing: tokenizer.ggml.model
ERROR: tensor blk.0.attn_q.weight extends past the end of the file
ERROR: tensor blk.0.attn_q.weight has offset 16, which is not a multiple of the alignment 32
ERROR: tensor blk.0.attn_q.weight is Q8_0 with 33 elements per row, which is not a multiple of its block size 32
ERROR: tensor blk.0.attn_q.weight is listed twice (entries 3 and 9)
ERROR: metadata key general.name is listed twice (entries 1 and 5)
ERROR: tensor shape mismatch for blk.0.attn_q.weight: [4096, 2048], expected [4096, 4096]
ERROR: non-finite logits at canvas position 3, step 1
ERROR: no GPU found: the GPU backend needs an NVIDIA GPU with compute capability 8.0 or newer (run with --device cpu)

non-finite logits comes from the LLaDA and Dream loops, on the CPU and the GPU: a canvas row whose softmax is not a number (a NaN or +inf logit, or every logit -inf) stops the run instead of committing token 0 there. The position counts from the start of the canvas and the step from 1. One -inf logit among finite ones is a token of probability 0 and passes for LLaDA; Dream's default entropy confidence has no value for such a row, so Dream refuses it too.

With --device gpu, setting MOJO_DLLM_GPU_PROFILE=1 makes --verbose split a step into GEMM, attention and LM head time. It synchronizes the GPU after each kernel group to measure them, so leave it off when you benchmark.

Benchmarks

LLaDA-8B-Instruct, Q4_K_M. 13th Gen Intel(R) Core(TM) i5-13420H, 12 threads, 33.3 GB RAM, performance power profile. Each prompt: 128 generated tokens, 32 denoising steps; median of 3 runs x 3 prompts.

Runtimems / steptokens / sstartuppeak RSSlanguage
mojo-dllm3,0051.331.8 s5.9 GBMojo
mojo-dllm, logits for every position (ablation)3,2681.221.8 s5.9 GBMojo
llama.cpp llama-diffusion-cli5,8070.692.5 s8.4 GBC/C++
diffuse-cpp, cache off9,9960.402.3 s8.2 GBC++
diffuse-cpp, cache + entropy_exit †4,3801.122.6 s8.4 GBC++

Versions: mojo-dllm bdd268b (cpu-llada) and dd0a647 (cpu-diffuse, diffuse-cpp rows only), Mojo 1.1.0 (8189361e), llama.cpp 836d571, diffuse-cpp 1d2bd6a. Evidence: bench/results/2026-10-03-cpu-diffuse.json, bench/results/2026-10-03-cpu-llada.json.

† Not the same work: the inter-step cache reuses stale K/V and entropy_exit can stop early. Shown because it is diffuse-cpp's recommended mode.

LLaDA-8B-Instruct on the GPU, Q4_K_M. NVIDIA GeForce RTX 3050 6GB Laptop GPU (6144 MiB, driver 595.99.02), host 13th Gen Intel(R) Core(TM) i5-13420H. Each prompt: 128 generated tokens, 32 denoising steps; median of 3 runs x 3 prompts. Peak RSS is host memory.

Runtimems / steptokens / sstartuppeak RSSlanguage
mojo-dllm --device gpu22118.132.2 s1.1 GBMojo
llama.cpp llama-diffusion-cli, CUDA, all layers offloaded24316.461.4 s5.3 GBC/C++, CUDA

Step time of mojo-dllm over llama.cpp: median 0.906, range 0.896 to 0.908 across 9 paired runs, with mojo-dllm's step shorter in 9 of them. A ratio below 1 means mojo-dllm's step is shorter.

Versions: mojo-dllm 69e4024, Mojo 1.1.0 (8189361e), llama.cpp 7df8b87. Evidence: bench/results/2026-11-03-gpu-llada.json.

Dream-v0-Instruct-7B, Q4_K_M. 13th Gen Intel(R) Core(TM) i5-13420H, 12 threads, 33.3 GB RAM, performance power profile. Each prompt: 128 generated tokens, 32 denoising steps; median of 3 runs x 3 prompts.

Runtimems / steptokens / sstartuppeak RSSlanguage
mojo-dllm2,9671.351.7 s5.6 GBMojo
llama.cpp llama-diffusion-cli5,5660.722.5 s8.0 GBC/C++

Versions: mojo-dllm bdd268b, Mojo 1.1.0 (8189361e), llama.cpp 836d571. Evidence: bench/results/2026-10-03-cpu-dream.json.

Dream-v0-Instruct-7B on the GPU, Q4_K_M. NVIDIA GeForce RTX 3050 6GB Laptop GPU (6144 MiB, driver 595.99.02), host 13th Gen Intel(R) Core(TM) i5-13420H. Each prompt: 128 generated tokens, 32 denoising steps; median of 3 runs x 3 prompts. Peak RSS is host memory.

Runtimems / steptokens / sstartuppeak RSSlanguage
mojo-dllm --device gpu21918.292.0 s1.2 GBMojo
llama.cpp llama-diffusion-cli, CUDA, all layers offloaded21418.661.3 s5.0 GBC/C++, CUDA

Step time of mojo-dllm over llama.cpp: median 1.022, range 1.000 to 1.037 across 9 paired runs, with mojo-dllm's step shorter in 1 of them. A ratio below 1 means mojo-dllm's step is shorter.

Versions: mojo-dllm 69e4024, Mojo 1.1.0 (8189361e), llama.cpp 7df8b87. Evidence: bench/results/2026-11-03-gpu-dream.json.

LLaDA-8B-Instruct, llama.cpp thread sweep and forward pass alone: 13th Gen Intel(R) Core(TM) i5-13420H; median of 2 runs, one prompt, canvas 148 tokens.

Runtimethreadsms / stepforward pass alone
mojo-dllm123,0143,011 ms
llama.cpp47,2847,098 ms
llama.cpp66,4167,046 ms
llama.cpp86,1406,242 ms
llama.cpp125,7035,748 ms

The ms / step column runs llama-diffusion-cli. The forward pass alone is mojo-dllm's measured forward pass against llama.cpp's llama-bench at the same token count. llama-bench applies a causal mask and computes logits for one position; mojo-dllm's figure includes logits for up to 32 positions and copying the canvas in. CPU busy before a run: at most 6.5%. Evidence: bench/results/2026-10-09-fair-cpu-llada.json.

LLaDA-8B-Instruct on the GPU, forward pass alone: NVIDIA GeForce RTX 3050 6GB Laptop GPU; median of 3 runs, one prompt, canvas 148 tokens.

Runtimems / stepforward pass alone
mojo-dllm --device gpu225225 ms
llama.cpp CUDA, llama-bench -p 148243 ms

The forward pass alone is mojo-dllm's measured forward pass against llama.cpp's llama-bench at the same token count. llama-bench applies a causal mask and computes logits for one position; mojo-dllm's figure includes logits for up to 32 positions and copying the canvas in. CPU busy before a run: at most 3.1%. Evidence: bench/results/2026-11-03-fair-gpu-llada.json.

What is compared

For LLaDA, all three runtimes load LLaDA-8B-Instruct Q4_K_M and denoise a canvas of L = prompt + 128 tokens in 32 steps. For Dream, mojo-dllm and llama.cpp (diffuse-cpp's Dream support needs its own converted file, not benchmarked here) do the same with Dream-v0-Instruct-7B Q4_K_M; both run Dream's timestep schedule over the whole canvas. A step is one full forward pass over the canvas in each of them, so the step counts and shapes match.

RuntimeHow it runsSchedule
mojo-dllmrun --max-tokens 128 --steps 32 --block-length 32LLaDA blocks of 32, low-confidence
mojo-dllm (ablation)the same plus --full-logitslogits for every position, as the other two compute them
llama.cppllama-diffusion-cli -ub L --diffusion-steps 32 --diffusion-eps 0.001timestep schedule
diffuse-cppdiffuse-cli -n 128 -s 32 --no-cache --remasking low_confidenceone block, cosine schedule
diffuse-cpp †diffuse-cli -n 128 -s 32 --remasking entropy_exitits recommended mode

diffuse-cpp cannot read llama.cpp's GGUF files, so it loads its own conversion of the same model (diffuse-cpp/LLaDA-8B-Instruct-GGUF, llada-8b-q4km.gguf, 5.5 GB): F16 embeddings and a slightly different Q4_K / Q6_K mix. It has no tokenizer and receives the token ids mojo-dllm produces. It prints no timings either, so the harness timestamps the two lines it writes around its generate call.

llama.cpp runs its timestep schedule because its block mode needs the prompt length to be a multiple of the block size. The schedules commit tokens in a different order, which changes the text but not, we expect, the cost of a step. That is an argument. The measurement that does not depend on the schedule is the "forward pass alone" table below, which times one pass at the same token count.

† diffuse-cpp's recommended mode reuses K/V between steps and stops early when the canvas has settled. It does less work, so it is not a like-for-like row.

On the GPU

bench/config-gpu.json and bench/config-dream-gpu.json compare mojo-dllm --device gpu with the same llama-diffusion-cli built with CUDA and run with -ngl 99; the harness refuses a CUDA run unless llama.cpp reports every layer on the GPU. Canvas, steps and prompts match the CPU runs. These configs leave the pytest watchdog off ("kill_pytest": false), so each result records how busy the CPU was before every run. Peak RSS is host memory; neither runtime reports GPU memory. The GPU tables above come from these configs.

The test GPU is a laptop part held to 30 W, whose clocks fall under sustained load (bench/checks/2026-10-12-gpu-power-under-load.txt and bench/checks/2026-11-03-gpu-power-limit.txt) and vary from day to day: the same llama.cpp build ran LLaDA steps much slower on 2026-10-12 than on 2026-10-09 (compare bench/results/archive/2026-10-09-gpu-llada.json with bench/results/archive/2026-10-12-gpu-llada.json). The 2026-11-03 file records three builds run back to back at about 1.1 GHz and 30 W: absolute milliseconds follow the clock, and ratios inside one file do not. Both runtimes in one result file run interleaved, prompt by prompt, so compare them within a file, not across dates.

bench/config-fair-cpu.json and bench/config-fair-gpu.json address two questions about fairness: whether llama.cpp does better below 12 threads on a hybrid CPU (a sweep over -t 4, 6, 8, 12), and how the forward passes compare without either runtime's sampler (mojo-dllm's measured forward pass against llama.cpp's llama-bench at the same token count).

Method

bench/run_bench.py does the measuring:

  • three prompts and, by default, three repetitions (the fairness configs use one prompt, and the CPU thread sweep ran two repetitions), runtimes interleaved within each repetition;
  • every run waits until a two-second sample shows the CPU under 10% busy, and starts anyway after 300 seconds (60 in the fairness configs); the busy fraction before each run is recorded as cpu_busy_before, and the tables disclose runs that started above the threshold;
  • the power profile is performance for the duration and restored after;
  • on battery, where a laptop caps CPU and GPU power, it pauses before the next run until the charger is back, and a run the charger left midway is thrown away and repeated (bench/power.py; each run records power_waited_s, power_retries, on_battery, power_source and a snapshot of the supplies it read, and --allow-battery measures on battery instead). The check fails closed: a supply file it cannot read stops the run, and a battery with the adapter offline counts as battery whatever its status says;
  • a watchdog kills any pytest process (other work on the machine started test suites mid-run), and the kills are logged in the result file;
  • generation time excludes model loading, which is reported as startup;
  • peak RSS comes from /usr/bin/time.

The table above is generated from bench/results/*.json by scripts/bench_table.py, and CI fails if a table differs from the evidence file it names. Two gaps are open: --check passes when a table marker is missing, a target file is missing, or the quality or eval evidence is deleted (#141), and an unknown mode exits 0 (#142).

Benchmark environment

A timed run reports whether it had the machine to itself. bench/isolation.py does the checking for bench/run_bench.py, bench/eval_gsm8k.py run and bench/eval_engine.py run; bench/power.py still owns the mains-power check. Each check below is a constant in bench/isolation.py, and an arm can change one only by passing a Thresholds object, which is recorded in the arm's environment file.

Before an arm starts (preflight, one 5 second sample). A FAIL stops the arm and names the offenders; --allow-noisy continues and records that it did. A source that cannot be read is a WARN that names the file, never a pass.

CheckFAIL whenSource
poweron battery (--allow-battery turns it into a WARN)bench/power.py
profilepowerprofilesctl get or /sys/firmware/acpi/platform_profile is not performanceLLVM "Benchmarking tips", pyperf "Tune the system for benchmarks"
governora CPU is not on performance, unless it is powersave under intel_pstate active with preference performancekernel admin-guide/pm/intel_pstate
CPU busymore than 10% of all CPUs busy over the sample/proc/stat
CPU pressurePSI CPU some avg10 above 5%kernel accounting/psi
swap movementany page swapped in or out over the sample/proc/vmstat pswpin, pswpout
memoryMemAvailable below 1.25 times the model file, or the model file's size cannot be read/proc/meminfo
thermala core or package throttle counter rose over the samplekernel admin-guide/thermal/intel_thermal_throttle
GPU processes (GPU arm)any compute process on the GPUnvidia-smi --query-compute-apps
GPU memory (GPU arm)more than 300 MiB usednvidia-smi --query-gpu=memory.used
GPU utilization (GPU arm)mean of three samples above 5%nvidia-smi --query-gpu=utilization.gpu
GPU clock reasons (GPU arm)any event reason other than idle (0x1), a software power cap (0x4) and the display clock setting (0x100)NVML clocks event reasons, clocks_event_reasons.active (clocks_throttle_reasons.active on older drivers)

Swap in use is a WARN, because a run that touches swapped pages measures the disk. The preflight also records the CPU governor and frequency, intel_pstate status and no_turbo, the energy performance preference, the load average, PSI for CPU and memory, MemAvailable, the top CPU processes, the GPU's state (persistence mode, compute mode, power draw, temperature, SM clock, maximum SM clock), the systemd timers due within the next hour (names only) and the number of lines in the user crontab.

Around every problem (Window, sampled every 0.5 s). The window subtracts the CPU time of the benchmark's own process tree, read from /proc/PID/stat, from the busy time in /proc/stat, so a benchmark that uses every core is not contention. A problem is noisy when:

CheckNoisy whenPer-window value recorded
other CPU, meanprocesses outside the tree use more than 5% of the machine's CPU capacityother_cpu_share_mean
other CPU, peakthe same share exceeds 25% in any 0.5 s sampleother_cpu_share_max
swapany page movedswap_pages
thermala throttle counter rosethermal_events
GPU clock reasons (GPU arm)hardware slowdown (0x8), software or hardware thermal (0x20, 0x40), hardware power brake (0x80), applications clocks (0x2) or sync boost (0x10) was activegpu_reasons_seen
GPU processes (GPU arm)a compute process outside the benchmark tree appearedgpu_foreign_pids
samplerthe sampler thread died or did not stopreason sampler-failed
unmeasureda source that exists on this machine could not be read during the windowreason unmeasured:SOURCE

Recorded in every window and never judged: PSI CPU and memory some stall time (psi_cpu_stall_ms, psi_memory_stall_ms), because a 12-thread run on 12 CPUs stalls 9% of the time by itself (measured on the reference laptop: 24 ms in 10 s idle, 908 ms with 12 busy threads), so no limit separates the run from a neighbour; a software power cap (0x4, gpu_power_capped), which the laptop's 30 W GPU hits under any compute load; the display clock setting (0x100); the SM clock minimum and maximum; the window length; and the processes outside the tree that used the most CPU (other_top_processes, percent of one core).

A source that does not exist on the host (no NVIDIA GPU on a CPU arm, no thermal_throttle files on an AMD CPU) is not checked.

The child that exits inside the window is handled: a tree member's ticks are its own utime + stime plus the cutime + cstime of the children it has waited for, and the window compares the sum over the tree between two samples. When a child exits and is reaped, its ticks move from the child to its parent and the sum does not drop. The Window's pid must outlive the window (the runner's own pid does).

The GPU is read every 2 s inside a window (two nvidia-smi processes per read, about 0.2 s of wall time and 0.1 s of CPU time per read, so about 5% of one core of 12 while a GPU arm runs; they are children of the runner, so they sit in the tree and are not counted as other). A GPU event reason that starts and ends between two reads is missed.

What the checks cannot see. Kernel threads and interrupt time are inside busy and are flagged as other CPU, but they cannot be named. Hypervisor and firmware (SMM) time, fan and embedded-controller policy and thermal paste are invisible. GPU use by the display server is not a compute app and counts only through the 300 MiB memory limit and the utilization sample. Other users' processes are seen, but only at the 0.5 s resolution, and a burst shorter than that, or shorter than the 10 ms tick, can fall between samples. A foreign GPU process that starts and ends between two GPU reads is missed. Processes in another pid namespace (containers) can differ from the pids nvidia-smi prints, so a GPU arm inside a container cannot match its own pid. A window can also miss one stolen core for a part of a long problem; the numbers are in the next paragraph.

Sensitivity. The limits (5% of capacity as a mean, 25% in any sample) were kept after measuring the baseline on the reference laptop with the shared machine in its normal state: 60 windows of 10 s around a child that burns one core, other-process share of total capacity: mean of the window means 9.0%, median 8.0%, 95th percentile 20.6%, maximum 25.3%; peak share mean 19.5%, 95th percentile 39.7%, maximum 47.5%. Only 14 of the 60 windows came out clean at these limits, because other sessions were using the machine; the windows named them (a terminal session, python3, qemu). Since the baseline is far above 1%, the limits are not tightened. The consequence is a blind spot: one stolen core for 30 s of a 90 s problem is 2.8% mean and 8.3% peak of 12 CPUs, and the CPU rules do not flag it. On a quiet machine (scripts/bench-quiet.sh, no other session) rerun the baseline and lower the limits by passing a Thresholds.

Before a problem the runner waits up to 60 seconds for a quiet moment (CPU busy, swap movement and, for a GPU arm, GPU processes), then runs it inside a window. A noisy problem is run again, up to --isolation-retries times (default 2); after that the row is kept with its reasons. bench/run_bench.py keeps its own busy_threshold and max_quiet_wait_s config keys for this wait, and records cpu_busy_before as before.

Reading ARM.env.json. The eval runners write bench/eval/runs/ARM.env.json and run_bench.py writes bench/results/env/NAME.env.json. It holds one entry per start of the arm under sessions: the preflight report (each check with its status and detail), snapshot_start and snapshot_end, the thresholds used, tuned (the profile before and after --tune), bench_quiet (true when the run was started under scripts/bench-quiet.sh) and bench_quiet_changes (what that script changed: timers_stopped, gpus_persistence_on, gpus_clocks_locked, and refused, the changes it tried and could not make, a refused clock lock among them; null when the script gave no detail), preflight_failed_allowed (true when --allow-noisy continued past a FAIL), a session_id, the retry limit and tool versions. Each row of an arm file carries isolation: the window values above, clean, reasons, retries (how many repeats this problem took), noisy, cpu_busy_before, waited_s, session (the session_id of the start that wrote the row), allow_noisy and preflight_failed_allowed.

eval_gsm8k.py compare adds generate_s_mean_clean (the mean over the problems whose window was clean in both arms, null when there is none), clean_rows, noisy_rows per arm and unchecked_rows per arm (rows written before windows existed). Accuracy is still computed over every row.

Quieting the machine. --tune sets the power profile to performance through powerprofilesctl, reads it back, and restores the previous profile on exit, on an exception, on Ctrl-C and on SIGTERM. sudo scripts/bench-quiet.sh -- COMMAND additionally stops the system timers sysstat-collect, apt-daily, apt-daily-upgrade, fwupd-refresh, anacron, man-db and motd-news, turns on NVIDIA persistence mode and locks the GPU clocks where the driver allows it, runs the command, and puts everything back; it never edits cron files. After a SIGKILL or a power loss no trap can run: sudo scripts/bench-quiet.sh --restore undoes the script's changes from its state file, and --tune (which keeps no state file) leaves the profile at performance, so run powerprofilesctl set PREVIOUS with the profile in tuned.previous of the arm's environment file. The one-minute user cron job is not stopped: a window that sees it is noisy and the problem is repeated. Not covered: kernel boot parameters (isolcpus, nohz_full) and BIOS settings. The rows keep the 12 threads they were measured with, so no core is left to pin.

Rows measured before this change. The published rows above (and the arms under bench/eval/ written before isolation existed) were not checked this way. They have cpu_busy_before for the CPU rows and no window. Do not read a ratio from them closer than their recorded spread.

Steps and quality

A diffusion model commits tokens / steps tokens per step. The table above uses 4 tokens per step, which is fast and shows in the text as repeated or dropped words. More steps cost proportionally more time, so the headline tokens / s above is not a quality-matched figure. In the first prompt below, the repeated words disappear at 64 steps, which takes about twice the time of 32 steps, so by that count mojo-dllm's quality-matched speed on that prompt is about half the headline rate. Other runtimes use other schedules. The code prompt shows no such damage at 32 steps.

LLaDA-8B-Instruct.Q4_K_M.gguf, 128 generated tokens.

Promptstepstokens / steptimerepeated wordsoutput (first 120 characters)
Explain speculative decoding in two sentences.32493 s99Speculative decoding is a process used in machine communication to improve the likelihood of a message being by by allow
Explain speculative decoding in two sentences.642191 s0Speculative decoding is a technique used in machine translation to improve the accuracy of the translated text by allowi
Explain speculative decoding in two sentences.1281377 s0Speculative decoding is a technique used in machine translation to improve the accuracy of the translated text by allowi
Write a short Python function that checks whether a number is prime.32497 s1def is_prime(n): if n <= 1: return False if n <= 3: return True if n % 2 == 0 or n % 3 == 0:
Write a short Python function that checks whether a number is prime.642194 s1def is_prime(n): if n <= 1: return False if n <= 3: return True if n % 2 == 0 or n % 3 == 0:
Write a short Python function that checks whether a number is prime.1281388 s0Here is a short Python function that checks whether a number is prime: ``` def is_prime(n): if n <= 1: retur

Evidence: bench/quality/2026-10-03-LLaDA-8B-Instruct.Q4_K_M.json (mojo-dllm bdd268b).

"Repeated words" counts immediate repetitions such as "the the", a rough proxy for parallel-decoding damage. bench/quality.py records the runs and scripts/bench_table.py renders this table.

Accuracy of --threshold and --dual-cache

Both options change what a step computes, so their answers can differ from the plain sampler's. A paired GSM8K run checks how often they still get the answer right (bench/eval_gsm8k.py, see Development):

LLaDA-8B-Instruct.Q4_K_M.gguf, GSM8K, 200 problems, flags --max-tokens 256 --block-length 32 --steps 256 --device gpu --json, mojo-dllm 3b2dac4.

Armaccuracyvs baseline95% CIdiscordantMcNemar ppasses / problemtime / problem
baseline80.5%237.079.7 s
dualcache81.5%+1.0 pt-4.4 to +6.4 pt14 / 160.86236.615.8 s
threshold0980.0%-0.5 pt-2.2 to +1.2 pt2 / 11.0073.424.1 s

Discordant counts problems only the baseline solved / only the arm solved. The 95% CIs are Wald intervals, which are too narrow when the arms disagree on few problems. Time per problem is the mean over the first 50 problems of each arm. These rounds carry no isolation record, so nothing here shows that the machine was otherwise idle.

Evidence: bench/eval/2026-10-29-baseline-vs-dualcache.json, bench/eval/2026-10-29-baseline-vs-threshold09.json, bench/eval/2026-10-29-baseline-vs-dualcache-first50.json, bench/eval/2026-10-29-baseline-vs-threshold09-first50.json.

Each row of an arm file (bench/eval/runs/ARM.jsonl) has a schema: mojo-cli for eval_gsm8k.py run, engine for eval_engine.py run. forward_passes is a count the engine measured and is null when it does not count them; an engine row then carries the configured count in steps, and eval_gsm8k.py compare reads the arm's flags.forward_passes_measured and prints no mean for an arm that did not measure. stopped_early is true when the answer-fixed stop rule fired on the step the run ended, and that step can be the last one scheduled: 140 of the 200 baseline rows are stopped_early with all 256 passes, so compare forward_passes with the configured steps to see how much work a row did. Engine rows also record completion_tokens and finish_reason; length means the token limit cut the answer off.

The authors' own code (Transformers arm)

bench/eval_engine.py run --engine hf answers the same 200 problems through the sampling code the model authors publish: generate from generate.py in a checkout of ML-GSAI/LLaDA for LLaDA, and diffusion_generate from the remote code of Dream-org/Dream-v0-Instruct-7B for Dream. The adapter imports the LLaDA file from your checkout and never copies it (the repository has no license file).

This arm runs different weights. The llama.cpp and mojo-dllm arms load the Q4_K_M GGUF. The Transformers arm loads the authors' bf16 safetensors, or a bitsandbytes 4-bit (nf4) copy of them with --quant 4bit. An accuracy difference between the arms therefore includes quantization as well as any difference between the implementations, and eval_gsm8k.py compare refuses such a pair unless you pass --across-weights. The output then holds both model names and "weights_differ": true.

Like-for-like settings: temperature 0, steps equal to the generation length, block length 32 and low-confidence remasking for LLaDA, entropy remasking for Dream (which has no blocks). The Dream README example uses temperature 0.2, top_p 0.95 and 512 steps, and the library default algorithm is origin; the arm departs from that, and the flags of every row say so. version records the model repository's commit sha (--model-revision, a full 40-character sha; null when you give none, and then the weights and the remote code are unpinned), the LLaDA checkout's commit, and the transformers, torch and bitsandbytes versions. The Python sampling loop is not a speed baseline; read accuracy from this arm, not seconds.

The two READMEs pin different environments (LLaDA: transformers==4.38.2; Dream: transformers==4.46.2 and torch==2.5.1), so each model gets its own virtual environment. The repository's frozen environment never installs them. The runner needs only the standard library besides the model libraries, so run it with the environment's own interpreter. The LLaDA README pins no torch; I pin 2.5.1 as for Dream, because an unpinned install resolves a CUDA 13 build that may not match the laptop's driver. Both lines also pin numpy<2; that numpy 1.x works with these transformers versions is an assumption I did not test. I checked each install line below with uv pip install --dry-run only (it resolves: torch==2.5.1 with the CUDA 12.4 libraries, numpy==1.26.4, and bitsandbytes==0.50.2 for the GPU lines); nothing was installed and nothing was run.

# LLaDA, GPU (4-bit). For the CPU path replace the install line with:
#   uv pip install --python "$V/bin/python" --extra-index-url https://download.pytorch.org/whl/cpu \
#       --index-strategy unsafe-best-match "torch==2.5.1+cpu" transformers==4.38.2 accelerate "numpy<2"
V=~/.cache/mojo-dllm-hf/llada
uv venv $V --python 3.12
uv pip install --python "$V/bin/python" torch==2.5.1 transformers==4.38.2 accelerate bitsandbytes "numpy<2"
git clone https://github.com/ML-GSAI/LLaDA ~/src/LLaDA
REV=$(curl -s https://huggingface.co/api/models/GSAI-ML/LLaDA-8B-Instruct | jq -r .sha)
scripts/capped.sh --mem 26G -- "$V/bin/python" bench/eval_engine.py run --engine hf --model GSAI-ML/LLaDA-8B-Instruct \
    --model-revision "$REV" --llada-repo ~/src/LLaDA --quant 4bit --device gpu --arm hf-llada-4bit --limit 50

# Dream, GPU (4-bit); the CPU line is the same with the CPU install line above and transformers==4.46.2
V=~/.cache/mojo-dllm-hf/dream
uv venv $V --python 3.12
uv pip install --python "$V/bin/python" torch==2.5.1 transformers==4.46.2 accelerate bitsandbytes "numpy<2"
REV=$(curl -s https://huggingface.co/api/models/Dream-org/Dream-v0-Instruct-7B | jq -r .sha)
scripts/capped.sh --mem 26G -- "$V/bin/python" bench/eval_engine.py run --engine hf --model Dream-org/Dream-v0-Instruct-7B \
    --model-revision "$REV" --quant 4bit --device gpu --arm hf-dream-4bit --limit 50

python3 bench/eval_gsm8k.py compare baseline hf-llada-4bit --across-weights

The repository's code runs on your machine. The adapter loads with trust_remote_code=True, which executes the Python files of the model repository (for Dream, modeling_dream.py and generation_utils.py) at load. Only a commit sha says which code that is, so --model-revision is required for a Hugging Face repository, and the run refuses without it unless you pass --allow-unpinned (the rows then say unpinned). Look up the sha when you start the arm and keep it: the curl lines above print the current one. A local directory is not on the hub, so it needs no revision; its path is recorded and the sha is marked unverified, because nothing ties the directory to a commit. version also records the LLaDA checkout's commit with -dirty when a tracked file differs from it.

Which arms count as different weights. compare --across-weights reads each arm's model. A 4-bit arm records LLaDA-8B-Instruct|4bit, so it differs from a bf16 arm of the same repository. Two revisions of one repository record the same model; they differ in version.model_revision, which compare does not read.

Memory. Two figures, both estimates and not measurements. Before it loads, the adapter reads MemAvailable from /proc/meminfo and refuses to start the load when it is below the load need: bf16 is 1.25 times the weights plus 3 GB of working memory (LLaDA's float64 Gumbel noise on the CPU is about 0.5 GB per tensor), 4-bit is the weights over 3.5 plus 3 GB. After the load the isolation preflight asks for the 3 GB of working memory only, because the weights are resident by then and MemAvailable already excludes them. The weights' size comes from a table for the two repositories, else from the *.safetensors of a local directory; for any other model the size is unknown and the run refuses before loading, as it does when /proc/meminfo cannot be read. The guard does not replace a cap: run the commands above under scripts/capped.sh (see Development). The loader also passes low_cpu_mem_usage=True.

The weights are 14.93 GiB (LLaDA) and 14.19 GiB (Dream) of safetensors, so one model fits a nearly full disk and both do not. The adapter loads the model before the preflight and the first problem, and refuses a model that is not on the requested device (a model that fell back to the CPU would still answer, slowly).

Not done yet. The generation timeout leaves the generation thread running until the stopped process exits. The tests run against stubs: no weights were downloaded and no GPU was used for this arm. The memory-fit spike (issue #178, scope item 1: peak RAM, peak VRAM and seconds per step for bf16 on the CPU and for 4-bit on the GPU) and the GSM8K arm itself (scope item 4) have not run, because the laptop had 4 GB of disk free and has to stay quiet for the timed benchmark runs. Whether 4-bit LLaDA fits 6 GB of VRAM is unmeasured, and so is the CPU step time. There are no Transformers rows in this document until the arm is committed under bench/eval/.

A running vLLM server (DiffusionGemma)

bench/eval_engine.py run --engine vllm answers the same 200 problems through the OpenAI-compatible endpoint of a vLLM server that is already running. vLLM serves one diffusion model, DiffusionGemma (issue #179). The engine is a client: it never starts or stops a server, loads no weights and installs nothing. Rent the GPU, start the server there, point the client at it, and destroy the instance after. The shared client is bench/engines/_http.py (standard library only); the SGLang engine reuses it.

What the client sends: POST /v1/chat/completions with one user message holding the prompt text every engine uses, temperature 0 and max_tokens equal to --gen-length. The server applies the model's chat template. The client sends no credentials and no cookies, follows no redirects, never retries, ignores http_proxy and its siblings unless you pass --use-env-proxy, refuses a response over 4 MiB, and gives each request one deadline (--timeout-s) that covers connect, headers and body. Text the server sends back (an error message, a model id) is stripped of control characters and capped at 400 characters before it reaches your terminal, so a server cannot write escape sequences into it.

This arm runs different weights. The recipe's FP8 checkpoint is not the Q4_K_M GGUF of the llama.cpp arm. The server does not report its quantization, so --quantization is required and goes into the arm's model (for example diffusiongemma-26B-A4B-it|fp8-dynamic). eval_gsm8k.py compare then refuses the pair unless you pass --across-weights, and an accuracy or speed difference includes the quantization.

What a row records. version is JSON with the release GET /version reports ({"version": ...}), and the checkpoint, quantization and attention backend. Only the release comes from the server. --quantization and --attention-backend are declared, and flags.declared_not_verified lists them. --checkpoint is compared with the root that GET /v1/models reports for the served name, and a difference is refused; with no --checkpoint, the reported root is the checkpoint. --model is the arm's label, and it must equal that root or the served name: a label for other weights is refused, so pass the checkpoint you served (for the FP8 run, RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic), not the base model's name. flags.model_matches_server records it. The chat template is the server's: flags.chat_template says applied_by: server, verified: false, and keeps what you declare with --chat-template. The run is refused before the first problem when the server does not list --served-model, when /version answers no version, and for an HTTP error, an empty completion, a response that names another model, or a response without token counts. The server's address is in neither version nor flags.

Steps. The response carries no denoising steps. vLLM's source defines the counter vllm:diffusion_num_denoising_steps for diffusion models (vllm/v1/spec_decode/metrics.py, read 2026-11-09). The source has no _total; prometheus_client adds that suffix when it exposes a counter, so /metrics should show vllm:diffusion_num_denoising_steps_total. That _total name is inferred from prometheus_client, not seen on a live /metrics, and the engine accepts both names. The counter is server-wide and sums every request the server runs. The engine reads it before and after each generation and records the change, which is this request's steps only when nothing else runs on the server. So the run is refused when the server exposes the counter and you have not declared --max-num-seqs 1, the value you started the server with (the client cannot read it; the recipe's own command sets 4). Run one client at a time even then. When /metrics or the counter is missing, forward_passes is null and flags.forward_passes_measured is false, and --max-num-seqs is not needed; the engine never fills in a number. Any other error from /metrics (403, 500) stops the run: only a 404 means "no such endpoint". A counter that does not move across a request also stops it. I have not checked this counter against a running server. Steps per canvas vary from request to request, because the recipe describes an entropy-bound sampler; the recipe guide's "up to 48 steps per block" is the figure it prints, not an expected value for this server, so read the first rows for their spread and do not check them against a constant. --steps, --block-length and --schedule do not reach the server: the step rule is set when the server starts (--diffusion-config), and a --steps different from --gen-length is refused.

The isolation checks see the client only. bench/isolation.py runs on the machine that runs the client. It does not observe the server's GPU, clocks, power or neighbours, so a clean preflight says nothing about the rented card. Every row says "server": "remote" in flags. --device cpu|gpu selects which local checks apply; run the client with --device cpu on a machine without a GPU. generate_s is the wall time of one request, which includes the network round trip and the server's queue.

Runbook: rent, serve, run, pull, destroy

TODO for the maintainer: the budget. The issue asks for the price and the hours in a pull request before any renting. They are not in this document. Name the card, the hourly price from the offer you pick, the hours and the total cost in the pull request that adds the arm.

Which card. The recipe lists FP8 as verified on H100, H200 and DGX Spark GB10, with a 32 GB minimum (recipes/models/Google/diffusiongemma-26B-A4B-it.yaml in vllm-project/recipes, read 2026-11-09), and vLLM 0.24.0 or newer. An RTX A6000 is not a verified target. The FP8-dynamic checkpoint is 25.33 GiB (Hugging Face API, 2026-11-09).

  1. One time on your machine. Install the Vast.ai CLI, read your API key into an environment variable and create a throwaway SSH key. The key prompt reads without echo and the key never appears in a command line; cancel with Ctrl-C and nothing is written.

    uv tool install vastai
    ssh-keygen -t ed25519 -N '' -C vast-dgemma-throwaway -f ~/.ssh/vast_dgemma
    echo "== Vast.ai API key (console.vast.ai, Account, API Keys) =="
    read -rs -p "key: " VAST_API_KEY; echo; export VAST_API_KEY
    vastai create ssh-key "$(cat ~/.ssh/vast_dgemma.pub)"
    

    Run every later vastai command in this same shell, and close it when you are done: the key lives in that shell's environment, and every program you start from it inherits the variable. This runbook does not use vastai set api-key KEY or --api-key KEY, because both put the key in the command's argument list, which every user of the machine can read through /proc/PID/cmdline (mode 0444 on this machine, checked with ls -l /proc/self/cmdline) while the command runs. A process's environment file, /proc/PID/environ, is mode 0400 here, readable by its owner and root. The CLI reads VAST_API_KEY when no --api-key is given: I read the default of --api-key in vast.py of vast-ai/vast-cli at commit 4727235e75470ab26b231c1bdb1415f6ca815d1d (os.getenv("VAST_API_KEY", ...)), and the Context7 documentation of /vast-ai/vast-cli gives the same order (explicit argument, then VAST_API_KEY, then the key file), both on 2026-11-09. I have not run the CLI, and I did not check which version uv tool install vastai installs. If your version ignores the variable, the first call fails with an authentication error and nothing is stored.

    This touches ~/.ssh/vast_dgemma* and your account's key list (the CLI also creates ~/.config/vastai). To undo it, close the shell, delete ~/.ssh/vast_dgemma* and remove the SSH key in the Vast.ai console, and revoke the API key there if you made it for this run. I have not run vastai create ssh-key here; check it with vastai create ssh-key --help.

  2. Find an offer. Filter by card, reliability, disk, download speed and driver. The field names come from the Vast.ai CLI reference (docs.vast.ai/cli/reference/search-offers); fill in the card name from its listing.

    vastai search offers 'gpu_name=<CARD> num_gpus=1 reliability>0.98 disk_space>=100 inet_down>500 rented=False' -o 'dph'
    

    Pick by the total, not the hourly price: the hourly price times your hours, plus the download (inet_down_cost per GB for about 26 GB of weights and the image). Add --raw to see the cost fields.

  3. Rent it. The recipe's image is vllm/vllm-openai:gemma. Billing starts when the instance runs.

    vastai create instance OFFER_ID --image vllm/vllm-openai:gemma --disk 100 --ssh --direct --label dgemma-vllm
    vastai show instances          # wait for "running"; note the instance ID
    vastai ssh-url INSTANCE_ID     # prints ssh://root@HOST:PORT
    

    --disk 100 is my estimate (25.33 GiB of weights, the image and the cache), not a measurement. I have not run these lines on Vast.ai, and I do not know whether its --ssh mode keeps the image's vllm entry point out of the way. Run vastai create instance --help first; if the server starts by itself, read the instance's logs before you start a second one.

  4. Serve. On the instance (ssh -p PORT -i ~/.ssh/vast_dgemma root@HOST), start the server with the command from the recipe page. The page https://recipes.vllm.ai/Google/diffusiongemma-26B-A4B-it was read on 2026-11-09. It prints this vllm serve command for its default configuration, verbatim:

    vllm serve google/diffusiongemma-26B-A4B-it \
      --max-num-seqs 4 \
      --tensor-parallel-size 1 \
      --enable-auto-tool-choice \
      --tool-call-parser gemma4 \
      --chat-template examples/tool_chat_template_gemma4.jinja \
      --reasoning-parser gemma4
    

    and this docker run for "Single GPU (H100/H200, BF16)", verbatim:

    docker run -itd --name diffusiongemma \
        --ipc=host --network host --gpus all \
        -v ~/.cache/huggingface:/root/.cache/huggingface \
        vllm/vllm-openai:gemma \
            --model google/diffusiongemma-26B-A4B-it \
            --max-model-len 262144 \
            --max-num-seqs 4 \
            --gpu-memory-utilization 0.85 \
            --host 0.0.0.0 --port 8000
    

    The page prints no command for the FP8 checkpoint. Its deployment-notes table also lists --generation-config vllm (the checkpoint's generation_config.json sets max_tokens 256 and would override the per-request limit), --gpu-memory-utilization 0.85 and --diffusion-config '{"canvas_length": 256}' as flags DiffusionGemma needs, and neither command above has them all. Which exact command to run for this arm is the maintainer's decision. The FP8 recipe variant names the model RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic; using that id in place of google/diffusiongemma-26B-A4B-it is my substitution, not the page's, and is not verified on hardware. To record steps the server must also run with --max-num-seqs 1 (the recipe's value is 4; see Steps above). Write the command you ran, as run, into the pull request. Without Docker on the instance (Vast.ai runs the image as the container), take the arguments after the image name and run them as vllm serve MODEL .... If you pass --served-model-name, give that name to --served-model below. The docker run block binds --host 0.0.0.0, and a vLLM server started without --api-key has no authentication. On a rented host that port may be reachable by anyone. Replace it with --host 127.0.0.1 (the tunnel in step 5 reaches it) or close the port with the host's firewall, and never leave an unauthenticated server exposed. The recipe's --reasoning-parser gemma4 and the Gemma 4 chat template can put the answer in the response's reasoning_content and leave message.content empty. The engine then stops with the server returned an empty completion. To see it, curl one chat request by hand against the tunnel and look at choices[0].message. What to change on the server is a question for the recipe: check the recipe page for how it turns thinking on and off, and do not guess a flag. The --generation-config vllm line in the page's deployment notes is mentioned above: the checkpoint's generation_config.json sets max_tokens 256, which would override the per-request limit.

  5. Run the arm from your machine. Open the tunnel in a second terminal, check that the server answers, then run. Results are written on your machine, to bench/eval/runs/ARM.jsonl, so nothing needs pulling off the instance. The client holds no key, and the instance never receives one.

    ssh -N -L 8000:127.0.0.1:8000 -p PORT -i ~/.ssh/vast_dgemma root@HOST      # terminal 2; Ctrl-C closes it
    curl -s http://127.0.0.1:8000/version                                      # {"version":"0.24.0"} or newer
    curl -s http://127.0.0.1:8000/v1/models                                    # note the id and root
    python3 bench/eval_engine.py run --engine vllm --model RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic \
        --arm vllm-dgemma-fp8 --device cpu --gen-length 256 --max-num-seqs 1 \
        --base-url http://127.0.0.1:8000 --served-model SERVED_NAME \
        --quantization fp8-dynamic --checkpoint CHECKPOINT_AS_SERVED --attention-backend ATTENTION_BACKEND_AS_SERVED
    

    Add --limit 3 first to look at the rows. Run one client at a time: the step counter is server-wide. After the run, ss -ltnp 'sport = :8000' must show no tunnel of yours.

  6. Compare. python3 bench/eval_gsm8k.py compare MOJO_OR_LLAMACPP_ARM vllm-dgemma-fp8 --across-weights.

  7. Destroy the instance. A stopped instance keeps billing for its disk.

    vastai destroy instance INSTANCE_ID
    vastai show instances          # must list nothing
    

    Then read the billing page in the console: the final charge should be within the total you computed in step 2. Delete the throwaway SSH key from the account, and remove ~/.ssh/vast_dgemma*.

SymptomLikely causeAction
the run refuses with the server reports the models ...--served-model differs from --served-model-namepass the id /v1/models lists
--checkpoint is ... but the server reports root ...the label does not match what was servedfix the label, or the command
forward_passes is nullthe server exposes no vllm:diffusion_num_denoising_steps counternone: the column stays null
the run refuses with --max-num-seqs 1the server exposes the counter and you did not declare 1restart the server with --max-num-seqs 1, declare it
the run stops on a counter that did not moveanother client, or /metrics is not this model'sstop other clients; read /metrics by hand
--model ... is neither the root ... nor the idthe label names other weights than the server runspass the served checkpoint as --model
returned an empty completionthe answer is in reasoning_contentcurl one request; check the recipe
answered HTTP 4xxfor example max_tokens over the server's limitthe message is the server's own
out of memory on the server--max-num-seqs above 4the recipe sets 4 because diffusion state buffers are large (1 uses less)

Not done yet. No server was run for this engine: the tests use a fake server on 127.0.0.1, and no vLLM, GPU or weights were involved. The fields above were read from vLLM's source and the recipe page, not observed on a live server (/version, /v1/models, /metrics, the counter's per-request meaning). The evaluation (issue #179, scope item 6: batch size 1, the same 200 problems, a 256-token canvas, on a rented card) has not run, so there are no vLLM rows in this document. If the harness's number differs from vLLM's published figures by more than 25 percent on the same card class, explain it here (prompt length, canvas, checkpoint) when the arm lands.

A running SGLang server (DiffusionGemma)

bench/eval_engine.py run --engine sglang answers the same 200 problems through the OpenAI-compatible endpoint of an SGLang server that is already running (issue #180). SGLang serves DiffusionGemma with its dLLM framework and the Gemma4Renoise algorithm, and its cookbook says the model is "Not benchmarked for speed". The engine is a client in the same way as the vLLM one: it never starts or stops a server, loads no weights and installs nothing. It uses the shared bench/engines/_http.py (no credentials, no cookies, no redirects, no retries, one deadline per request, a 4 MiB bound, server text stripped of control characters and capped at 400 characters; see the vLLM section above) and bench/engines/_openai.py (the model-list and completion checks the two engines share).

Every field name below comes from SGLang's source, read on 2026-11-09 at main commit dfaca5681ddc20e31d50f2dfe5dfada269f12bbd, plus the cookbook page https://docs.sglang.io/cookbook/autoregressive/Google/DiffusionGemma.md read the same day. The routes /v1/models, /model_info and /server_info and the Gemma4Renoise algorithm were also compared with the release tag v0.5.23 (2026-11-02) and give the same field names and the same request rules; the other files were read at main only. The issue text and the source differ in two places, and the engine follows the source: SGLang has no GET /version (the version is a field of GET /server_info), and the issue's pull request #23722 for sequence-level forward counts was closed without a merge. No field was seen on a live server.

What the client sends. POST /v1/chat/completions with four fields: model, one user message holding the prompt text every engine uses, max_completion_tokens equal to --gen-length, and stream false (the cookbook says streaming is block-level, one denoised canvas per chunk, and the tokenizer manager's _wait_one_response yields an unfinished output only for a streaming request). Gemma4Renoise.validate_request returns an error for a request that carries logprobs, a grammar (json_schema, regex, ebnf, structural_tag), a penalty, min_new_tokens, min_p, a seed, a logit bias, a custom logit processor or a sampling mask, and the cookbook says such a request is rejected with a 400. The engine sends none of them, and a test scans every request body for them. temperature, top_p and top_k are accepted and have no effect, because the renoise schedule governs sampling; the engine sends none of them either, and flags.sampling says so. A rejected request stops the run with the server's own message. The server applies the chat template; flags.chat_template says applied_by: server, verified: false, and holds the chat_template that GET /server_info reports (null means the checkpoint's own template) and what you declare with --chat-template.

SGLang's own measurement, benchmark/dllm/bench_diffusion_gemma.py, sends pretokenized prompts to /v1/completions with EOS stopping off and a fixed output length, and its README says the token rate it reports includes canvas positions after EOS. This engine sends chat messages and lets the model stop, and it times a whole answer. The two numbers are not the same measurement and should not be compared.

This arm runs bf16 weights. Gemma4Renoise.configure_server_args refuses a quantized checkpoint ("DiffusionGemma does not support quantized loading"), and the engine refuses a server whose /server_info reports a quantization. The arm's model is <name>|<dtype>, for example diffusiongemma-26B-A4B-it|bfloat16, so eval_gsm8k.py compare refuses the pair with a Q4_K_M GGUF arm or the vLLM FP8 arm unless you pass --across-weights, and a difference in accuracy or speed includes the precision.

What a row records, and where each value comes from. version is JSON with the SGLang release, the diffusion algorithm, the checkpoint and the dtype.

ValueRead fromIf the server cannot say
release (version)GET /server_info, field versionthe run is refused; so is an empty string and the development fallback 0.0.0.dev0, which names no release or commit
checkpointGET /model_info, field model_path--checkpoint is recorded as declared and the label stays unconfirmed
algorithmGET /server_info, field dllm_algorithm--algorithm is needed; a null field means the server runs no diffusion model and the run is refused
dtypeGET /server_info, field dtypeauto and a missing field mean "cannot say": --dtype is needed (half and float16 are one dtype, as are float and float32)
batch sizeGET /server_info, field max_running_requestsnull means "cannot say": --max-running-requests 1 is needed

A value the server reports is the value. A declared value that differs from a reported one is refused, and a declared value the server could not confirm is listed in flags.declared_not_verified. --model must equal the reported model_path or the served name (SGLang's /v1/models puts the served name in both id and root, so unlike vLLM root cannot confirm the checkpoint). flags.model_matches_server is true when --model is the reported model_path, false when it matches only the served name (an alias, so the label may name other weights than the checkpoint recorded in flags.checkpoint_reported_by_server), and null when the server reports no model_path. The server's own text is never copied beyond these named fields: /server_info also holds a launch_command and the scheduler's internal state, and neither reaches an arm file. flags.algorithm_config_set says whether the server was started with a --dllm-algorithm-config file (its content is not readable from the client). The server's address is in neither version nor flags.

Batch size 1 is required. generate_s is the wall time of one request, and a second client would add its queue time to it, so the run is refused unless the batch size is 1, reported by the server or declared with --max-running-requests 1. SGLang's own LLaDA 2.1 cookbook serves with --max-running-requests 1 (https://docs.sglang.io/cookbook/autoregressive/InclusionAI/LLaDA-2.1.md, read 2026-11-09). Run one client at a time.

Steps are not recorded. SGLang exposes no per-request step or forward-pass count on main or in v0.5.23. Its Prometheus metrics (python/sglang/srt/observability/metrics_collector.py) have no dLLM counter, and the only pass counter, sglang:cuda_graph_passes_total, is server-wide and counts passes by CUDA-graph mode, so a change of it across one request is not that request's denoising steps. Pull request sgl-project/sglang#48675 ("Expose cumulative request-level forward metrics", which would add meta_info.dllm_metrics to /generate) was open and unmerged when I read it on 2026-11-09, and its description opens with "dLLM responses do not report how many model forwards a request uses". SGLang's own benchmark records --denoising-steps as metadata only. I read the metrics collector at both main and v0.5.23, and neither has a dLLM metric. So forward_passes is null in every row, flags.forward_passes_measured is false, and the engine never reads /metrics and never infers a count. --steps, --block-length and --schedule do not reach the server (--dllm-algorithm-config sets the step rule when the server starts), and a --steps different from --gen-length is refused.

The isolation checks see the client only. bench/isolation.py runs on the machine that runs the client, and every row says "server": "remote" in flags. Run the client with --device cpu on a machine without a GPU.

Runbook: serve, run, destroy

TODO for the maintainer: the budget. The issue asks for the price and the hours in a pull request before any renting. They are not in this document, and I name no card and no price. State the card (or cards), the hourly price from the offer you pick, the hours and the total in the pull request that adds the arm.

Which hardware. The bf16 checkpoint google/diffusiongemma-26B-A4B-it has 51 647 701 024 bytes of .safetensors files, 48.1 GiB (Hugging Face API, revision f7f5b7f5fa82ffc52addd066915886d497f5517b, read 2026-11-09). That is more than a 48 GB card holds before the cache is allocated, so a 48 GB card cannot run this arm alone; the issue's reading is two cards or tensor parallelism. The cookbook prints no memory or parallelism figure and no command with tensor parallelism. ServerArgs has --tp-size (alias --tensor-parallel-size), and I have not run it with this model. Whether one 80 GB card holds the weights and the cache is not settled by any page I read. Settle the hardware with a short trial of a few problems before the full run. SGLang's roadmap issue #39499 lists quantized checkpoints (FP8 first) for DiffusionGemma as an unchecked item, and the algorithm refuses quantized loading today, so plan for bf16.

  1. Rent the machine. Steps 1 to 3 of the vLLM runbook above apply (Vast.ai CLI, key, offer, instance), with these changes. The install guide (https://docs.sglang.io/docs/get-started/install.md, read 2026-11-09) publishes the Docker image lmsysorg/sglang on Docker Hub, says latest is the newest stable release, and says to use a specific release or dated nightly tag for a reproducible deployment. I did not check which tag matches v0.5.23, and I have not run the image on Vast.ai. Size the disk from the 48.1 GiB of weights plus the image; I have no measurement of the image. Billing starts when the instance runs.

  2. Serve. On the instance, start the server with the command from the cookbook. The page https://docs.sglang.io/cookbook/autoregressive/Google/DiffusionGemma.md was read on 2026-11-09. Section 3.1 prints this command, verbatim (the issue quotes the same flags on one line):

    sglang serve \
      --model-path google/diffusiongemma-26B-A4B-it \
      --dllm-algorithm Gemma4Renoise \
      --trust-remote-code \
      --host 0.0.0.0 \
      --port 30000
    

    I compared this block with curl -s of the page's markdown source on 2026-11-09 and the bytes are identical. The page says the Triton attention backend, eager mode and unchunked prefill are selected automatically for Gemma4Renoise, so none is passed. Three changes to it are mine and are not on the page. Add --max-running-requests 1, so the batch size is 1 and the server reports it. Replace --host 0.0.0.0 with --host 127.0.0.1: the page's command binds every interface, and a server started without --api-key has no authentication, so on a rented host that port may be reachable by anyone; the tunnel in step 3 reaches a loopback bind. Do not add --api-key, because the client sends no credential and every request would stop with HTTP 401. Optionally add --dtype bfloat16: without it /server_info reports auto, and you declare --dtype bfloat16 in step 4 (the argument's help text says auto uses BF16 for a BF16 model), which the row then lists as declared and not verified. Write the command you ran, as run, into the pull request.

    Leave --dllm-algorithm-config unset for the reference arm, so the checkpoint's defaults apply (the cookbook lists 48 denoising steps, an entropy bound of 0.1 and temperatures from 0.4 to 0.8). The cookbook's example config marks seed as optional and as what makes the sampling reproducible, so without one two runs of the same problem can answer differently. The page's command has no --reasoning-parser; if you add one, the answer may move to reasoning_content and the engine then stops with the server returned an empty completion.

  3. Run the arm from your machine. Open the tunnel in a second terminal and look at what the server reports. Results are written on your machine to bench/eval/runs/ARM.jsonl, and the instance never receives a key.

    ssh -N -L 30000:127.0.0.1:30000 -p PORT -i ~/.ssh/vast_dgemma root@HOST    # terminal 2; Ctrl-C closes it
    curl -s http://127.0.0.1:30000/v1/models                                   # note the id
    curl -s http://127.0.0.1:30000/model_info | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("model_path"), d.get("served_model_name"))'
    curl -s http://127.0.0.1:30000/server_info | python3 -c 'import json,sys; d=json.load(sys.stdin); print({k: d.get(k) for k in ("version", "dllm_algorithm", "dtype", "quantization", "max_running_requests", "chat_template")})'
    
  4. Run. --model is the checkpoint you served.

    python3 bench/eval_engine.py run --engine sglang --model google/diffusiongemma-26B-A4B-it \
        --arm sglang-dgemma-bf16 --device cpu --gen-length 256 \
        --base-url http://127.0.0.1:30000 --dtype bfloat16
    

    Add --limit 3 first to look at the rows. Add --served-model NAME when the server uses --served-model-name. After the run, ss -ltnp 'sport = :30000' must show no tunnel of yours.

  5. Compare. python3 bench/eval_gsm8k.py compare MOJO_OR_LLAMACPP_ARM sglang-dgemma-bf16 --across-weights.

  6. Destroy the instance. Step 7 of the vLLM runbook above: vastai destroy instance INSTANCE_ID, then vastai show instances must list nothing, then read the billing page.

SymptomLikely causeAction
the run refuses with the server reports the models ...the served name differs from --modelpass --served-model with the id /v1/models lists
GET /server_info ... HTTP 401the server was started with --api-keyrestart it without one and keep the loopback bind
GET /server_info did not report a version string or 0.0.0.dev0a source install without version metadatarun a release, or a checkout with its git history
the server reports no diffusion algorithmthe server runs an autoregressive model, or no --dllm-algorithmserve with the cookbook command
the server does not report its dtype/server_info says autopass --dtype bfloat16 (declared), or serve with --dtype bfloat16
max_running_requests refusalthe server was not started with --max-running-requests 1restart it with the flag
--model ... is neither the model path ...the label names other weights than the server runspass the served checkpoint as --model
answered HTTP 400 ... does not support request-level ...the server rejected a request fieldthe message names the field; report it, the engine sends none of them
returned an empty completionthe answer is in reasoning_content, or the model produced nothingcurl one chat request by hand and look at choices[0].message
out of memory on the serverthe weights plus the cache do not fit the cardsee Which hardware

Not done yet. No server was run for this engine: the tests use a fake server on 127.0.0.1, and no SGLang, GPU or weights were involved. The fields above were read from SGLang's source and cookbook, not observed on a live server (/model_info, /server_info, the dtype and max_running_requests values a real launch reports, the usage.completion_tokens of a dLLM answer, the finish_reason). The evaluation (issue #180, scope item 5: batch size 1, the same 200 problems, a 256-token canvas, on a rented machine) has not run, so there are no SGLang rows in this document.

DiffusionGemma across engines

The four engines that can answer DiffusionGemma are meant to run the same 200 problems (bench/eval/gsm8k-200.jsonl), with the same prompt, the same scoring and the same arm-file format, so eval_gsm8k.py compare --across-weights pairs any two of them. This table is a skeleton and holds no measured value. A row gets a number only from a committed arm file under bench/eval/runs/; until then it says no committed run yet. Weights differ between rows, so every comparison includes the precision as well as the engine.

ArmWeights (as the arm's model records them)AlgorithmHow it runsCommitted run
llama.cpp reference (the branch of ggml-org/llama.cpp#24423)diffusiongemma-26B-A4B-it-Q4_K_M.gguf, the file docs/correctness.md namesthat branch's own loop; I did not read it for this tablethe build is scripts/build_dg_ref.sh; the GSM8K runner for llama.cpp's diffusion CLI is issue #177 (open), and I have not checked that bench/engines/llamacpp.py drives this buildno committed run yet
vLLMdiffusiongemma-26B-A4B-it|fp8-dynamic, or the bf16 checkpoint (the choice is the maintainer's, see the vLLM runbook)vLLM's entropy-bound sampler, as the recipe page describes it--engine vllm (issue #179), runbook aboveno committed run yet
SGLangdiffusiongemma-26B-A4B-it|bfloat16Gemma4Renoise--engine sglang (issue #180), runbook aboveno committed run yet
mojo-dllmthe Q4_K_M GGUF of the llama.cpp rowits own entropy-bound loop, docs/diffusion.md ("DiffusionGemma's loop")mojo-dllm run does not run DiffusionGemma yet: wiring the model into the generation loop is issue #277 (open), see docs/introduction.mdno committed run yet

The six defaults in docs/cli.md (--eb-steps 48, --eb-t-min 0.4, --eb-t-max 0.8, --eb-bound 0.1, --eb-stability 1, --eb-confidence 0.005) are the values the SGLang cookbook lists for Gemma4Renoise (max_denoising_steps, temperature_schedule, entropy_bound, stability_threshold, confidence_threshold; read 2026-11-09). I did not check that the two loops agree step for step. When the arms exist, name each one's commit, server release and algorithm settings next to its row, from the arm's own version and flags.

Limits of this evidence

  • One machine: a 13th Gen Intel i5-13420H laptop for the CPU tables, with an RTX 3050 6 GB laptop GPU for the GPU tables. Another CPU, memory speed or GPU can change the ratios.
  • The four main tables use the median of 3 runs on each of 3 prompts (9 runs); the forward-pass tables use 2 or 3 runs on one prompt; the quality table is a single run per cell. Each table has one model file and one quantization (Q4_K_M).
  • The schedules differ between runtimes (see above), so tokens / s compares runtimes at one step count, not at one output quality.
  • Each table names its commit in a Versions or Evidence line, except the two forward-pass tables, whose commits are in their evidence files. They differ, because the evidence was recorded as the work went on; no single commit holds all of it, and the release tag is not one of them.
  • The quality column counts immediate repeated words on two prompts. It is not a perplexity or a task score. The only task evidence is the GSM8K run for --threshold and --dual-cache on LLaDA.
  • diffuse-cpp loads its own conversion of the model, so its rows are indicative.
  • No binary is published yet (#105): you build from source.

Reading the numbers

  • ms / step is the cost of one denoising pass over the canvas.
  • tokens / s is 128 divided by generation time. A diffusion model's tokens/s depends on the step count you choose, so compare ms / step first.
  • The ablation row shows what computing logits only for candidate positions saves.
  • On the GPU, read ms / step, the step-time ratio line under each GPU table and peak RSS together. The ratio pairs the two runtimes run by run, and one GPU with one quantization (Q4_K_M) stands behind it. mojo-dllm also holds far less in host memory, because it uploads the weights once and releases the file's pages behind it.
  • The thread sweep and the forward-pass-alone tables test the CPU comparison. If llama.cpp ran faster below 12 threads, or mojo-dllm's lead came only from skipping logits, those tables would show it. The sweep stops at 12 because the test CPU runs 12 threads.
  • A note under a table counts the runs that started on a busy CPU and gives the medians over the quiet runs alone.

Diffusion decoding in five minutes

An autoregressive model such as Llama writes left to right. Each forward pass produces one token, and an append-only KV cache makes the next pass cheap.

A masked diffusion model such as LLaDA writes the way you fill in a crossword. It starts from a canvas of [MASK] tokens after the prompt and, at every step, looks at the whole canvas at once, proposes a token for every blank, and commits the few it is most confident about. The rest stay masked for the next step.

prompt: "Explain speculative decoding"           canvas (8 positions shown)

step 0   [M] [M]   [M]     [M]  [M]    [M] [M]     [M]
step 1   [M] [M]   [M]     [M]  [M]    [M] [M]     .
step 2   A   [M]   [M]     [M]  [M]    [M] [M]     .
step 3   A   draft [M]     [M]  [M]    [M] tokens  .
  ...
step 8   A   draft model   guesses     several     tokens .

Each step runs the full transformer over prompt plus canvas. Attention is bidirectional: position 3 can read position 7, which a causal model forbids.

The loop LLaDA uses

mojo-dllm implements the reference sampler from the LLaDA repository (generate.py), with its defaults:

  1. Split the generated region into blocks of --block-length positions and fill them left to right (semi-autoregressive decoding).
  2. Give each block steps / blocks steps and spread its masked positions over them evenly: with 32 masks and 8 steps, 4 tokens per step.
  3. At each step, for every masked position in the block, take the argmax token and its softmax probability.
  4. Commit the k positions with the highest probability ("low-confidence remasking": the uncertain ones stay masked).

At temperature 0 the loop is deterministic. Above 0, LLaDA adds Gumbel noise to the logits before the argmax; mojo-dllm does the same.

Why the runtime looks different

  • No KV cache by default. Any position can change between steps, so every step recomputes keys and values for the whole canvas. --dual-cache (LLaDA only) keeps a block-level cache and gives up exactness; see Command line. A step costs a full prompt pass, about 2 × 7 billion operations per token on the canvas in LLaDA-8B, plus 2 × 0.5 billion for each row whose logits are computed.
  • GEMMs, not GEMVs. A step multiplies weight matrices by a hundred or more activation rows at once. The CPU kernel is built for that shape; see The quantized GEMM.
  • Few rows need logits. Only the masked positions the sampler can commit are candidates (the current block for LLaDA, every masked position for Dream), so mojo-dllm computes the output projection (126 464 columns for LLaDA, 152 064 for Dream) for those rows alone. The LLaDA benchmark ablation measures what that saves.

DiffusionGemma's loop

DiffusionGemma does not mask. Its canvas starts as 256 random token ids over the whole vocabulary and every step re-scores all of them:

  1. Take the temperature t = t_min + (t_max - t_min) * c / S, with c counting down from S (48) to 1: 0.8 on the first step, 0.4 + 0.4/48 on the last.
  2. For each canvas position, find the argmax of its logits, the entropy of softmax(logits / t) and one token drawn from that softmax. mojo-dllm reads each 262 144-wide row once for the argmax and the entropy.
  3. Sort positions by entropy, lowest first, and accept each one while the entropies of the positions before it sum to at most 0.1. The lowest-entropy position is always accepted.
  4. Accepted positions keep their drawn token and every other position gets a fresh random id.
  5. The answer is the argmax canvas. The run stops once that canvas has held for one step and its mean entropy is below 0.005, or after 48 steps.

The six numbers are --eb-steps, --eb-t-min, --eb-t-max, --eb-bound, --eb-stability and --eb-confidence (see Command line). A run takes as many forward passes as steps, so the sampler decides the cost.

Architecture

                 mojo-dllm run
                      |
        +-------------+--------------+
        |             |              |
   GGUF reader    tokenizer     diffusion loop
   (mmap)         (BPE)         (diffusion/sampler.mojo)
        |                            |
        |                       model forward
        |                       (models/transformer.mojo)
        |                            |
        +----> repacked weights ---> quantized GEMM, RMSNorm, RoPE,
               (kernels/packed.mojo)  attention, SwiGLU (kernels/)

Source map

PathWhat it does
src/main.mojo, src/cli/command line: run, tokenize, inspect, logits
src/mojo_dllm/formats/gguf.mojoGGUF v2/v3 reader over a memory map, bounds-checked
src/mojo_dllm/formats/utf8.mojothe UTF-8 validity check the GGUF reader and the tokenizer share
src/mojo_dllm/quant/kquants.mojoscalar F32 / F16 / BF16 / Q5_0 / Q8_0 / Q4_K / Q5_K / Q6_K decoders (the test oracle, and embedding rows)
src/mojo_dllm/kernels/packed.mojoweights repacked 8 rows per SIMD register; the GEMM used for every projection
src/mojo_dllm/kernels/qgemm.mojoactivation quantization (int8, one scale per 256) and a row-major reference GEMM
src/mojo_dllm/kernels/ops.mojoRMSNorm, NORM and NEOX RoPE, bidirectional attention, SwiGLU
src/mojo_dllm/kernels/moe.mojoDiffusionGemma's routing and grouped experts on the CPU
src/mojo_dllm/kernels/softembed.mojoDiffusionGemma's self-conditioning: the softmax of the previous logits times the embedding table
src/mojo_dllm/models/transformer.mojoconfig from GGUF metadata, weight loading, the CPU forward pass, and the DenoisingModel trait the samplers drive
src/mojo_dllm/models/diffusion_gemma.mojoDiffusionGemma's CPU forward pass: sliding and global layers, the region masks and scalars, the unified and the cached path, self-conditioning (logits drives it)
src/mojo_dllm/gpu/kernels.mojoGPU kernels: int8 tensor-core GEMM over Q4_K / Q6_K blocks, flash-style attention, RMSNorm, RoPE, SwiGLU
src/mojo_dllm/gpu/model.mojothe forward pass on the GPU: weights uploaded once, activations kept on the device
src/mojo_dllm/tokenizer/bpe.mojobyte-level BPE with the bailingmoe and qwen2 pre-tokenizers, and the Tokenizer that picks it or gemma4 from tokenizer.ggml.model
src/mojo_dllm/tokenizer/vocab.mojochecks on a token table read from a GGUF file, shared by both tokenizers
src/mojo_dllm/tokenizer/unicode_tables.mojothe code point classes of the pre-tokenizer, generated by scripts/gen_unicode_tables.py
src/mojo_dllm/tokenizer/gemma4.mojoDiffusionGemma's tokenizer (U+2581 spaces, newline-run split, merges by rank, <0xNN> byte fallback) and its chat template
src/mojo_dllm/diffusion/sampler.mojothe denoising loops of LLaDA, Dream and DiffusionGemma (its entropy-bounded loop) and the PRNG
src/mojo_dllm/sys/aligned buffers (mem.mojo), mmap, a dynamically scheduled parallel-for

A forward pass

For a canvas of L tokens and a list of positions whose logits the caller needs:

  1. Embedding rows are decoded from the memory-mapped embedding table (Q4_K in a Q4_K_M file), one row per token.
  2. For each layer (32 in LLaDA-8B): RMSNorm, then the activations are quantized to int8 once and reused by the Q, K and V projections. RoPE rotates Q and K; attention runs over all L positions with no mask; the output projection and residual follow. The FFN quantizes once for gate and up, applies SiLU, quantizes again for down.
  3. In the last layer, only the requested positions continue past attention.
  4. Final RMSNorm and the output projection run on the requested rows only.

The forward pass allocates nothing: every activation buffer is sized once for the longest canvas the run will use.

Memory

The GGUF file is memory-mapped. At load, every projection matrix is repacked into a heap block (see The quantized GEMM) and its mapped pages are released, so the model is resident once. Q4_K stays 4-bit after repacking. Q6_K expands to one byte per weight, which costs about 0.4 GB on LLaDA-8B and buys a simpler inner loop. The embedding table stays mapped and faults in as needed. Norm vectors and biases are decoded into f32 arrays at load.

Threads

sys/parallel.mojo hands out work items from a shared atomic counter. On a hybrid CPU (performance and efficiency cores) a static split makes every parallel region wait for the slowest core; claiming items one at a time lets the fast cores take more. On the development laptop the static split left every GEMM waiting on the four efficiency cores.

The quantized GEMM

Most of a denoising step is matrix multiplication: seven projections per layer, 32 layers in LLaDA-8B, every step (without --dual-cache), over the whole canvas. run --verbose prints the split for your machine. This page explains the kernel that does it.

The shape of the problem

A step multiplies each weight matrix W (N rows × K columns, 4 to 8 bits per weight) by an activation matrix X (L rows × K). L is the canvas length, a few hundred at most. That is a skinny GEMM, and it is compute-bound: each weight is read once per step and used L times.

Activations: int8 with a scale per 256

For a Q4_K or Q6_K projection, each activation row is quantized to int8 with one f32 scale per 256 values, and the sum of each group of 16 values is kept as a float. This is llama.cpp's Q8_K scheme, which keeps the comparison with it fair. Integer dot products are 4× denser than f32 FMAs on AVX2.

Weights: 8 rows per register

GGUF stores each row contiguously. A row-at-a-time dot product needs a horizontal sum per output, and the block scales arrive one scalar at a time. So the loader repacks every matrix once:

for each group of 8 rows, for each 256-weight super-block:
    Q4_K:  d[8], dmin[8] as f32 | scales[8][8], mins[8][8] as u8 | 1024 bytes of nibbles
    Q6_K:  d[8] as f32          | scales[16][8] as i8           | 2048 bytes, one value per byte

Inside the weight bytes, a 32-byte load holds 4 consecutive weights of each of the 8 rows. One vpdpbusd (AVX-VNNI) multiplies those 32 unsigned weights by 4 activation bytes broadcast to every lane and adds into 8 int32 accumulators: 8 rows × 4 multiply-adds in one instruction. The scale of a sub-block is an 8-lane vector, one entry per row, applied once per 16 or 32 weights.

The zero points do not touch the inner loop. Q4_K computes w = d·sc·q − dmin·m, so the second term becomes dmin·m times the activation sum of the sub-block, which the quantizer already stored. Q6_K stores q + 32, and the 32 comes off the same way.

Tiling and threads

A work item is 16 output rows. It walks the tokens 8 at a time for Q4_K and 4 at a time for Q6_K, keeping that many int32 and f32 accumulators in registers, so each loaded weight vector feeds 8 or 4 tokens. Items go to threads through an atomic counter (see Architecture).

Checks

tests/test_quant.mojo compares the decoders with the gguf Python package's dequantization, and both GEMMs with an f64 reference over the same dequantized weights. Swapping the nibble halves or dropping the Q6_K zero point fails these tests; that mutation check runs before changes to this file land.

bench/bench_qgemm.mojo measures the kernel alone on LLaDA's layer shapes.

Q5_0 and Q8_0 CPU matrices

PackedMatrix and pgemm also accept Q5_0 and Q8_0 weights. Their blocks contain 32 weights and one f16 scale, so their activations use QActs(tokens, K, block_size=32). This supports widths such as 704 and 2112 that do not fit the K-quant activation layout. The existing default remains 256, and pgemm checks that the activation block size matches the matrix.

PackedMatrix(src_addr, src_bytes, kind, N, K) takes the size of its source as well as the address, and raises unless src_bytes equals the bytes of N rows of K weights. The tensor loaders pass the byte count the GGUF reader checked against the file, so a shape that disagrees with the tensor is an error and not a read past it.

Each packed block holds eight f32 weight scales and interleaved bytes for eight rows. Q5_0 uses unsigned values with zero point 16; Q8_0 uses zero point 128. The activation sums remove those offsets after the integer dot product. On AVX2, the Q8_0 path splits the high bit before pmaddubsw so full-range unsigned weights cannot saturate its signed 16-bit intermediate. VNNI uses the full-byte dot product directly.

tests/test_packed32.mojo compares both formats at the model's two widths with scalar products of independently dequantized weights. It also covers ragged token tiles, output strides, scale changes between blocks and extreme Q8_0 values, including a build that explicitly targets AVX2 without VNNI.

Routing and grouped CPU experts

kernels/moe.mojo provides the layer primitives for DiffusionGemma's CPU forward-pass integration. route_tokens applies unweighted RMS normalization, the inverse square root of the hidden width, the router scale vector and an F32 projection. It computes softmax, selects the highest-weight experts with lower expert IDs winning ties, and renormalizes the selected weights.

ExpertWeights repacks each expert's slice once, after checking that the byte counts of the fused gate/up tensor and the down tensor match the expert count and dimensions. MoEWorkspace stores reusable buffers large enough for every token to select the same expert. grouped_experts counts and groups token/expert pairs, skips empty experts, and runs batched gate/up and down projections for each active expert. Between the projections it applies gelu_tanh(gate) * up; the scatter-add multiplies each output by the expert scale and the token's router weight.

The tiny MoE tests compare routing and final outputs with a separate scalar reference, including mixed K-quant gate/up and 32-weight-block down matrices. The MoE benchmark measures a complete synthetic layer at the intended model dimensions and retains its raw timing samples.

The GPU path

src/mojo_dllm/gpu/kernels.mojo does the same arithmetic on an NVIDIA GPU, with a layout chosen for tensor cores instead of AVX registers.

Weights stay in GGUF blocks. Upload copies each Q4_K matrix as it is and pads each 210-byte Q6_K block to 224 bytes, so every 16-byte chunk of a block is aligned and the kernels can load 16 bytes at a time.

Activations get one scale per 32. The GPU quantizes activations to int8 with one f32 scale per 32 values (llama.cpp's Q8_1 shape) and keeps scale x sum(q) next to it. One warp quantizes one 32-value block, a value per lane, and reduces its max and integer sum with shuffles, so loads and stores are coalesced. A 32-weight Q4_K sub-block then meets exactly one activation scale:

sum w x = d*sc*dx * sum(q * qx)  -  dmin*m * (dx * sum(qx))

One mma per scale group. A Q6_K thread block computes 64 weight rows by a tile of 32, 64 or 80 tokens. For each 256-wide super-block it copies the weights into shared memory (see the Q6_K paragraph below), copies the matching activations next to them, and each of its 4 warps runs one mma.m16n8k16.s8 per 16-weight scale group for its 16 rows and every 8 tokens. (A Q4_K block scales every 32 weights, so it runs mma.m16n8k32.s8 per sub-block instead.) The int32 result goes to f32 with the two scales and, for Q4_K, the min term.

Q4_K: packed in shared memory, unpacked in registers. Removing the staging work cut the GEMM roughly in half (bench/checks/2026-10-13-gemm-cost-split.json: with staging done once, the GEMM took about 100 ms instead of about 220), while removing the mma saved only a few milliseconds. So a Q4_K block copies each row's 144-byte super-block as it is, in 16-byte loads: the 16-byte header (d, dmin and the packed scales) to one array, the 128 bytes of nibbles to another. A warp unpacks its A fragments in registers: word i of nibble pair p holds sub-block 2p in its low nibbles and 2p + 1 in its high ones at the same position, so one shared load feeds two mmas, and the 6-bit scales are decoded from the header per sub-block. With the weights packed, a block holds 128 rows and runs 8 warps in the same 48 KiB, which halves the activation copies per output. The arithmetic is unchanged, so the logits are too (bench/checks/2026-10-13-gemm3-logits.json).

Prefetching the next super-block. On the kernel before the packed layout, removing staging saved about 120 ms of the GEMM and removing the mma 4 to 7 ms, both measured against the integer-accumulation variant (bench/checks/2026-10-13-gemm-cost-split.json). The packed layout (128-row blocks) took that harness from 258.2 to 215.0 ms (median of three runs, v3_vs_base in the same file) without a breakdown of the staging share. The loop still copied a super-block, waited at a barrier, and only then computed. Now each thread reads super-block b + 1 from global memory into registers (weights, activation bytes and activation scales) before the mma loop of b, and stores it to shared memory at the top of the next iteration, after the barrier that ends b. The Q4_K GEMM went from 148.8 to 87.2 ms per forward (median launch time times launches, middle value of three interleaved traces, bench/profiles/2026-10-13-gemm4-*), and the logits match main on all 84 rows (bench/checks/2026-10-13-gemm4-logits.json). That the gain comes from overlapping the loads with compute is the reading the numbers fit; no hardware counters were collected to confirm it.

Q6_K: the same layout and prefetch. The Q6_K kernel took the route the Q4_K one did. A block of 64 rows copies each row's 224 bytes (low nibbles, high bits, 16 int8 group scales, d) into shared memory as they are, in 16-byte loads; a warp rebuilds each 16-weight group's A fragment in registers (_q6k_frag: nibble, two high bits, minus 32 by an xor and a sign extension), and the next super-block waits in registers while the mma loop runs. Groups accumulate in the same order as before, so the result is unchanged. The padded block is 12 bytes larger than the 212-byte padding it replaces: with 6,217,728 Q6_K blocks in LLaDA-8B Q4_K_M (output, 16 attn_v and 16 ffn_down tensors) that is 74.6 MB (71.2 MiB) more device memory.

Tile width per call. The default rule minimizes token tiles, breaking ties toward the narrower width. RTX Blackwell (compute capability major 12) uses a shape-aware rule for Q4_K: start from that default, then choose the narrowest width that does not increase the estimated number of block waves across the device's SMs. The estimate assumes one resident block per SM; it is a dispatch heuristic, not an occupancy measurement. It can trade more blocks for less work within each block when a wide GPU has spare parallelism. Other device families keep their existing rule. The SM count is read when activation buffers are created, not on every GEMM call.

The LM head's small row count still uses the narrow tile. The available token widths remain 32, 64 and 80; 80 is the widest whose shared memory fits the 48 KiB static allocation limit. Weight layout, activation quantization and accumulation order are unchanged. Token tiles of one weight tile run back to back, so they read it through L2. The exact cross-width GPU test also covers ragged token counts and partial weight-row tiles.

Q6_K on compute capability 12 also offers a 16-token tile for canvases of at most 320 tokens. It chooses among widths 16, 32, 64 and 80 by minimizing (width + 16) * ceil(blocks / SM_count), preferring the narrower width on a tie. For vocabulary projections with at least 65,536 weight rows and at most 32 tokens, it instead uses a 128-row, 32-token tile. Both Q4_K and Q6_K read the same cached SM count; Q6_K keeps its previous rule on other devices and longer canvases. Direct-launch tests compare the new Q6_K tiles exactly with the original tile, including partial token and weight-row tiles.

Shared-memory layout. Rows are padded so the 8 rows a fragment load reads fall in different banks: activation rows are 272 bytes apart (256 plus 16), Q4_K nibble rows 144 bytes (128 plus 16) and Q6_K block rows 240 bytes (224 plus 16). Activation scales are stored by sub-block, then token, then (scale, sum), so the pair of tokens a thread folds after an mma is one 128-bit load. tests/gpu/test_gpu.mojo checks the fragment layout against a CPU matmul on random int8 matrices, and the whole kernel against an f64 GEMM.

Row statistics on the device. A greedy step needs, per masked row, the argmax and a confidence. Copying the logits to the host (in LLaDA, 32 rows of 126,464 floats for a 32-token block) and looping over them there is the largest piece of a step outside the forward pass. One block per row now reads its logits twice: the first pass finds the first index of the maximum (a tie goes to the smaller index, as the host loop picks it), the second accumulates, in f64, the sum of exp(x - max), the sum of exp(x - max)(x - max) and the largest logit at any other index. The sampler finishes from those four numbers: p = 1 / z for LLaDA and maskgit_plus, p minus the runner-up's probability for topk_margin, and sum e (x - max) / z - log z for Dream's entropy. The exponentials are f32, so a confidence differs from the host's f64 loop in the sixth digit; tests bound it at 1e-5 relative.

Fused elementwise kernels. The quantized GEMMs want int8 activations, so each norm and the FFN activation end in a quantize. Run separately, a layer wrote and re-read the normalized rows and the product of the gate and up projections through global memory. Three kernels remove that traffic: add_rmsnorm_quantize adds the projection output into the residual stream, normalizes the sum and quantizes it; rmsnorm_quantize does the same for layer 0, whose input has no add; silu_mul_quantize quantizes silu(g) * u without storing it. The norm kernels give thread t the values t, t + 256, ... so a warp holds 32 consecutive values at each step, which is one quantization block, and the quantize step is the same _quantize_lane code the plain kernel runs. The float expressions and their order match the unfused kernels', so the fused output is identical to the chain's, which the tests assert exactly. The attention output still goes through the plain quantize kernel.

Attention on f16 tensor cores. Each warp owns 16 queries and walks the keys 32 at a time. Q is scaled by 1/sqrt(d) and held in registers as f16 fragments; K is staged row-major and V transposed, both as f16, so every operand load is one 32-bit word. mma.m16n8k16 computes S = Q K^T in f32, the four threads that share a row reduce its max with two shuffles, and the online softmax rescales the running sums. S's accumulator for 16 keys has the same register layout as the A operand of the next mma, so P goes from S to P V without leaving the registers. On LLaDA-8B the logits stay as close to the f32 CPU path as with f32 attention (bench/checks/2026-10-13-attention-f16-logits.json). The kernel never stores a score row, so the canvas length has no fixed limit.

Correctness

mojo-dllm is checked at four levels, each against an implementation it does not share code with.

1. Formats and kernels (unit tests, every commit)

  • GGUF parsing, Q4_K / Q5_K / Q6_K / Q5_0 / Q8_0 / F16 / BF16 decoding: compared with the gguf Python package's dequantization on fixtures it generated (max abs difference 1e-6).
  • Both GEMMs: compared with an f64 reference over the same dequantized weights; relative error under 1% with int8 activations.
  • RMSNorm, RoPE, softmax, bidirectional attention, SwiGLU: closed-form cases.
  • The forward pass of a 2-layer model with random K-quant weights: compared with a numpy forward pass; relative logit error under 2% on the LLaDA model and under 5% on the Dream model (its large Q/K biases sharpen the softmax, which amplifies int8 error), and the same argmax on every position.

The GPU has its own suite, tests/gpu/test_gpu.mojo, which runs on a machine with an NVIDIA GPU (scripts/run-gpu-tests.sh; CI only builds it). It checks each GPU kernel against the CPU function it replaces, the int8 mma fragment layout against a CPU matmul, and the GPU forward pass of both tiny models against the numpy reference, with a 5% bound on LLaDA (attention runs in f16) and 2% on Dream, and the same argmax as the CPU path on every position.

2. Tokenizer (unit tests, every commit)

36 strings covering English, contractions, numbers, accents, combining marks, CJK, Cyrillic, Arabic, Devanagari, emoji, CRLF, non-breaking and em spaces, code, and LLaDA's chat-template special tokens. Every id matches llama-tokenize on the real LLaDA GGUF and on the real Dream GGUF, and decoding restores the input byte for byte.

DiffusionGemma's gemma4 tokenizer has its own 54 cases (runs of spaces and newlines, CJK, emoji, byte-fallback characters, a literal U+2581, and the chat template's markers), with ids from llama-tokenize on the real model file (tests/fixtures/gemma4-tokenizer-golden.tsv, and tests/fixtures/gemma4-tokenizer.gguf holds the vocabulary they are encoded with). Each case is also checked with the BOS the model adds, and decoded back; 53 of the 54 restore the input, and the one that holds U+2581 comes back with spaces, as llama.cpp decodes it. tests/test_gemma4_tiny.mojo pins each rule on a vocabulary small enough to work out by hand.

3. Logits on the real model

tools/ref/llada_ref.cpp links libllama and dumps llama.cpp's logits for a fixed token sequence; mojo-dllm logits dumps ours; bench/parity.py compares them. The sections marked GPU run mojo-dllm logits --device gpu.

The GPU sections also compare against llama.cpp built with CUDA. That build quantizes activations with one scale per 32 values, as the GPU path does, while llama.cpp on the CPU uses one per 256, as the CPU path does. On Dream's prompt rows the two llama.cpp builds disagree with each other, so a single reference would mix that spread into mojo-dllm's numbers. The table under each GPU section shows all three pairs.

LLaDA-1.5.Q4_K_M.gguf

Canvasrowsmedian cosinemasked rows medianworst cosinemedian rel. RMSsame argmaxtop-5 overlap
64 tokens640.999870.999890.9972 (row 13)0.01861 / 644.75 / 5
126 tokens1260.999900.999910.9993 (row 19)0.016112 / 1264.67 / 5

Evidence: bench/parity/2026-10-30-llada-LLaDA-1.5.Q4_K_M.json (mojo-dllm 5c82e8c).

LLaDA-8B-Instruct.Q4_K_M.gguf

Canvasrowsmedian cosinemasked rows medianworst cosinemedian rel. RMSsame argmaxtop-5 overlap
64 tokens640.999850.999880.9951 (row 13)0.01862 / 644.83 / 5
126 tokens1260.999880.999890.9992 (row 19)0.017120 / 1264.67 / 5

Evidence: bench/parity/2026-10-03-llada.json (mojo-dllm 0b3e17a).

LLaDA-8B-Instruct.Q4_K_M.gguf, GPU

Canvasrowsmedian cosinemasked rows medianworst cosinemedian rel. RMSsame argmaxtop-5 overlap
64 tokens640.999810.999830.9960 (row 13)0.02162 / 644.77 / 5
126 tokens1260.999880.999880.9984 (row 24)0.017120 / 1264.74 / 5

The same canvases against a second reference, llama.cpp CUDA, and the two references against each other:

Canvascomparedmedian cosineworst cosinemedian rel. RMSsame argmax
64 tokensmojo-dllm vs llama.cpp CPU0.999810.99600.02162 / 64
64 tokensmojo-dllm vs llama.cpp CUDA0.999880.99520.01760 / 64
64 tokensllama.cpp CPU vs llama.cpp CUDA0.999720.99600.02660 / 64
126 tokensmojo-dllm vs llama.cpp CPU0.999880.99840.017120 / 126
126 tokensmojo-dllm vs llama.cpp CUDA0.999920.99870.015120 / 126
126 tokensllama.cpp CPU vs llama.cpp CUDA0.999830.99730.021117 / 126

Evidence: bench/parity/2026-10-09-llada-gpu.json (mojo-dllm cedc0b5).

DiffuCoder-7B-cpGRPO.Q4_K_M.gguf

Canvasrowsmedian cosinemasked rows medianworst cosinemedian rel. RMSsame argmaxtop-5 overlap
71 tokens710.999940.999960.9964 (row 10)0.01966 / 714.86 / 5
133 tokens1330.999930.999940.9612 (row 2)0.016128 / 1334.72 / 5

Evidence: bench/parity/2026-10-30-dream-DiffuCoder-7B-cpGRPO.Q4_K_M.json (mojo-dllm 5c82e8c).

Dream-Coder-v0-Instruct-7B.Q4_K_M.gguf

Canvasrowsmedian cosinemasked rows medianworst cosinemedian rel. RMSsame argmaxtop-5 overlap
71 tokens710.999920.999930.9795 (row 6)0.03664 / 714.76 / 5
133 tokens1330.999890.999900.9872 (row 22)0.031115 / 1334.41 / 5

Evidence: bench/parity/2026-10-30-dream-Dream-Coder-v0-Instruct-7B.Q4_K_M.json (mojo-dllm 5c82e8c).

Dream-v0-Instruct-7B-Q4_K_M.gguf

Canvasrowsmedian cosinemasked rows medianworst cosinemedian rel. RMSsame argmaxtop-5 overlap
71 tokens710.999930.999940.9668 (row 26)0.03264 / 714.61 / 5
133 tokens1330.999860.999880.9782 (row 11)0.03392 / 1334.22 / 5

Evidence: bench/parity/2026-10-03-dream.json (mojo-dllm 0b3e17a).

Dream-v0-Instruct-7B-Q4_K_M.gguf, GPU

Canvasrowsmedian cosinemasked rows medianworst cosinemedian rel. RMSsame argmaxtop-5 overlap
71 tokens710.999910.999930.9455 (row 26)0.04262 / 714.63 / 5
133 tokens1330.999840.999860.9016 (row 11)0.03785 / 1334.22 / 5

The same canvases against a second reference, llama.cpp CUDA, and the two references against each other:

Canvascomparedmedian cosineworst cosinemedian rel. RMSsame argmax
71 tokensmojo-dllm vs llama.cpp CPU0.999910.94550.04262 / 71
71 tokensmojo-dllm vs llama.cpp CUDA0.999940.81790.02064 / 71
71 tokensllama.cpp CPU vs llama.cpp CUDA0.999870.64920.04462 / 71
133 tokensmojo-dllm vs llama.cpp CPU0.999840.90160.03785 / 133
133 tokensmojo-dllm vs llama.cpp CUDA0.999900.89750.02297 / 133
133 tokensllama.cpp CPU vs llama.cpp CUDA0.999790.87840.03980 / 133

Evidence: bench/parity/2026-10-09-dream-gpu.json (mojo-dllm cedc0b5).

Exact equality is not the goal: both runtimes quantize activations to int8, but they block and accumulate in different orders. The llama.cpp maintainers recommend the same kind of statistical comparison for diffusion models.

The table scores every position of each canvas. On the original Dream-v0 checkpoint the median agrees as closely as on LLaDA, but a few prompt positions do not: they sit on special tokens and newlines, where Qwen-family models carry very large outlier activations that magnify int8 rounding. Masked positions, the ones the sampler reads, agree closely. Which of the two runtimes is nearer an f32 reference at those outlier rows is not measured yet.

Agreement with llama.cpp shows that two int8 implementations land near each other. It does not show that either is right. The only f32 forward-pass reference in this repository is the numpy pass on the tiny models (section 1), and the GPU tests compare against it too.

4. Generated tokens on the real model

llama.cpp's own diffusion example cannot serve as a token-level reference. On the pinned commit it never applies --diffusion-cfg-scale or --diffusion-alg-temp, adds Gumbel noise to one row only, ranks entropy in the wrong direction, and samples from the distribution even at temperature 0. So llada_ref generate re-implements LLaDA's generate.py (temperature 0, low-confidence remasking) on top of llama.cpp's logits, and the result is compared token by token with mojo-dllm run. For Dream, llada_ref generate-dream does the same with Dream's diffusion_generate.

A diffusion loop can amplify small differences: when two candidates are nearly tied, the runtimes can commit different tokens at one step, and every later step then sees a different canvas. The table's first difference is a position in the final output; it does not identify the denoising step or the cause of the mismatch. Exact token agreement is specific to the measured prompt and configuration.

What this section covers: two prompts, 32 and 64 generated tokens (the benchmarks use 128), temperature 0, 32 steps, with low-confidence remasking for LLaDA and entropy ranking for Dream. The tables below give the count of equal tokens and the first difference for each model; on the speculative-decoding prompt several are well short of a full match, and Dream-v0's code prompt diverges within its first few tokens. Not covered: Gumbel noise (temperature above 0), random remasking, Dream's other remasking algorithms, classifier-free guidance (cfg_scale), alg_temp, and generation at 128 tokens. --threshold and --dual-cache have a paired GSM8K comparison instead (Benchmarks).

LLaDA-1.5.Q4_K_M.gguf

Promptgeneratedstepssame tokensfirst difference
Explain speculative decoding in two sentences.323224 / 32position 21
Write a short Python function that checks whether a number is prime.643264 / 64none

Evidence: bench/parity/2026-10-30-llada-LLaDA-1.5.Q4_K_M.json (mojo-dllm 5c82e8c).

LLaDA-8B-Instruct.Q4_K_M.gguf

Promptgeneratedstepssame tokensfirst difference
Explain speculative decoding in two sentences.32328 / 32position 5
Write a short Python function that checks whether a number is prime.643264 / 64none

Evidence: bench/parity/2026-10-03-llada.json (mojo-dllm 0b3e17a).

LLaDA-8B-Instruct.Q4_K_M.gguf, GPU

Promptgeneratedstepssame tokensfirst difference
Explain speculative decoding in two sentences.32328 / 32position 5
Write a short Python function that checks whether a number is prime.643264 / 64none

Evidence: bench/parity/2026-10-09-llada-gpu.json (mojo-dllm cedc0b5).

DiffuCoder-7B-cpGRPO.Q4_K_M.gguf

Promptgeneratedstepssame tokensfirst difference
Explain speculative decoding in two sentences.323222 / 32position 17
Write a short Python function that checks whether a number is prime.643263 / 64position 14

Evidence: bench/parity/2026-10-30-dream-DiffuCoder-7B-cpGRPO.Q4_K_M.json (mojo-dllm 5c82e8c).

Dream-Coder-v0-Instruct-7B.Q4_K_M.gguf

Promptgeneratedstepssame tokensfirst difference
Explain speculative decoding in two sentences.323219 / 32position 18
Write a short Python function that checks whether a number is prime.643213 / 64position 13

Evidence: bench/parity/2026-10-30-dream-Dream-Coder-v0-Instruct-7B.Q4_K_M.json (mojo-dllm 5c82e8c).

Dream-v0-Instruct-7B-Q4_K_M.gguf

Promptgeneratedstepssame tokensfirst difference
Explain speculative decoding in two sentences.323224 / 32position 22
Write a short Python function that checks whether a number is prime.643243 / 64position 3

Evidence: bench/parity/2026-10-03-dream.json (mojo-dllm 0b3e17a).

Dream-v0-Instruct-7B-Q4_K_M.gguf, GPU

Promptgeneratedstepssame tokensfirst difference
Explain speculative decoding in two sentences.323226 / 32position 23
Write a short Python function that checks whether a number is prime.643228 / 64position 1

Evidence: bench/parity/2026-10-09-dream-gpu.json (mojo-dllm cedc0b5).

Reproduce

The added LLaDA-1.5, Dream-Coder-v0-Instruct-7B and DiffuCoder-7B-cpGRPO records also contain CPU smoke output, peak RSS, prompt token IDs, GGUF SHA256 and pinned Hub revisions. Their metadata audits compare vocabulary, layer count, context length and RoPE theta with the source HF configs. These runs use four CPU threads and llama.cpp 836d57176dc699a726c55418e4f96b8ca628e1bf; they add no GPU evidence.

LLaDA-1.5 and Dream-Coder carry the same chat templates as their HF tokenizer configs. DiffuCoder-cpGRPO has no HF chat template, while its GGUF uses the Dream-Coder template. Its HF pad token is <|dlm_pad|> (151667), but its GGUF padding ID is 151643. The smoke records explicitly note whether that pad token or a mask token appeared in the text. DiffuCoder's HF license is apple-amlr, despite the GGUF license field saying unknown; obtain it separately and pass a local file.

Build llama.cpp at the revision above with GGML_CUDA=OFF, then set LLAMA to that checkout before compiling the reference:

c++ -O2 -std=c++17 tools/ref/llada_ref.cpp -I$LLAMA/include -I$LLAMA/ggml/include \
    -L$LLAMA/build/bin -lllama -lggml -lggml-base -Wl,-rpath,$LLAMA/build/bin -o build/llada_ref
uv run --only-group fixtures python bench/parity.py --arch llada --model models/LLaDA-8B-Instruct.Q4_K_M.gguf
uv run --only-group fixtures python bench/parity.py --arch dream --model models/Dream-v0-Instruct-7B-Q4_K_M.gguf
uv run --only-group fixtures python bench/parity.py --arch llada --threads 4 --model models/LLaDA-1.5.Q4_K_M.gguf
uv run --only-group fixtures python bench/parity.py --arch dream --threads 4 --model models/Dream-Coder-v0-Instruct-7B.Q4_K_M.gguf
uv run --only-group fixtures python bench/parity.py --arch dream --threads 4 --model models/DiffuCoder-7B-cpGRPO.Q4_K_M.gguf
uv run --no-project python scripts/bench_table.py --write

llada_ref refuses a token, a row or a count that is not an integer, a row outside the tokens, a token outside the vocabulary, an output it cannot write and any non-finite logit in a row it reads. Each refusal prints a line that begins llada_ref: and exits non-zero (1 for the arguments, 2 for the model or a file, 3 for llama.cpp or a NaN). It writes OUT.part and moves it to OUT only after every row passed, so a failed run leaves no file under the name bench/parity.py reads. tests/python/test_llada_ref.py builds it with the host compiler against a stand-in for llama.cpp and runs each refusal.

The GPU tables need --device gpu and a second reference. Compile llada_ref once more against the CUDA build of llama.cpp (build-cuda/, see Development), name the result build/llada_ref_cuda, and add --device gpu --ref2 build/llada_ref_cuda to the bench/parity.py commands.

DiffusionGemma

Mainline llama.cpp cannot load a DiffusionGemma GGUF. The reference is the branch of ggml-org/llama.cpp#24423, built at one commit by scripts/build_dg_ref.sh (CPU only, no network after the fetch) into its own tree, $DG_LLAMA_SRC, beside the mainline checkout. The commit is the PINNED_COMMIT in that script, and tests/fixtures/gemma4-tokenizer-golden.provenance.json records the same one. tools/ref/dg_ref.cpp is linked against that build and writes the canvas logits of one denoising step: the unified [prompt | canvas] pass or the prompt K/V path (a prefill of the prompt, then one decode step), each without self-conditioning and with it, from a previous-step buffer given on the command line.

The model file is 16.81 GB and mojo-dllm keeps its own repacked copy beside the mapping the reference holds, so the two never run together. bench/parity.py runs them in turns and compares saved files:

scripts/build_dg_ref.sh
uv run --only-group fixtures python bench/parity.py --arch diffusion-gemma --phase ref \
    --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --llama-src "$DG_LLAMA_SRC"
# mojo-dllm writes build/dg-parity/<model>/caseN/<variant>.mojo.f32 (the forward pass is #64)
uv run --only-group fixtures python bench/parity.py --arch diffusion-gemma --phase compare \
    --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf

Each <variant>.mojo.f32 has a <variant>.mojo.f32.json beside it with the case_sha256 of its case (bench/parity.py dg_case_hash: prompt, prompt tokens and canvas) and the mojo_dllm_commit of the build that wrote it. The comparison needs the cases case0 to case{N-1} of the current prompt set and stops with a message while any mojo-dllm file or sidecar is missing, was written for another case, or comes from a different build than the others, and refuses a file that is byte-identical to the reference's. It writes evidence to bench/parity/ only when every variant is no worse than the worst case of the LLaDA comparison in bench/parity/2026-10-03-llada.json, on each of minimum cosine, argmax agreement, top-5 overlap and maximum relative RMS. The bound is read from that file, not typed. A fifth metric holds the largest logit of every row to the same relative error as LLaDA's rows reach: the final soft-cap pins that logit near 30, so the first four cannot see the cap removed. bench/check_dg_bound.py shows what the bound rejects.

Each case records how far the reference lies from itself: the prompt K/V path against the unified pass, and the unified pass again with the previous-step logits changed by one part in a million. On the two LLaDA prompts with a random canvas those distances exceed the LLaDA bound (the one-in-a-million change moved the output by a median relative RMS of 0.18 to 0.26). No LLaDA spread was measured for comparison. The cause is untested: expert routing flips (8 of 128 experts per token) carried to every row by bidirectional attention are a hypothesis. A failed comparison prints these distances beside the verdict. No DiffusionGemma result is recorded yet, so the tables above hold none.

The tokenizer goldens in tests/fixtures/gemma4-tokenizer-golden.tsv fix the encoding, and tests/test_tokenizer_gemma4.mojo checks the Mojo tokenizer against them. Decoding the ids through the GGUF vocabulary restores 53 of the 54 strings; the string holding a literal U+2581 does not come back, because decoding turns every U+2581 into a space.

Development

Setup

uv sync --frozen                      # the pinned Mojo toolchain, in .venv
scripts/ci-local.sh --install-hooks

scripts/ci-local.sh also needs shellcheck, markdownlint-cli2 (npm install --global markdownlint-cli2@0.23.2), mdBook 0.4.40 and an authenticated gh. A missing tool fails its step; it never skips it.

The hooks run scripts/ci-local.sh --fast before each commit (hygiene scan, mojo format, -Werror build) and the full set before each push. The pre-commit build runs before its commit exists, so its binary records the parent HEAD, with a -dirty suffix when the commit changes src/. The launcher rebuilds when a source file is newer than the binary, or when build/mojo-dllm.stamp (written by the launcher's own build, not by scripts/build.sh) is missing or names another Mojo version or GPU architecture. After a hook build with a matching stamp the launcher sees a current binary and keeps reporting the parent commit: run scripts/build.sh after committing to stamp the new HEAD.

The gates

scripts/ci-local.sh runs the gates of .github/workflows/ci.yml and, on a machine with a GPU, the GPU tests that CI only builds:

StepCommandFails when
hygienescripts/check_hygiene.sh --trackeda credential pattern (a provider token, a private key block, a database URL with a password) or the author's employer name appears in tracked text, or a tracked Markdown file outside bench/results/ holds an em or en dash; file names are read NUL separated, so a name with non-ASCII or special characters is scanned too)
formatscripts/check_format.sh (mojo format -q src tests bench, comparing every file before and after)mojo format changes any file, an untracked one included (Mojo has no check mode)
buildscripts/build.shany compiler warning (-Werror)
embedded build commitscripts/check_build_id.shthe built binary does not report the expected commit, or a build without Git metadata does not report unknown
testsscripts/run-tests.sha test fails, a file runs zero tests, the total differs from tests/test-count.txt, or a file's count differs from tests/test-counts.txt
launcher testsuv run --locked --only-group test pytest tests/python -qa test of the ./mojo-dllm launcher fails
script lockuv lock --script mojo-dllm --checkmojo-dllm.lock is missing, or the # dependencies header of ./mojo-dllm no longer matches it. After you change the pin, run uv lock --script mojo-dllm and commit the lock
shellcheck, yamllint, markdownlintshellcheck --severity=warning scripts/*.sh .githooks/*; uv run --frozen --only-group lint yamllint --strict .github; markdownlint-cli2 "*.md" "docs/**/*.md" "assets/*.md" ".github/*.md" "#node_modules"any of the three reports a finding
claimsuv run --no-project python scripts/bench_table.py --checka table in the README, docs/benchmarks.md or docs/correctness.md differs from what bench/results, bench/parity, bench/quality and bench/eval render (a missing table marker, target file or quality or eval evidence passes, #141; an unknown mode exits 0, #142)
docsmdbook build + scripts/check_links.pya relative link is broken
action pinsscripts/verify-action-pins.sh (needs an authenticated gh)a pinned SHA is not the tag its comment names
eval self-testuv run --no-project python bench/eval_gsm8k.py --self-testanswer extraction or the paired statistics break, or gsm8k-200.jsonl is not 200 rows with numeric answers
checker self-testuv run --frozen --only-group fixtures python scripts/check_weights.py --self-testa safetensors or index guard in check_weights.py stops rejecting its case, or stops naming it
no GPUscripts/check_no_gpu.sh--device gpu without a GPU does anything but exit 1 with a message
visual optionsscripts/check_visual.shan invalid --visual-interval does not exit 1, or --json with --visual does not exit 2, with a message naming the flag before model loading
fixturesuv run --frozen --only-group fixtures python scripts/check_fixtures.pyscripts/gen_fixtures.py or scripts/gen_moe_fixture.py writes a file that differs from the tracked one (they run into a scratch directory), a file that is not tracked, or a tracked fixture that no generator writes and that is not a tokenizer fixture, a tokenizer golden or golden-generation.tsv
GPU test countscripts/check_gpu_test_count.shthe def test_ functions of tests/gpu/test_*.mojo number other than tests/gpu/test-count.txt pins or differ per file from tests/gpu/test-counts.txt, or a file defines none; needs no GPU, runs in CI and ci-local.sh
GPU testsscripts/run-gpu-tests.sh (local only)a GPU kernel or the GPU forward pass disagrees with the CPU or the f32 reference, or the count differs from tests/gpu/test-count.txt or tests/gpu/test-counts.txt

GitHub's runners have no GPU, so CI builds the GPU suite without running it. ci-local.sh runs it when nvidia-smi lists a GPU and otherwise builds it and prints a SKIP line for the run, never a PASS; run-gpu-tests.sh on its own exits 1 without a GPU.

The launcher tests run pytest -rs, so every skip is listed in the summary, and tests/python/skip_policy.py fails the run on a skip that tests/python/allowed-skips.txt does not name (a test id and the text of the reason). Add a line there in the commit that adds a skip. --locked makes uv run fail when uv.lock no longer matches pyproject.toml; --frozen does not compare them.

Branch protection on main requires four checks: hygiene, mojo and docs of the CI workflow and trufflehog of the secret scan. The job build of the "Deploy docs to GitHub Pages" workflow skips while the repository is private, and a skipped job satisfies a required check, so it must not be required. CI already runs the same mdbook build in docs.

The test count is pinned because Mojo's TestSuite exits 0 after running zero tests. If you add tests, run UPDATE_TEST_COUNT=1 scripts/run-tests.sh (for tests/gpu/, UPDATE_TEST_COUNT=1 scripts/run-gpu-tests.sh) and commit the new count with them.

Tests

scripts/run-tests.sh compiles each tests/test_*.mojo and runs it. Fixtures live in tests/fixtures/ and come from scripts, never from hand:

FixtureMade byChecks
kquants.gguf, bad-magic.gguf, tiny-llada.gguf, tiny-llada.expected.f32, tiny-dream.gguf, tiny-dream.expected.f32, malformed/, metadata/, gen_fixtures.manifestscripts/gen_fixtures.pyGGUF parsing and rejection, decoders, ModelConfig validation, the forward pass against a numpy reference
tiny-diffusion-gemma.gguf, diffusion-gemma/scripts/gen_fixtures.pythe diffusion-gemma config (seed 610061): a 2-layer model with one sliding and one global layer, 4 experts, the tensor names and types of the real file, and one file per refusal (metadata only, or zero-filled tensors)
tiny-diffusion-gemma.p0sc.case, .p9.case, .p9sc.case, .p150.case, .p150sc.case (tiny-diffusion-gemma.*.case)scripts/gen_fixtures.py (the forward pass is scripts/dg_numpy.py)the tokens and numpy logits that logits on the tiny DiffusionGemma file must reproduce, for a prompt of 0, 9 and 150 tokens, with and without self-conditioning (sc)
entropy-rows.tsvscripts/gen_fixtures.py (seed 65065, so no other fixture changes)logits rows with the numpy f64 argmax, entropy and inverse-CDF draw that tests/test_entropy_bound.mojo pins
utf8-cases.tsvscripts/gen_fixtures.pybyte strings with the code points and the first invalid offset that Python's own UTF-8 decoder gives, for utf8_decode
dequant-types.gguf, unsupported-embedding.ggufscripts/gen_fixtures.pyQ8_0/Q5_K/BF16/Q5_0 references (seed 340034) and embedding rejection before CPU/GPU allocation (no RNG)
llada-tokenizer.gguf, llada-tokenizer-golden.tsv, llada-tokenizer-golden.provenance.json, dream-tokenizer.gguf, dream-tokenizer-golden.tsv, dream-tokenizer-golden.provenance.jsonscripts/gen_tokenizer_fixture.py (--golden-only rewrites the golden and its provenance)tokenizer ids against llama-tokenize. Each provenance file records the llama.cpp commit, the model file and its sha256, the command and the golden's sha256. The two goldens were committed with the first import, and regenerating them with llama.cpp 2e715c0639 on 2026-11-09 gave the same bytes
gemma4-tokenizer.ggufscripts/gen_tokenizer_fixture.py ... gemma4the 262,144 tokens of the DiffusionGemma file and the 472 of its 514,906 merges that apply to the 54 golden strings, tokenizer keys only and without the scores (5.5 MB)
gemma4-tiny/scripts/gen_fixtures.pya 279-token gemma4 vocabulary whose ids can be worked out by hand, and variants with one thing wrong or unusual each (a missing byte token, a merge with no space, mismatched counts, nested special tokens, an unknown-type token, an end-of-turn token the vocabulary lacks)
gpt2-tiny/scripts/gen_fixtures.pya six-token byte-level BPE vocabulary with padding, unknown and EOS ids and no end-of-turn token: the ids that LLaDA's and Dream's files leave out
gemma4-tokenizer-golden.tsv, gemma4-tokenizer-golden.provenance.jsonscripts/gen_tokenizer_fixture.py ... gemma4 (--golden-only rewrites only these)DiffusionGemma tokenizer ids against llama-tokenize from the ggml-org/llama.cpp#24423 branch at the commit scripts/build_dg_ref.sh pins; the provenance file records that commit, the model file's sha256 and the command
golden-generation.tsvUPDATE_GOLDEN=1 build/tests/test_goldentiny LLaDA/Dream token sequences, forward-pass counts, early stopping and per-step commits
src/mojo_dllm/tokenizer/unicode_tables.mojoscripts/gen_unicode_tables.pycode point classes from llama.cpp's own tables
moe-q5.bin, moe-q8.bin, moe-q4-q5.bin, moe-q4-q8.binscripts/gen_moe_fixture.pydeterministic raw expert tensors for scalar MoE comparisons, including mixed activation block sizes
GGUF/source tensor shape reportscripts/check_weights.py (--self-test)LLaDA and Dream name mapping and safetensors header shapes; run with uv run --only-group fixtures python scripts/check_weights.py MODEL.gguf INDEX.json (the second argument may be one .safetensors file)

tests/gpu/test_gpu.mojo holds the GPU tests: each kernel against the CPU function it replaces, and the GPU forward pass on both tiny models against the f32 reference.

The tiny model is 2 layers of hidden size 256 with random K-quant blocks. Its expected logits come from a float numpy forward pass over the weights that the gguf package dequantizes, so the Mojo forward pass is checked against the reference formats, not against itself.

Sampler goldens

tests/test_golden.mojo pins current behavior on the tiny fixtures. These outputs come from mojo-dllm itself: they detect changes, not independent correctness. The numpy forward-pass tests and real-model parity runs provide that evidence. No model download or GPU is needed.

Run from the repository root:

bash scripts/build.sh tests/test_golden.mojo build/tests/test_golden
build/tests/test_golden
# Only after reviewing an intentional behavior change:
UPDATE_GOLDEN=1 build/tests/test_golden

Only UPDATE_GOLDEN=1 writes the file, and scripts/run-tests.sh drops that variable from the environment (and says so) unless you run scripts/run-tests.sh --update-golden: the test rewrites the golden and then compares against what it wrote, so an exported variable would make every run pass. Normal runs reject missing, empty, duplicate, unknown or incomplete cases and compare tokens and schedule metadata exactly. A mismatch names the case, the first differing token position (including the prompt), both token lists and both complete rows. A golden diff in a PR needs a sentence explaining why behavior changed, and test_model.mojo must still pass. Do not regenerate merely to silence a failure on another machine; inspect the differing logits and select a case without an unstable near-tie.

The 14 cases cover LLaDA block geometry, uneven transfer counts, serial commits, random remasking, temperature and early stop, plus Dream's three confidence rules, uneven timesteps, nondefault eps and early stop. LLaDA uses prompt [5, 17, 300]; Dream uses [448, 50, 134] because the original prompt gave identical all-49 outputs for D1 and D5. To avoid duplicate token lists, L5 uses seed 1, D2 uses eps=0.2, D3 uses a 12-token canvas and D4 uses eps=0.1. D1/D5 retain the same prompt, length and steps and differ only in eps. L8/D6 use fixed stop ids 347/49 taken from the baseline outputs, never selected at runtime. All cases disable threshold and dual-cache acceleration.

Heavy jobs

A job that holds more memory than the machine can spare makes the kernel's global OOM killer pick a victim. With systemd's default OOMPolicy=stop, systemd then stops every process in the victim's scope, and a job started from a terminal lives in that terminal's scope: the shell, the editor session and every other session in the window end with it. A measured run of DiffusionGemma-26B on the CPU peaked at about 25.6 GiB resident, and on a 30 GiB machine with a 2 GiB swapfile that run ended in exactly this way.

scripts/capped.sh starts a command in a scope of its own, under a hard cap, so only that scope can fail:

scripts/capped.sh --mem 26G -- build/mojo-dllm run --model model.gguf --prompt "hello"

It sets four things, each for a reason you can check in the documentation of the kernel's cgroup v2 memory controller and of systemd's resource control:

SettingEffect
MemoryMax=26Gthe hard cap (memory.max); the kernel OOM killer runs inside the scope when it is reached and kills nothing outside it
MemorySwapMax=0no swap, so a job over its cap dies instead of dragging the desktop through swap
OOMPolicy=killone OOM kill ends the whole scope (memory.oom.group), not half of it
CPUQuota=N00%only with --cpus N, so a job leaves cores for the desktop

It never sets MemoryHigh. Below the cap, MemoryHigh throttles a process that has nothing to reclaim until it stands still (state D, no progress); a job that needs its memory should be killed, not frozen.

The script also refuses to start a job in two cases:

  • /proc/meminfo reports less MemAvailable than --need (default: --mem). A cap larger than the available memory does not protect the machine, because the global OOM killer still runs. Exit status 75.
  • The systemd user manager cannot enforce a memory limit (no cgroup v2, or the memory controller is not delegated to it). It never runs the command uncapped. Exit status 69.

--dry-run runs both checks and prints the systemd-run command without running it. When a capped job dies with exit status 137, the cap was most likely reached; journalctl -k | grep oom-kill tells the cases apart: constraint=CONSTRAINT_MEMCG is the cap working, while constraint=CONSTRAINT_NONE with global_oom means the whole machine ran out of memory.

The test suite checks the limits the kernel reports for a capped scope (memory.max, memory.swap.max, memory.oom.group). The test that kills a process at its cap runs only with CAPPED_TEST_KILL=1, because GNOME's gsd-housekeeping answers every such kill with an "Application Stopped" notification.

For a container, set the same two limits on the container: podman run --memory 12g --memory-swap 12g (--memory-swap counts memory plus swap, so equal values mean no swap).

What does not isolate a job on its own:

  • ulimit -v limits address space, which counts mapped files such as a 16 GB model that is read lazily, and it does not stop a process that stays below the limit but takes the machine's last free page.
  • systemd-oomd reacts to memory pressure and swap use over seconds, and a job that fills memory in one allocation burst can reach the kernel's killer first.
  • A swapfile only delays the kill and slows the machine while it lasts.

References: the kernel's Control Group v2 memory interface, systemd's resource control and scope units (OOMPolicy=).

Benchmarks

uv run --no-project python bench/run_bench.py --config bench/config.json --reps 3
uv run --no-project python scripts/bench_table.py --write

bench/config.json runs every CPU runtime in one pass and names the model files and the llama.cpp and diffuse-cpp checkouts. The committed CPU benchmark tables came from bench/config-main.json, bench/config-diffuse.json and bench/config-dream.json; the thread sweep came from bench/config-fair-cpu.json. scripts/bench_table.py refuses a runtime that appears in two result files, so move the files a new run replaces into bench/results/archive/ before --write. The harness waits for an idle CPU before each run, interleaves the runtimes, switches the power profile to performance for the duration, and writes a result file with every raw run. On a laptop it measures on mains power only: unplugged, it pauses before the next run and prints that it is waiting for the charger, and a run the charger left midway is repeated. The eval, the quality table and scripts/gpu_profile.py do the same through bench/power.py; --allow-battery measures on battery instead and records it. The benchmark harness kills any process named pytest while it runs, because test suites from other work on the development machine kept starting mid-benchmark. Set "kill_pytest": false in a config to turn that off; the GPU configs (bench/config-gpu.json, bench/config-dream-gpu.json) do, since a GPU run leaves the CPU nearly idle. They compare --device gpu with llama.cpp's diffusion example built with CUDA in build-cuda/:

cmake -B build-cuda -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build-cuda --target llama-diffusion-cli

GPU kernels

scripts/gpu_profile.py traces one run --device gpu with Nsight Systems (nsys --trace=cuda, which ran here without root although the driver keeps hardware counters for admins) and writes the device time per forward pass of each kernel group to bench/profiles/. It does not synchronize the stream, unlike MOJO_DLLM_GPU_PROFILE=1. --root points it at another checkout's build for before/after comparisons, and it reports a median-based total next to the plain sum, because a laptop GPU's clocks move under sustained load. --runtime llama.cpp --seq-len N traces llama.cpp's CUDA llama-diffusion-cli on the same prompt instead (it replays its forward pass as a CUDA graph, so the trace expands graph nodes), with the same kernel groups. bench/bench_gpu_gemm.mojo times each LLaDA weight shape alone, adding up the per-forward total from the tensor types in the GGUF.

On the development laptop the GPU is capped at 30 W and its clocks drop under sustained load (bench/checks/2026-10-12-gpu-power-under-load.txt), so compare kernels with interleaved runs (A, B, A, B) in one session rather than against a number from another day.

Comparing GPU dispatch changes

bench/compare_gpu.py compares two already-built checkouts through run_bench.run_mojo, interleaving A/B and B/A rounds on one assigned GPU. It excludes a warmup pair per generation length, records every raw run, and rejects failures or mismatched forward counts and canvas sizes. It leaves the system power profile and other processes alone. Keep each pair on the same GPU; another GPU can run a different model's independent comparison.

Both tools, and scripts/compare_logits.py, refuse a base and a new that are the same checkout, the same clean commit or byte-identical binaries, and record the SHA-256 of each build/mojo-dllm in the evidence, because a stale build carries the label of whatever commit is checked out. compare_gpu.py sets CUDA_DEVICE_ORDER=PCI_BUS_ID so that --gpu (an index or a GPU- uuid) names the same card to CUDA and to nvidia-smi, records the index and the uuid read back from nvidia-smi, and runs the isolation preflight and a window around each run of bench/isolation.py (--allow-noisy continues after a failed preflight). check_gpu_dispatch.py records CUDA_VISIBLE_DEVICES and CUDA_DEVICE_ORDER as it found them, so export CUDA_DEVICE_ORDER=PCI_BUS_ID there too.

python3 bench/compare_gpu.py --base ../base --new . --model MODEL.gguf \
  --gpu GPU-UUID --lengths 128,256,512 --reps 3 --prompts 3 \
  --out bench/results/dispatch/run.json
CUDA_VISIBLE_DEVICES=GPU-UUID python3 bench/check_gpu_dispatch.py \
  --base ../base --new . --model MODEL.gguf --arch llada \
  --out bench/checks/dispatch-exact.json

Use --arch dream for Dream. The exact check uses that model's mask token, compares all finite logit values byte for byte at several canvas sizes, and compares complete generated token lists. It checks preservation of the base GPU computation, not correctness against an independent runtime. The timing comparison runs without the synchronizing MOJO_DLLM_GPU_PROFILE mode.

Quality

bench/eval_gsm8k.py answers a fixed 200-problem GSM8K subset (bench/eval/gsm8k-200.jsonl, see bench/eval/SOURCE.md) and compares two arms problem by problem: accuracy difference with a 95% interval and McNemar's exact test. Any change that is not exact (a sampler that commits more per step, a K/V cache) needs a comparison against the one-token-per-step baseline before it becomes a default.

uv run --no-project python bench/eval_gsm8k.py run --model M.gguf --arm baseline
uv run --no-project python bench/eval_gsm8k.py run --model M.gguf --arm candidate -- --some-flag
uv run --no-project python bench/eval_gsm8k.py compare baseline candidate
uv run --no-project python bench/eval_gsm8k.py compare baseline candidate --limit 50

compare writes bench/eval/<date>-<a>-vs-<b>.json, and scripts/bench_table.py turns those files into the accuracy table in Benchmarks. Time per problem depends on what else the machine runs, so --limit N compares each arm's first N problems in run order and writes a -firstN file; the table takes its times from that file. Run that round with nothing else on the machine.

Toolchain

Mojo is pinned to 1.1.0 and max-core to 26.6.0 in pyproject.toml, locked in uv.lock; both come from PyPI. Bump them together by editing the two pins, running uv lock, and committing that on its own with a green run: Mojo 1.1 removed fn, alias and @parameter closures, and each release has moved APIs this code uses. max-core supplies max.algorithm.parallelize and the GPU API (max.gpu, max.gpu.host). The Mojo wheels need glibc 2.34 or newer.

The Python scripts run through uv too. uv run --no-project python script.py runs a script that uses only the standard library. The fixture generators and bench/parity.py need numpy and gguf, pinned in the fixtures group: uv run --only-group fixtures python scripts/gen_fixtures.py.

Releasing

A release is a version bump, a dated CHANGELOG section, a tag and a GitHub release. Pushing the tag and creating the release need write access to the repository. X.Y.Z stands for the new version.

  1. Check that CI is green on the commit of main that you ship.

  2. In CHANGELOG.md, rename ## [Unreleased] to ## [X.Y.Z] - YYYY-MM-DD and add an empty ## [Unreleased] above it.

  3. Set the version in the three places that carry it: version under [project] in pyproject.toml, comptime VERSION in src/main.mojo, and uv.lock. Run uv lock for the third; its diff is one line, the version of the mojo-dllm entry. Result files under bench/ record the version of the binary that produced them; leave them.

  4. Run scripts/ci-local.sh. On a machine with an NVIDIA GPU also run scripts/run-gpu-tests.sh, because CI builds the GPU suite and does not run it.

  5. Commit the files that steps 2 and 3 changed (CHANGELOG.md, pyproject.toml, src/main.mojo, uv.lock) with the subject Release X.Y.Z, together with any other file that still names the old version or milestone as current (for example docs/plan.md). Get the commit onto main.

  6. Tag that commit. The existing tag v0.1.0 is annotated and not signed:

    git tag -a vX.Y.Z -m "mojo-dllm X.Y.Z"
    

    For a signed tag use git tag -s vX.Y.Z -m "mojo-dllm X.Y.Z" and check it with git tag -v vX.Y.Z.

  7. Push the tag with git push origin vX.Y.Z. Check from the remote side that it arrived: git ls-remote origin 'refs/tags/vX.Y.Z^{}' prints the commit that git rev-parse 'vX.Y.Z^{commit}' prints.

  8. Create the GitHub release with the CHANGELOG section as its notes. Skip this step once the release workflow exists (#131): pushing the tag in step 7 then creates the release.

    notes="$(mktemp)"
    awk -v v="X.Y.Z" '/^## \[/ { p = (index($0, "## [" v "]") == 1); next } p' CHANGELOG.md > "$notes"
    test -s "$notes"
    gh release create vX.Y.Z --verify-tag --title "mojo-dllm X.Y.Z" --notes-file "$notes"
    

    test -s stops the sequence when the section is missing or empty. gh release view vX.Y.Z shows the result.

Porting a model

mojo-dllm runs the LLaDA and Dream architectures, and reads the metadata of diffusion-gemma (see below). Tested Q4_K_M checkpoints are LLaDA-8B-Instruct, LLaDA-1.5, Dream-v0-Instruct-7B, Dream-Coder-v0-Instruct-7B and DiffuCoder-7B-cpGRPO. The added checkpoints have CPU smoke and parity evidence; GPU evidence remains limited to LLaDA-8B and Dream-v0. This checklist follows the path Dream took.

1. Read the metadata

Run mojo-dllm inspect model.gguf and note general.architecture, the <arch>.* hyperparameters, tokenizer.ggml.model and tokenizer.ggml.pre, the mask token id, and every tensor type. Dream differs from LLaDA in four places that matter:

LLaDA-8BDream-7B
attention32 heads, MHA28 query heads, 4 KV heads (GQA)
QKV biasnoneyes (Qwen2 lineage)
logitsposition i predicts token ishifted: position i predicts token i+1
samplerlow-confidence blockstimestep schedule, entropy / margin variants

Metadata contract

<arch> is llada or dream. Required keys fail with a message naming the missing key. This table describes ModelConfig; the tokenizer also needs its own metadata.

KeyRequired/defaultInterpretation
general.architecturerequiredllada or dream; errors list the supported set
general.nameoptional, ?display name
<arch>.block_countrequiredtransformer layer count
<arch>.embedding_lengthrequiredhidden width
<arch>.attention.head_countrequiredquery head count
<arch>.attention.head_count_kvoptional, head countKV head count; must divide query head count
<arch>.feed_forward_lengthrequiredfeed-forward width
<arch>.context_lengthoptional, 4096the longest canvas (prompt plus --max-tokens) run accepts
<arch>.attention.layer_norm_rms_epsilonrequiredRMSNorm epsilon
<arch>.rope.freq_baseoptional, 10000RoPE frequency base
<arch>.vocab_sizeoptional, rows of output.weightvocabulary size
tokenizer.ggml.mask_token_idrequiredno fallback; tokenizer reads the same key
diffusion.shift_logitsoptional, false for LLaDA / true for Dreamprediction-row shift
<arch>.attention.causalignoredattention stays bidirectional, as in llama.cpp for these architectures
<arch>.attention.key_lengthoptional, head dimensionother values are rejected
<arch>.rope.dimension_countoptional, head dimensionRoPE rotates the full head; other widths are rejected
<arch>.rope.scaling.typeoptional, noneonly none or linear with factor 1 are accepted
<arch>.rope.scaling.factoroptional, 1.0other factors are rejected; no scaled RoPE is implemented

The head dimension is embedding_length / head_count, with exact divisibility required. block_count, embedding_length, head_count, head_count_kv, feed_forward_length and vocab_size must be positive, and the epsilon and the RoPE base must be finite and above zero; a value outside those ranges stops the load with <key> must be positive, got N or <key> must be finite and above zero, got X. Other scaling metadata is not interpreted, and context_length is not range checked. run refuses a canvas longer than it; whether to reject attention.causal = true remains a separate policy decision.

DiffusionGemma metadata

A diffusion-gemma file is read into a ModelConfig, and mojo-dllm inspect prints the result, or the refusal and its key. DiffusionLM and the GPU model stop on it with diffusion-gemma does not run on this model. DiffusionGemma runs it on the CPU through logits, and tokenize prints the ids of its chat prompt; run still waits for the model to be wired into the generation loop (#277; the forward pass is #64 and the denoiser #65), so a clean inspect does not mean run works. The names come from the unmerged ggml-org/llama.cpp#24423 and appear once each, as the GEMMA_* constants at the top of models/transformer.mojo. Keys carry the diffusion-gemma. prefix except diffusion.canvas_length and tokenizer.ggml.mask_token_id.

KeyRequired/defaultInterpretation
block_count, attention.head_count, attention.key_length, attention.key_length_swa, attention.sliding_window, expert_count, expert_feed_forward_length, diffusion.canvas_lengthrequired, positivelayers, query heads, head size of global and of sliding layers, window, experts, expert width, canvas tokens
embedding_length, feed_forward_lengthrequiredhidden width, dense feed-forward width
attention.head_count_kvrequired, i32 array with one entry per blockK/V heads of each layer; one value per kind of layer, a divisor of attention.head_count
attention.sliding_window_patternrequired, bool array with one entry per blocktrue marks a sliding layer, false a global one
expert_used_countrequired, from 1 to expert_countexperts per token
rope.freq_base, rope.freq_base_swarequiredRoPE base of global and of sliding layers
attention.layer_norm_rms_epsilon, tokenizer.ggml.mask_token_idrequiredRMSNorm epsilon, mask token
attention.value_length, attention.value_length_swa, rope.dimension_count, rope.dimension_count_swarequired, equal to the key length of their kindother widths are refused
context_lengthoptional, 4096stored in config; run does not take this architecture yet
final_logit_softcappingoptional, 30 when absent (llama.cpp's default); 0 turns it offa negative value is refused
attention.shared_kv_layers, embedding_length_per_layer_inputoptional, 0any other value is refused: no forward pass handles it
attention.causaloptional, falsetrue is refused; the canvas attends in both directions

The config keeps the global layers' values in n_kv_heads, head_dim and theta, and the sliding layers' in the _swa fields. kv_heads(i), head_size(i) and rope_base(i) pick the right one for layer i. The vocabulary is the row count of token_embd.weight, because the file has no output.weight (the logits are tied to the embedding).

The tensor directory has to agree with the flags and keys, and a mismatch names the tensor and the keys that set its shape:

  • blk.N.attn_v.weight exists on a sliding layer and on no global one (a global layer reuses K as V).
  • blk.N.attn_k.weight is [embedding_length, head_count_kv * key_length] of that layer, with key_length_swa on a sliding layer.
  • blk.N.ffn_gate_up_exps.weight is [embedding_length, 2 * expert_feed_forward_length, expert_count] and blk.N.ffn_down_exps.weight is [expert_feed_forward_length, embedding_length, expert_count].
  • rope_freqs.weight has key_length / 2 entries when any layer is global, and token_embd.weight is embedding_length wide.

For LLaDA and Dream the same kind of check covers the two vocabulary-sized tensors: token_embd.weight and, when the file has it, output.weight must be [embedding_length, vocabulary], where the vocabulary is <arch>.vocab_size or the rows of output.weight. ModelConfig raises tensor shape mismatch for <name>: [..], expected [..] before any loader runs, on the CPU and on the GPU.

To convert a Hugging Face source with llama.cpp and quantize the result:

python convert_hf_to_gguf.py <hf-dir> --outtype f16 --outfile model-f16.gguf
llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

The converter accepts f32, f16, bf16, q8_0, tq1_0, tq2_0, and auto as output types. Conversion success alone does not establish mojo-dllm support; inspect the metadata and tensor types before running the model.

2. Load the weights

Extend ModelConfig and Layer in models/transformer.mojo (the architecture check at the top of ModelConfig.__init__ lists the supported names). Reuse PackedMatrix for every projection. A tensor type it does not support stops the load with unsupported GGUF tensor type: <type> (tensor <name>) when no decoder exists. A decodable type without a matrix kernel reports tensor <name> is <type>: matrix kernels exist for Q4_K and Q6_K. Reference decoders accept F32, F16, BF16, Q5_0, Q8_0, Q4_K, Q5_K and Q6_K for embedding rows and vectors; matrix kernels still require Q4_K or Q6_K. Add the decoder and packed kernel with fixtures from scripts/gen_fixtures.py before advertising a new matrix type as supported end to end.

Before loading a large conversion, compare its tensor shapes with the source:

uv run --only-group fixtures python scripts/check_weights.py \
  model-Q4_K_M.gguf <hf-dir>/model.safetensors.index.json
# For an unsharded source, pass model.safetensors instead of the index.

The checker reads safetensors headers only and reverses GGUF dimension order before comparing. It reports mismatches, missing tensors on either side, and unmapped source names; dtypes are shown for reference, not compared. Use --ignore 'rotary.*inv_freq' for source buffers intentionally omitted during conversion; ignored N counts these source entries, and their mapped GGUF counterparts are also omitted. Index shard names must be bare .safetensors filenames resolving inside the index directory. The checker accepts 1 to 1024 blocks to bound name-map construction. Malformed inputs return 2, including JSON nesting errors. An output tensor absent from a tied source is still reported as missing, with an explanation. Exit 0 means at least one tensor was compared and all comparisons passed, 1 means findings, and 2 means no comparison or invalid input. Shape agreement does not establish matching weights or logits.

3. Prove the forward pass

  1. Extend scripts/gen_fixtures.py with a tiny model of the new layout and its numpy reference logits. Write the test first and watch it fail.
  2. On the real model, compare logits with llama.cpp through tools/ref/llada_ref.cpp (logits mode) and bench/parity.py.

4. Tokenizer

If tokenizer.ggml.pre is new, add its pre-tokenizer to tokenizer/bpe.mojo and golden cases to scripts/gen_tokenizer_fixture.py. llama.cpp's regex for each pre-tokenizer is in src/llama-vocab.cpp. A model whose tokenizer.ggml.model is not gpt2 needs its own module, as gemma4 has in tokenizer/gemma4.mojo; Tokenizer picks it from that key.

5. Sampler

Add the model's reference sampler next to LLaDA's in diffusion/sampler.mojo, with unit tests for its schedule.

Development plan

Written 2026-10-03 from the draft spec (docs/spec.md) and three research passes: the Mojo 1.1 toolchain, the diffusion-LLM landscape, and the house style of the author's other repositories. Where research contradicted the spec, the correction is recorded here and the code follows the correction. The plan predates the code: where the code later moved, the text below carries the as-built wording, and Architecture maps what shipped.

Corrections to the spec

Spec saysReality (verified)Consequence
Q4_K_M support, optionally Q8_0/F16A Q4_K_M GGUF of LLaDA-8B holds Q4_K (incl. token_embd), Q6_K (attn_v/ffn_down in half the layers, output) and F32 normsv0.1 needs F32 + Q4_K + Q6_K
mojoproject.tomlmagic is gone; Mojo installs from PyPIpin mojo==1.1.0, max-core==26.6.0 in pyproject.toml, commit uv.lock
mojo testremoved; TestSuite, each test file built with mojo build and run as a binary; an empty suite exits 0CI pins the test count
llama.cpp as generation referenceits diffusion sampler skips CFG/alg_temp, noises one row, inverts entropy, samples at temp 0llama.cpp is the logit reference; token parity uses the LLaDA generate.py algorithm re-implemented over llama.cpp logits
diffuse-cpp as drop-in comparisonown diffuse GGUF schema, no tokenizer, unverified paritybenchmarked with its own file, on token IDs
RoPEGGUF converter un-permutes Q/K: GGUF uses adjacent-pair (NORM) RoPENORM RoPE when reading GGUF
UnsafePointer, fn, parallelize in std1.1 removed fn/alias; parallelize moved to max.algorithmwrite 1.1 Mojo only

Rows 3, 6 and 7 record assumptions from the research passes. docs/spec.md uses fn, and names none of mojo test, UnsafePointer, parallelize or a RoPE variant.

Model facts used (LLaDA-8B-Instruct, GGUF llada)

32 layers, d = 4096, 32 heads × 128 (MHA), FFN 12288 SwiGLU, RMSNorm ε = 1e-5, RoPE θ = 500 000, vocab 126 464, mask 126 336, BOS 126 080, EOS 126 081, EOT 126 348, non-causal, untied output. Tokenizer: byte-level BPE (gpt2), pre-tokenizer bailingmoe.

Compute design

By default a denoising step runs the full transformer over the whole canvas (prompt + generation, L ≈ 100 to 160 tokens), so linear layers are GEMMs, not GEMVs. Cost per step ≈ 2 × 7.0·10⁹ × L FLOPs, plus 2 × 0.5·10⁹ per row whose logits are computed (LLaDA-8B): compute-bound, not bandwidth-bound.

  • Activations are quantized per token to int8 with one f32 scale per 256 values (the Q8_K scheme), plus the sum of each group of 16 values as a float.
  • At load, each Q4_K and Q6_K matrix is repacked so that a 256-bit register holds 4 consecutive weights of each of 8 rows. The two types have their own layouts and tile functions; The quantized GEMM has both.
  • The inner product uses vpdpbusd (AVX-VNNI) when the CPU has it, and pmaddubsw plus pmaddwd otherwise. One vpdpbusd multiplies 8 rows by 4 activation bytes, and no horizontal sum is needed until the row ends.
  • The LM head runs only on the masked positions the sampler can commit in the step: the current block for LLaDA (at most --block-length rows, 32 by default) and every masked position of the canvas for Dream (at most --max-tokens rows), not all L rows. This is a diffusion-specific saving that a generic runtime does not take.
  • Threads split output rows into many small chunks so the 4 P-cores and 4 E-cores of a hybrid CPU balance.

Correctness ladder

  1. Unit: GGUF parse (synthetic files generated by scripts/gen_fixtures.py and committed under tests/fixtures/), Q4_K/Q6_K decode against hand-built blocks, quantized dot vs f32 dot, RMSNorm, RoPE, softmax, attention, scheduler transfer counts against generate.py.
  2. Tokenizer: golden token IDs produced by llama-tokenize on a fixed corpus.
  3. Logits: a small C++ tool over libllama dumps logits for fixed token sequences; mojo-dllm is compared on cosine similarity, relative RMS error and argmax agreement. Correctness has the measured values.
  4. Generation: the same tool runs LLaDA's generate.py greedy algorithm over llama.cpp logits; the tokens are compared one by one, and a run that diverges reports the first position that differs.

Benchmark protocol

Same machine, same prompts, same thread count, release builds, model load excluded from generation time and reported on its own. Configuration: gen 128, block 32, steps 32 (4 tokens per step, 32 forward passes), plus per-step latency. Three runs each, median reported, raw numbers in bench/results/*.json. The README table is rendered from that JSON and a gate fails when they disagree. No estimated numbers are published.

Repository gates

Sized to what this repo can break, copied from sysknife's reasoning:

GateGuards against
staged secret + employer-name scan (pre-commit)a key or the employer name in history
mojo format then git diff --exit-codeunformatted code (there is no check mode)
build with -Werrordeprecated APIs piling up across Mojo releases
tests with a pinned counta suite that selects 0 tests and still exits 0
shellcheck, yamllint, markdownlintbroken scripts and workflows
action pins verified against tagsa floating @v4 or a mislabelled SHA
README numbers equal bench/resultshand-typed benchmark claims
mdBook build + internal link checkbroken docs

Development lists the full set, including the launcher tests, the GPU suite, the no-GPU check and two self-tests.

Local first: scripts/ci-local.sh runs everything; pre-commit runs the fast set, pre-push the full set. GitHub Actions runs the same gates on push and PR, except that it builds the GPU suite without running it, split into three jobs (.github/workflows/ci.yml). .github/workflows/docs.yml deploys the book to GitHub Pages on pushes to main that touch docs/, book.toml, theme/ or assets/.

Milestones

Deliverable
M0repo, CI, logo, docs skeleton
M1mojo-dllm inspect model.gguf
M2Q4_K/Q6_K decode, quantized GEMM, tests, microbenchmark
M3RMSNorm, RoPE, attention, SwiGLU
M4LLaDA forward, logit parity with llama.cpp
M5tokenizer, scheduler, mojo-dllm run
M6threading and SIMD tuning, benchmark report vs llama.cpp and diffuse-cpp
M7NVIDIA backend (--device gpu), shipped in 0.2.0
M8Dream-7B, shipped in 0.1.0
laterblock diffusion; inter-step caching beyond --dual-cache

M0 to M6 and M8 shipped in 0.1.0. M7 shipped in 0.2.0.

The repository went public after M6, with parity documented and the license of shipped binaries reviewed (Modular runtime libraries are not Apache; NOTICE records the terms).

mojo-dllm: Technical Specification

Status: Original specification, draft v0.1, written on 2026-10-03 before the code. It is not kept up to date. The development plan lists the corrections that research made, Architecture maps the source as built, and Command line lists the current flags.
Project type: Open-source ML inference systems project
Primary language: Mojo
Initial model: LLaDA-8B, quantized GGUF
Initial target: x86 CPU
Secondary target: NVIDIA consumer GPU
Long-term targets: Apple Silicon, AMD GPU
License target: Apache-2.0 or MIT-compatible, subject to dependency review


1. Summary

mojo-dllm is a lightweight, Mojo-native runtime for local inference of diffusion language models (dLLMs).

The project will begin with LLaDA-8B in quantized GGUF form and implement the complete inference path in Mojo, including:

  • GGUF model loading;
  • tokenization;
  • quantized matrix multiplication;
  • transformer execution;
  • bidirectional attention;
  • diffusion masking and remasking;
  • confidence/entropy-based token selection;
  • iterative denoising;
  • deterministic sampling;
  • CLI inference.

The first milestone is deliberately CPU-only and correctness-focused.

The primary long-term technical thesis is:

A single Mojo inference implementation can execute diffusion language models efficiently across heterogeneous consumer hardware, while sharing substantially more kernel/runtime code than traditional C++ + CUDA + Metal + ROCm stacks.

The project is not intended to introduce a new diffusion-model algorithm initially. Its novelty is primarily systems-oriented: portable, quantized, local dLLM inference in Mojo.


2. Motivation

Most current high-performance language-model runtimes are optimized around autoregressive decoding.

Diffusion language models use a fundamentally different inference loop:

  1. initialize part or all of the generation region with mask tokens;
  2. execute a bidirectional transformer pass;
  3. estimate token distributions for unresolved positions;
  4. select some positions to commit;
  5. optionally remask low-confidence positions;
  6. repeat until the generation region is resolved.

This creates runtime requirements that differ from ordinary autoregressive inference:

  • repeated whole-sequence or block-level transformer evaluation;
  • bidirectional attention rather than a purely causal mask;
  • diffusion-specific scheduling state;
  • mask/remask operations;
  • per-position confidence or entropy calculations;
  • opportunities for inter-step state reuse;
  • potentially different memory/computation tradeoffs.

Existing implementations primarily use:

  • PyTorch;
  • C/C++;
  • CUDA;
  • Triton;
  • specialized serving frameworks.

Mojo is a strong candidate for this workload because it aims to support portable low-level CPU and accelerator programming from a common language and type system.


3. Project Goals

3.1 v0.1 goals

The first release MUST:

  • run LLaDA-8B Q4_K_M GGUF locally;
  • require no Python at inference time;
  • execute entirely on x86 CPU;
  • parse GGUF directly in Mojo;
  • tokenize prompts locally;
  • perform iterative diffusion generation;
  • expose a simple CLI;
  • support deterministic generation with a fixed seed;
  • produce numerically reasonable parity with a reference runtime;
  • provide reproducible benchmark tooling.

Target invocation:

mojo-dllm run \
  --model ./LLaDA-8B-Q4_K_M.gguf \
  --prompt "Explain speculative decoding." \
  --max-tokens 128 \
  --steps 128 \
  --seed 42

3.2 v0.2 goals

  • optimized CPU SIMD;
  • thread-parallel transformer execution;
  • faster Q4_K matrix multiplication;
  • fused normalization;
  • improved sampling kernels;
  • benchmark parity against diffuse-cpp and llama.cpp.

3.3 v0.3 goals

  • NVIDIA GPU backend;
  • RTX 30-series support as an initial consumer target;
  • device-side quantized GEMM;
  • device-side attention;
  • device-side diffusion sampling operations;
  • optional CPU/GPU split execution.

3.4 v0.4+ goals

  • Dream model support;
  • Apple Silicon GPU backend;
  • AMD GPU backend;
  • inter-step cache experiments;
  • block diffusion;
  • runtime integration layer for projects such as molla;
  • API/library mode in addition to CLI mode.

4. Non-Goals

The initial project MUST NOT attempt to implement:

  • an OpenAI-compatible HTTP server;
  • multi-user continuous batching;
  • distributed inference;
  • tensor parallelism;
  • speculative decoding;
  • training;
  • fine-tuning;
  • arbitrary Hugging Face model loading;
  • every GGUF quantization format;
  • MAX serving integration;
  • custom model download infrastructure;
  • a general-purpose LLM runtime;
  • novel dLLM research algorithms before baseline correctness.

This project should remain a small inference core, not become another general serving framework.


5. Initial Model

5.1 Model

Initial target:

LLaDA-8B

Preferred representation:

GGUF Q4_K_M

Reasons:

  • laptop-friendly model size;
  • existing public quantized artifacts;
  • simple enough transformer structure;
  • multiple independent reference implementations;
  • existing C/C++ implementations provide useful correctness oracles;
  • bidirectional transformer execution makes it a representative dLLM baseline.

5.2 Second model

The second supported model SHOULD be:

Dream-7B

Adding Dream forces the runtime to generalize beyond a single LLaDA-specific model layout and helps validate the abstraction around:

  • GQA;
  • model metadata;
  • tokenizer configuration;
  • diffusion scheduler differences;
  • transformer architecture differences.

6. High-Level Architecture

                    +----------------------+
                    |      mojo-dllm       |
                    +----------+-----------+
                               |
             +-----------------+-----------------+
             |                 |                 |
             v                 v                 v
        GGUF Loader        Tokenizer      Diffusion Runtime
             |                                   |
             |                          +--------+--------+
             |                          |                 |
             v                          v                 v
       Tensor Store              Diffusion State      Scheduler
             |                          |                 |
             +-------------+------------+-----------------+
                           |
                           v
                    Model Execution
                           |
             +-------------+-------------+
             |                           |
             v                           v
       Quantized Linear           Bidirectional Attention
             |                           |
             +-------------+-------------+
                           |
                           v
                     Device Backend
                  CPU / NVIDIA / ...

7. Repository Layout

Suggested initial structure:

This is the layout the draft proposed. The code differs: modules live under src/mojo_dllm/, tests are in tests/, benchmarks in bench/, and pyproject.toml pins the Mojo toolchain. Architecture maps the files as built.

mojo-dllm/
├── README.md
├── LICENSE
├── pyproject.toml              # only if needed for tooling; not runtime
├── mojoproject.toml
├── src/
│   ├── main.mojo
│   ├── cli/
│   │   └── run.mojo
│   ├── formats/
│   │   ├── gguf.mojo
│   │   └── tensor_store.mojo
│   ├── tokenizer/
│   │   ├── tokenizer.mojo
│   │   └── bpe.mojo
│   ├── quant/
│   │   ├── q4_k.mojo
│   │   ├── q8_0.mojo
│   │   └── dequant.mojo
│   ├── kernels/
│   │   ├── matmul.mojo
│   │   ├── rmsnorm.mojo
│   │   ├── rope.mojo
│   │   ├── attention.mojo
│   │   ├── softmax.mojo
│   │   └── sampling.mojo
│   ├── models/
│   │   ├── model.mojo
│   │   └── llada.mojo
│   ├── diffusion/
│   │   ├── state.mojo
│   │   ├── scheduler.mojo
│   │   ├── masking.mojo
│   │   ├── confidence.mojo
│   │   └── sampler.mojo
│   └── runtime/
│       ├── device.mojo
│       ├── cpu.mojo
│       └── executor.mojo
├── tests/
│   ├── test_gguf.mojo
│   ├── test_quant.mojo
│   ├── test_attention.mojo
│   ├── test_scheduler.mojo
│   └── test_llada.mojo
├── benchmarks/
│   ├── bench_matmul.mojo
│   ├── bench_attention.mojo
│   ├── bench_generation.mojo
│   └── scripts/
└── docs/
    ├── architecture.md
    ├── model-porting.md
    └── benchmarks.md

8. Core Data Structures

The structs, traits and signatures in sections 8, 10.2, 13, 17.1 and 24 are sketches written before the code. The code has ModelConfig and the DenoisingModel trait in src/mojo_dllm/models/transformer.mojo, TensorInfo in src/mojo_dllm/formats/gguf.mojo, and GenConfig with generate in src/mojo_dllm/diffusion/sampler.mojo. It has no DiffusionState, DiffusionScheduler, Device or DiffusionModel, and it uses def where the sketches use fn.

8.1 Model configuration

struct ModelConfig:
    var vocab_size: Int
    var hidden_size: Int
    var intermediate_size: Int
    var num_layers: Int
    var num_attention_heads: Int
    var num_kv_heads: Int
    var head_dim: Int
    var max_position_embeddings: Int
    var rope_theta: Float64
    var mask_token_id: Int

8.2 Diffusion state

struct DiffusionState:
    var tokens: Tensor[Int32]
    var masked: BitSet
    var committed: BitSet
    var confidence: Tensor[Float32]
    var step: Int
    var total_steps: Int

Future versions MAY add:

var dirty_positions: BitSet
var block_state: BlockState
var cache_state: DiffusionCache

but cache-related structures SHOULD NOT be required for v0.1.

8.3 Tensor descriptor

struct TensorInfo:
    var name: String
    var shape: List[Int]
    var ggml_type: Int
    var offset: UInt64
    var byte_length: UInt64

9. GGUF Support

9.1 Required v0.1 support

The GGUF reader MUST support:

  • GGUF header validation;
  • metadata parsing;
  • tensor metadata;
  • tensor offsets;
  • memory-mapped or direct file-backed reads;
  • little-endian decoding;
  • required tokenizer metadata;
  • LLaDA tensor-name mapping.

Initial quantization support:

  • Q4_K_M;
  • optionally Q8_0;
  • optionally F16 for correctness tests.

9.2 Unsupported formats

Unknown tensor types MUST fail explicitly:

unsupported GGUF tensor type: IQ3_XS (tensor blk.0.attn_q.weight)

No silent fallback.

A type with a reference decoder but without a matrix kernel currently fails at matrix load, for example:

tensor blk.0.attn_q.weight is Q8_0: matrix kernels exist for Q4_K and Q6_K

9.3 Loading strategy

Prefer memory mapping where practical:

GGUF file
   |
   v
memory map
   |
   +--> metadata
   |
   +--> tensor descriptors
   |
   +--> quantized tensor views

Avoid eagerly dequantizing the entire model into FP32.


10. Quantized Compute

10.1 Initial requirement

Implement Q4_K_M matrix multiplication directly against quantized weights.

Conceptually:

Q4_K_M weights
      |
      v
block decode
      |
      v
vectorized dot product
      |
      v
FP32/BF16 accumulation

The baseline MAY initially use a simple reference implementation.

The optimized implementation SHOULD:

  • exploit SIMD;
  • operate blockwise;
  • avoid full-tensor dequantization;
  • reuse scales efficiently;
  • minimize temporary allocations.

10.2 Kernel API

Example:

fn linear_q4k[
    T: DType
](
    x: TensorView[T],
    weight: QuantizedTensorView,
    out: TensorView[T],
):
    ...

10.3 Benchmark requirements

Q4_K_M GEMM MUST be benchmarked independently.

Measurements:

  • matrix dimensions;
  • latency;
  • effective memory bandwidth;
  • effective FLOP/s where meaningful;
  • allocations;
  • thread count.

11. Transformer Execution

A LLaDA transformer layer will initially include:

input
  |
RMSNorm
  |
QKV projection
  |
RoPE
  |
bidirectional attention
  |
output projection
  |
residual
  |
RMSNorm
  |
MLP
  |
residual

11.1 Critical difference from autoregressive execution

The attention mask is not purely causal.

The runtime must support full/bidirectional attention over the relevant diffusion region.

Therefore, the initial implementation SHOULD NOT assume an append-only KV cache.


12. Diffusion Inference Loop

The core execution loop should be explicit and readable.

Pseudo-code:

tokens = tokenize(prompt)
canvas = append_masks(tokens, max_new_tokens)

for step in 0 .. num_steps:
    logits = model.forward(canvas)

    unresolved = get_masked_positions(canvas)

    scores = confidence(logits, unresolved)

    selected = scheduler.select(
        unresolved,
        scores,
        step,
        num_steps
    )

    sampled = sample(logits[selected])

    canvas[selected] = sampled

    if scheduler.requires_remasking:
        remask_low_confidence(canvas, scores)

return decode(canvas)

The scheduler MUST be modular.


13. Scheduler Interface

trait DiffusionScheduler:
    fn initialize(
        tokens: Tensor[Int32],
        generated_length: Int
    ) -> DiffusionState

    fn select_positions(
        state: DiffusionState,
        logits: TensorView[Float32]
    ) -> Selection

    fn update(
        state: DiffusionState,
        selection: Selection,
        sampled_tokens: TensorView[Int32]
    )

Initial scheduler:

  • reference implementation matching LLaDA inference behavior.

Future schedulers:

  • confidence-based;
  • entropy-based;
  • fixed block count;
  • adaptive steps;
  • block diffusion.

14. Sampling

Required primitives:

  • softmax;
  • log-softmax where needed;
  • entropy;
  • argmax;
  • top-k;
  • categorical sampling;
  • temperature;
  • deterministic PRNG;
  • masked-position gather/scatter.

Possible future fused kernel:

logits
  |
softmax
  |
entropy/confidence
  |
top-k
  |
sample
  |
scatter

15. Tokenizer

v0.1 MUST perform tokenization without invoking Python.

Possible implementation paths:

  1. reuse or adapt an existing Mojo BPE tokenizer;
  2. implement the minimum tokenizer behavior required by the target model;
  3. parse tokenizer metadata from GGUF where available.

The tokenizer MUST correctly support:

  • prompt encoding;
  • mask token;
  • BOS/EOS if required;
  • decoding generated IDs;
  • byte fallback if required by the model tokenizer.

16. CPU Runtime

The initial backend is x86 CPU.

Priority order:

Stage 1: correctness

  • scalar/reference kernels;
  • single thread;
  • deterministic output.

Stage 2: vectorization

  • SIMD vector operations;
  • vectorized dequantization;
  • vectorized dot products;
  • optimized softmax.

Stage 3: threading

Potential parallel regions:

  • output channels in linear layers;
  • attention heads;
  • MLP projections;
  • batch-independent work.

Threading MUST be benchmark-driven.


17. NVIDIA Backend

The NVIDIA backend begins only after the CPU runtime passes correctness tests.

Initial GPU target:

  • consumer NVIDIA RTX 30-series;
  • FP16/BF16 where supported;
  • quantized model weights;
  • device-resident hot tensors.

Required kernels:

  • Q4_K_M GEMM;
  • RMSNorm;
  • RoPE;
  • attention;
  • softmax;
  • diffusion sampling operations.

17.1 Device abstraction

The runtime SHOULD expose:

trait Device:
    fn allocate(...)
    fn copy_to_device(...)
    fn copy_from_device(...)
    fn synchronize(...)

Kernel implementation should avoid unnecessary backend-specific code where Mojo abstractions are sufficient.


18. CPU/GPU Hybrid Execution

Future laptop-focused mode:

system RAM
   |
model weights
   |
+--+--------------------+
|                       |
CPU                  GPU VRAM
|                       |
fallback layers      hot layers
|                       |
+-----------+-----------+
            |
       diffusion state

Potential strategies:

  • layer offload;
  • operator offload;
  • GPU-only attention;
  • GPU-only matrix multiplication;
  • dynamic VRAM budget.

This is NOT required for v0.1.


19. Correctness Strategy

The project MUST use at least two external reference implementations when available.

Recommended references:

  • official PyTorch LLaDA implementation;
  • llama.cpp diffusion implementation;
  • diffuse-cpp.

Validation levels:

Level 1: tensor-level

Compare:

  • embeddings;
  • norm outputs;
  • Q/K/V projections;
  • RoPE outputs;
  • attention outputs;
  • MLP outputs;
  • final logits.

Use small contexts and fixed inputs.

Level 2: diffusion-step parity

Compare:

  • mask positions;
  • confidence values;
  • selected positions;
  • sampled token IDs;
  • updated canvas.

Level 3: generation parity

With fixed:

  • model;
  • prompt;
  • seed;
  • temperature;
  • diffusion steps;
  • generation length;

compare final output and intermediate states.

Exact parity may not always be possible across floating-point implementations, so tolerances MUST be documented.


20. Testing

Minimum unit tests:

File format

  • valid GGUF header;
  • invalid magic;
  • metadata parsing;
  • tensor offset calculations;
  • Q4_K block parsing.

Quantization

  • known Q4_K block decode;
  • scalar vs SIMD result;
  • matrix-vector correctness.

Transformer

  • RMSNorm;
  • RoPE;
  • softmax;
  • attention;
  • MLP;
  • full layer output.

Diffusion

  • mask initialization;
  • confidence computation;
  • selection logic;
  • remasking;
  • deterministic RNG.

Integration

  • tiny synthetic model;
  • single transformer layer;
  • short prompt;
  • one diffusion step;
  • full LLaDA run.

21. Benchmark Plan

21.1 Comparison targets

Primary:

  • llama.cpp;
  • diffuse-cpp.

Secondary:

  • official PyTorch/Hugging Face implementation.

21.2 Metrics

Measure:

  • time to first completed denoising step;
  • time per denoising step;
  • total generation latency;
  • generated tokens per second;
  • denoising steps per second;
  • peak RSS;
  • peak VRAM;
  • model load time;
  • CPU utilization;
  • GPU utilization;
  • energy consumption where practical.

21.3 Benchmark matrix

Example:

RuntimeDeviceQuantPromptOutputSteps
llama.cppCPUQ4_K_M128128128
diffuse-cppCPUQ4_K_M128128128
mojo-dllmCPUQ4_K_M128128128
mojo-dllmNVIDIAQ4_K_M128128128

The published benchmarks use 128 generated tokens in 32 steps (4 tokens per step) on three short prompts, with --block-length 32 for LLaDA. See Benchmarks.


22. Performance Philosophy

Correctness comes before optimization.

Optimization order:

correct model execution
        |
        v
Q4_K compute
        |
        v
CPU SIMD
        |
        v
threading
        |
        v
fused kernels
        |
        v
GPU
        |
        v
inter-step caching
        |
        v
block diffusion

Do not introduce caching before a fully correct uncached baseline exists.


23. Future Caching Research

After the baseline runtime is stable, experiment with:

23.1 Inter-step cache

Track which intermediate state can be reused between adjacent denoising iterations.

23.2 Dirty-position tracking

struct DirtyState:
    var changed_positions: BitSet
    var changed_blocks: BitSet

23.3 Layer-level reuse

Explore whether selected intermediate layer outputs can be reused under configurable error thresholds.

23.4 Block diffusion

Support models that denoise blocks rather than the entire generation canvas.

All cache optimizations MUST quantify:

  • speedup;
  • additional memory use;
  • output divergence;
  • benchmark quality impact.

24. Integration with Other Mojo Runtimes

The core should eventually expose a small library API.

Example:

var model = DiffusionModel.load("model.gguf")

var config = GenerationConfig(
    max_new_tokens=128,
    steps=128,
    temperature=0.2,
    seed=42,
)

var output = model.generate(
    "Explain KV-cache quantization.",
    config
)

A runtime such as molla could then provide:

  • HTTP serving;
  • OpenAI-compatible APIs;
  • model lifecycle;
  • chat templates;
  • request scheduling.

mojo-dllm should remain focused on dLLM execution.


25. CLI

Initial interface:

mojo-dllm run [OPTIONS]

Required flags:

--model PATH
--prompt TEXT
--max-tokens N
--steps N
--seed N

Only --model and --prompt are required. --max-tokens defaults to 128, --steps to the value of --max-tokens, and --seed to 42. --device takes cpu or gpu. The built binary is build/mojo-dllm, and the ./mojo-dllm launcher takes the prompt as its first word. Command line lists every flag.

Optional:

--temperature FLOAT
--threads N
--device cpu
--verbose
--dump-step-stats

Example:

mojo-dllm run \
  --model ./llada-8b-q4_k_m.gguf \
  --prompt "What is paged attention?" \
  --max-tokens 128 \
  --steps 128 \
  --seed 1234 \
  --threads 8

26. Logging and Diagnostics

Verbose mode SHOULD print:

model: LLaDA-8B
quantization: Q4_K_M
device: CPU
threads: 8
prompt tokens: 17
generation tokens: 128
diffusion steps: 128

load time: 1.21 s
step 1: 83.2 ms
step 2: 81.7 ms
...
generation: 10.4 s
peak RSS: 6.1 GB

This sample shows the format only. run --verbose prints the model, device, token counts, steps and load time, then the generation time, the per-step time with its GEMM, attention and LM head split, tokens per second and peak RSS; --dump-step-stats adds one line per step. The ms / step column of Benchmarks has the measured step time on the test laptop.

Debug mode MAY expose:

  • selected token positions;
  • confidence distributions;
  • per-layer timings;
  • kernel timings.

27. Error Handling

The runtime should fail with actionable errors.

Examples:

ERROR: unsupported model architecture: diffusion_gemma (supported: llada, dream)
ERROR: unsupported GGUF tensor type: IQ2_XXS (tensor blk.0.attn_q.weight)
ERROR: required metadata key missing: tokenizer.ggml.model
ERROR: tensor shape mismatch for blk.0.attn_q.weight
ERROR: model requires 8.1 GB but available memory is 5.9 GB

The CPU path has no memory pre-check: a failed allocation prints out of memory: could not allocate N bytes. --device gpu prints the model needs N MiB of GPU memory and M MiB are free.

Avoid assertion-only failure paths in user-facing code.


28. Memory Strategy

For the initial CPU backend:

  • memory-map GGUF;
  • repack each projection matrix into a heap block at load and release its mapped pages (see Architecture), keeping the embedding table mapped;
  • allocate reusable activation buffers;
  • avoid layer-by-layer heap churn;
  • preallocate diffusion state;
  • reuse attention workspaces.

Ideal high-level layout:

mapped GGUF
    |
    +--> quantized weights

runtime arena
    |
    +--> hidden state
    +--> Q/K/V
    +--> attention workspace
    +--> MLP workspace
    +--> logits
    +--> diffusion scheduler state

29. Milestones

M0: repository bootstrap

Deliverables:

  • build system;
  • CLI skeleton;
  • CI;
  • basic tests;
  • coding conventions.

M1: GGUF inspection

Deliverables:

mojo-dllm inspect model.gguf

Outputs:

  • architecture;
  • tensor count;
  • tensor names;
  • shapes;
  • quantization formats;
  • tokenizer metadata.

M2: Q4_K_M primitives

Deliverables:

  • parser;
  • dequantization;
  • reference matvec;
  • numerical tests;
  • microbenchmarks.

M3: transformer primitives

Deliverables:

  • RMSNorm;
  • RoPE;
  • softmax;
  • attention;
  • MLP;
  • transformer layer tests.

M4: LLaDA forward pass

Deliverables:

  • model configuration;
  • tensor mapping;
  • embedding;
  • transformer stack;
  • logits;
  • comparison against reference implementation.

M5: diffusion generation

Deliverables:

  • mask canvas;
  • scheduler;
  • confidence;
  • sampling;
  • iterative generation;
  • complete text output.

At this milestone:

mojo-dllm run ...

must work.

M6: CPU optimization

Deliverables:

  • SIMD;
  • multithreading;
  • fused kernels;
  • benchmark report.

M7: NVIDIA backend

Deliverables:

  • device allocation;
  • GPU kernels;
  • end-to-end GPU inference;
  • consumer RTX benchmark.

M8: second architecture

Deliverables:

  • Dream support;
  • model abstraction cleanup;
  • generalized dLLM scheduler interface.

30. Initial Definition of Done

v0.1 is complete when all of the following are true:

This list records the 0.1 target. The changelog and the README status table record what shipped.

  • LLaDA Q4_K_M GGUF loads directly in Mojo.
  • No Python process is required during inference.
  • Prompt tokenization works.
  • Full transformer forward pass executes.
  • Bidirectional attention is correct.
  • Diffusion scheduler is implemented.
  • The model generates readable text.
  • Fixed-seed runs are deterministic.
  • Tests cover quantization, attention and scheduler primitives.
  • Peak memory fits within a reasonable consumer laptop RAM budget.
  • Benchmark comparison with diffuse-cpp is published.
  • Benchmark comparison with llama.cpp is published.
  • README documents known differences and limitations.

31. Risks

31.1 Mojo ecosystem churn

Mojo APIs may still evolve quickly.

Mitigation:

  • keep backend abstraction small;
  • pin compiler/toolchain versions;
  • isolate unstable APIs;
  • use CI against a documented supported version.

31.2 Q4_K performance

Naive Q4_K dequantization may make CPU inference unusably slow.

Mitigation:

  • optimize Q4_K early;
  • benchmark it independently;
  • borrow mathematical layout ideas from permissively licensed implementations while respecting licenses.

31.3 GGUF complexity

GGUF supports many metadata conventions and quantization formats.

Mitigation:

  • support only the exact subset required by LLaDA initially;
  • reject unsupported formats explicitly;
  • expand only after v0.1.

31.4 Tokenizer mismatch

Small tokenizer differences can invalidate inference.

Mitigation:

  • validate token IDs against a known reference tokenizer;
  • add golden prompt/token test cases.

31.5 Floating-point divergence

Different kernel implementations may diverge slightly.

Mitigation:

  • use tensor-level tolerances;
  • compare logits statistically;
  • separate numerical mismatch from algorithmic mismatch.

31.6 GPU portability may underperform

A single portable kernel may not match highly specialized vendor kernels.

Mitigation:

  • measure rather than assume portability;
  • allow compile-time specialization;
  • permit small backend-specific sections if required;
  • quantify shared vs backend-specific LOC.

32. Success Criteria

Minimum success

A working pure-Mojo CPU implementation of LLaDA-8B Q4_K_M.

Strong success

CPU performance within a reasonable factor of diffuse-cpp or llama.cpp, with a considerably smaller and more readable runtime.

High-impact success

One shared Mojo implementation runs efficiently on:

  • x86 CPU;
  • NVIDIA GPU;
  • Apple GPU;
  • AMD GPU;

with limited backend-specific code.

Research-quality success

The project demonstrates a useful dLLM-specific optimization, such as efficient inter-step reuse or portable block diffusion, that produces a measurable speedup while preserving output quality.


33. Project Positioning

Recommended one-line description:

Portable local inference for diffusion language models, written in Mojo.

Alternative:

Run LLaDA and Dream locally from GGUF using a Mojo-native diffusion inference runtime.

Avoid claims such as:

  • "first diffusion inference engine";
  • "first fast dLLM runtime";
  • "novel diffusion caching";
  • "faster than autoregressive LLMs";

unless validated by reproducible benchmarks and a fresh prior-art review.

A defensible early claim is:

A standalone Mojo-native runtime for quantized local diffusion-language-model inference.


34. README Benchmark Target

The eventual repository front page should contain a table like:

RuntimeHardwareModelQuantLatencyPeak RAMRuntime language
llama.cppCPULLaDA-8BQ4_K_MmeasuredmeasuredC/C++
diffuse-cppCPULLaDA-8BQ4_K_MmeasuredmeasuredC++
mojo-dllmCPULLaDA-8BQ4_K_MmeasuredmeasuredMojo
mojo-dllmRTX 3050LLaDA-8BQ4_K_MmeasuredmeasuredMojo

No synthetic or estimated performance numbers should be published.


35. Immediate Implementation Order

The recommended sequence is:

1. GGUF metadata reader
2. LLaDA tensor mapping
3. Q4_K_M block decoder
4. reference Q4_K matvec
5. RMSNorm
6. RoPE
7. softmax
8. bidirectional attention
9. MLP
10. single transformer layer
11. full transformer
12. tokenizer
13. diffusion canvas
14. confidence calculation
15. scheduler
16. sampling
17. full generation
18. correctness comparison
19. SIMD optimization
20. NVIDIA backend

This sequence minimizes the number of moving pieces during debugging.


36. First Engineering Spike

Before committing to the full runtime, implement a one-week feasibility spike:

Deliverable A

mojo-dllm inspect llada.gguf

Correctly prints model metadata and tensor layout.

Deliverable B

Load one Q4_K_M matrix and reproduce a reference matrix-vector multiplication within numerical tolerance.

Deliverable C

Implement one complete LLaDA transformer layer and compare output tensors against a PyTorch or C++ reference.

If these three succeed cleanly, proceed with the full project.

If Q4_K_M or GGUF support proves disproportionately difficult because of current Mojo limitations, reconsider whether the first version should use Q8_0 before returning to Q4_K_M.


37. Core Principle

The project should optimize for one question:

Can Mojo provide a clean, performant, portable execution layer for diffusion language models on ordinary local hardware?

Everything that does not help answer that question should remain outside the initial scope.