Introduction
mojo-dllm runs two families of diffusion language models, LLaDA and Dream, from
quantized GGUF files on your own CPU or on an NVIDIA GPU. LLaDA-8B and Dream-7B
are measured; LLaDA-1.5, Dream-Coder and DiffuCoder are tested on the CPU (see
Quick start). The whole
inference path is Mojo: the GGUF reader, the tokenizer, the quantized matrix
multiply, the GPU kernels, the transformer and the denoising loop. The
build/mojo-dllm binary needs no Python at runtime, and the GPU is optional.
The ./mojo-dllm launcher that builds and starts it is a Python script run
by uv.
A diffusion language model writes differently from the models you know. An
autoregressive model appends one token at a time. LLaDA starts from a canvas
of [MASK] tokens and fills it in over a fixed number of steps, committing the
tokens it is most sure about first. Diffusion decoding in five
minutes walks through it with a picture.
What works today
mojo-dllm rungenerates text from LLaDA-8B-Instruct and Dream-v0-Instruct-7B (Q4_K_M) on x86 CPUs with AVX2, using AVX-VNNI when present.--device gpuruns both models on NVIDIA GPUs with compute capability 8.0 or newer. It has been measured on an RTX 3050 Laptop GPU only.- The LLaDA and Dream tokenizers match llama.cpp id for id on a 36-case
multilingual corpus, and DiffusionGemma's
gemma4tokenizer on 54 cases. - Logits match llama.cpp's implementation of both models; the measured agreement is in Correctness.
- Benchmarks against llama.cpp and diffuse-cpp on the same machine, with every measurement table generated from a result file in the repository.
What does not work yet
- The GPU path is measured on one laptop part, an RTX 3050 held to 30 W, against llama.cpp's CUDA build; Benchmarks has the numbers. It supports NVIDIA only.
- Two architectures run: LLaDA and Dream. A
diffusion-gemmafile is read,inspectprints its config,tokenizeprints the ids of its chat prompt, andlogitsruns it on the CPU: one unified pass over the prompt and the canvas, checked against a numpy forward pass on a tiny fixture (#64). The entropy-bounded denoiser is written too (#65), butrunstops on this file withdiffusion-gemma does not run on this modeluntil the model is wired into the generation loop (#277), on the CPU and the GPU. Other architectures stop withunsupported model architecture: <name> (runs: llada, dream; read only: diffusion-gemma). - One quantization mix: Q4_K and Q6_K projection matrices, as in a Q4_K_M
file, with an embedding table and norm vectors that the reference decoders
read (F32, F16, BF16, Q5_0, Q8_0, Q4_K, Q5_K, Q6_K). A matrix of any other type
stops the load with an error that names it. DiffusionGemma's
logitsalso reads Q5_0 and Q8_0 matrices for the down projections.
Where to go next
- Run it: Quick start.
- Read the numbers: Benchmarks.
- Read the code: Architecture, then the source under
src/mojo_dllm/.
Quick start
You need Linux on x86-64 with AVX2, glibc 2.34 or newer, about 6 GB of free
memory, and uv. uv installs the pinned Mojo
toolchain (Mojo 1.1.0, max-core 26.6.0) into the project's .venv; nothing
goes system-wide.
1. Set up
curl -LsSf https://astral.sh/uv/install.sh | sh # skip if you have uv
git clone https://github.com/dllm-labs/mojo-dllm
cd mojo-dllm
./mojo-dllm setup
The uv installer puts uv in ~/.local/bin. If your shell does not find it
afterwards, open a new terminal or run source ~/.local/bin/env.
setup runs four steps and prints each one:
- Toolchain. It checks the platform, glibc and AVX2, stops
on anything that cannot work, and runs
uv sync --frozen. - Build. It compiles
build/mojo-dllm, or reports that the binary is newer than its inputs and was built for this Mojo version and GPU architecture. - Model. It asks before downloading LLaDA-8B-Instruct Q4_K_M (4.9 GB)
into the Hugging Face cache, and checks the file's sha256 against the one
the hub publishes.
--yesanswers the question for you;--no-modelskips the step;--modelpicks another model. - Check. It generates 16 tokens to prove the binary, the model and the
device work together. The check fails when the binary exits non-zero, prints
nothing, or runs longer than 300 seconds. With
--no-modelthe step is skipped and setup ends withSetup complete (no smoke test).
setup --offline (or HF_HUB_OFFLINE=1) uses the model cache only, as run and pull do.
Without a terminal to ask on (a script, CI), setup does not download
anything unless you pass --yes. It then exits 3 and prints the command that
finishes the job.
./mojo-dllm doctor runs those checks without changing anything, and adds free
memory, the Mojo version, the binary's age, the GPU and the model cache. Every
line that is not ok says how to fix it, with the command where one exists; for a
missing GPU it says that the CPU will be used.
2. Generate
./mojo-dllm run "Explain speculative decoding in two sentences."
./mojo-dllm run "Write a haiku about rain." --max-tokens 64 --steps 32 --verbose
The prompt is the first word that is not a flag; quote it. run rebuilds the
binary if a source file, pyproject.toml, uv.lock or scripts/build.sh changed, or if the
Mojo pin or the GPU architecture (MOJO_DLLM_GPU_ARCH, else the local GPU) differs from the one
recorded in build/mojo-dllm.stamp, finds the model, picks the device and hands
the rest of the flags to build/mojo-dllm run. Command line lists
them all.
A flag and its value are two arguments: --steps 8, not --steps=8. The launcher
refuses the = form and prints the two-argument form to use. A prompt that starts
with - goes after -- (./mojo-dllm run -- "-h"); -h and --help after --
belong to the prompt and do not print the help.
--max-tokens sets the canvas size, --steps the number of denoising passes.
Fewer steps run faster and commit more tokens per step, which costs quality.
Each step runs the full model over prompt and canvas, so expect seconds per
step on a laptop CPU. --verbose prints load time, per-step time and peak
memory.
The prompt goes through the model's chat template. Pass --no-chat to feed
raw text instead.
The device
--device auto is the default. It picks the first NVIDIA GPU with compute
capability 8.0 or newer that nvidia-smi lists, and the CPU when there is
none. The binary opens CUDA's default device, which can be another card than
the first one nvidia-smi lists, so the launcher sets CUDA_VISIBLE_DEVICES to
the UUID of the card it named (with CUDA_DEVICE_ORDER=PCI_BUS_ID) before it
starts the binary. A CUDA_VISIBLE_DEVICES you already set (even an empty one)
is kept: the launcher does not pin the card and prints a warning saying so.
--device gpu runs the same check: it pins the first
qualifying card, and prints a warning when nvidia-smi
finds no card or only one below compute capability 8.0. Status lines go to
stderr, so stdout holds only the model's text:
device: gpu (NVIDIA GeForce RTX 3050 6GB Laptop GPU, compute 8.6, 585 MiB free)
auto does not compare the card's free memory with the model's size. In that
line another program held most of the card, so the run that followed would
have said how much memory it needs and exited. Pass --device cpu to run
anyway.
3. Other models
-m (or --model) takes one of three forms:
| Form | Example | What happens |
|---|---|---|
| a file | -m ~/models/llada.gguf | used as is |
| the alias | -m llada-8b | mradermacher/LLaDA-8B-Instruct-GGUF, Q4_K_M (the default) |
| a repo | -m owner/Some-Model-GGUF:Q4_K_M | the file matching that quant, downloaded once |
Without :QUANT, a repo resolves to its Q4_K_M file, then Q4_K_S, then Q6_K,
the quants the matrix kernels load. A repo with none of them, and a quant that
matches nothing, fail with the list of files the repo has, and owner/repo:QUANT
selects one. Split (sharded) GGUF files are not supported. A Q8_0 file downloads
when you ask for it with :Q8_0 but does not load: the matrix kernels need Q4_K
or Q6_K.
Tested Q4_K_M checkpoints are LLaDA-8B-Instruct, LLaDA-1.5,
Dream-v0-Instruct-7B, Dream-Coder-v0-Instruct-7B and DiffuCoder-7B-cpGRPO.
The added checkpoints were checked on CPU; pass --device cpu to reproduce
those runs. See the smoke and parity evidence for the exact
files and generation agreement.
Other quantizations still need compatible tensor types. An unsupported
architecture stops with an error naming it.
For DiffuCoder, obtain the file separately under its apple-amlr license and
pass its local path with --model; it is not bundled with the project.
./mojo-dllm pull MODEL downloads without generating. Models live in the
Hugging Face cache ($HF_HUB_CACHE, else $HF_HOME/hub, else
$XDG_CACHE_HOME/huggingface/hub, else ~/.cache/huggingface/hub), shared
with every other tool that uses it. A gated or private repo needs HF_TOKEN.
The sha256 check runs on download. A file already in the cache is checked
against the hub's sha256 the first time it is used, and the launcher
remembers the result in ~/.cache/mojo-dllm/verified.json (under
$XDG_CACHE_HOME when that is set). A file that fails the check is deleted,
and the next run downloads it again. With --offline the check cannot run,
so an unchecked file is used after a warning. pull --offline and setup
then print already downloaded (not verified) and exit with status 3, and
doctor marks the model with !. Run ./mojo-dllm pull once online to check it.
4. Offline
./mojo-dllm pull # on a connected machine
./mojo-dllm run "Hello" --offline # later, with no network
--offline (or HF_HUB_OFFLINE=1) uses the cache only and fails with the
pull command to run when the model is missing.
5. Look inside a model
./mojo-dllm pull # prints the model file's path
./mojo-dllm inspect PATH.gguf | head -40
You get the metadata, a count of tensor types, and every tensor with its
shape. The binary's own commands (inspect, tokenize, logits) go to
build/mojo-dllm unchanged, after the same rebuild check. Any other word stops
with an error, and names the closest command when one is near.
Without the launcher
The launcher is a convenience. The binary works on its own:
uv sync --frozen
scripts/build.sh
build/mojo-dllm run --model PATH.gguf --prompt "Hello" --device cpu
build/mojo-dllm defaults to --device cpu and takes only file paths.
Command line
mojo-dllm run --model PATH --prompt TEXT [options]
mojo-dllm tokenize --model PATH --prompt TEXT [--no-chat]
mojo-dllm inspect PATH
mojo-dllm logits --model PATH --tokens IDS --rows POSITIONS --out FILE [--threads N] [--device D]
[--path unified|cached] [--prev FILE --prev-temp-inv X]
This page describes the binary, build/mojo-dllm. The ./mojo-dllm run launcher
takes the prompt as its first word, accepts -m for --model, defaults
--device to auto and passes the other flags on to the binary; see the
Quick start. A CUDA_VISIBLE_DEVICES you set before the launcher
starts is kept: the launcher then does not pin the card and prints one warning
saying so.
mojo-dllm COMMAND --help (or -h) prints that command's usage and exits 0.
Help wins over every other problem in the arguments, wherever it appears, except
as the value of a flag (--prompt --help sets the prompt to --help).
mojo-dllm, mojo-dllm help and mojo-dllm --help print the usage of every
command. Each command declares its flags: a flag the command does not declare
stops the run (see Errors).
mojo-dllm --version prints the package version on its first line and the
commit the binary was built from on its second, then exits 0:
mojo-dllm 0.2.0
commit: abc123456789
The package version is the "version" key of run --json. The commit is the
12-character HEAD of the checkout the binary was built from, with a -dirty
suffix when src/ had pending changes, or unknown when the build cannot read
Git metadata. A binary keeps reporting the commit it was built from after
HEAD moves; rebuild to stamp the new one.
run
| Flag | Default | Meaning |
|---|---|---|
--model PATH | required | LLaDA or Dream GGUF file |
--prompt TEXT | required | the user message |
--max-tokens N | 128 | tokens to generate (the masked canvas). The prompt tokens plus N must not exceed the file's <arch>.context_length (4096 when the key is absent); the check runs after the header is read and before any weight is mapped |
--steps N | --max-tokens | denoising steps; a multiple of the block count |
--block-length N | 32, or --max-tokens when that is smaller | LLaDA's semi-autoregressive block size; must divide --max-tokens. A Dream run refuses it |
--temperature T | 0 | LLaDA: 0 is greedy; above 0 adds Gumbel noise as LLaDA does. Dream supports only 0 and stops with an error above it |
--remasking S | low_confidence | LLaDA: low_confidence or random. Dream: entropy (the default), maskgit_plus, topk_margin, origin, or llama.cpp-style random |
--threshold P | 0 (off) | A number from 0 to 1. LLaDA: each step also commits every masked token whose confidence is at least P, so easy blocks finish in fewer steps (Fast-dLLM's parallel decoding). A step still commits at least what the schedule says. Not exact: see Benchmarks for its measured effect on accuracy |
--dual-cache | off | LLaDA: cache every position's keys and values and, inside a block, recompute only that block's rows (Fast-dLLM's DualCache). The cache is refreshed at the first step of each block. Not exact: the prompt and later blocks see keys from the block's first step; see Benchmarks for its measured effect on accuracy. Cannot combine with --full-logits |
--seed N | 42 | seeds this program's PRNG (it does not reproduce a PyTorch run). It fixes DiffusionGemma's random canvas and draws as well |
--eb-steps N | 48 | DiffusionGemma: most denoising steps, at least 1 (use it instead of --steps, which that model refuses) (no effect yet, see the paragraph after the table) |
--eb-t-min T | 0.4 | DiffusionGemma: the temperature the schedule approaches, 0 or more; the last of --eb-steps steps runs at T + (t-max - T) / steps (no effect yet, see the paragraph after the table) |
--eb-t-max T | 0.8 | DiffusionGemma: the temperature of the first step, above 0; it falls linearly from there (no effect yet, see the paragraph after the table) |
--eb-bound B | 0.1 | DiffusionGemma: a position is accepted while the entropies of the lower-entropy positions before it sum to at most B, which must be at least 0 (no effect yet, see the paragraph after the table) |
--eb-stability N | 1 | DiffusionGemma: steps the argmax canvas must hold before the run may stop; 0 or more, and 0 lets the first step stop it (no effect yet, see the paragraph after the table) |
--eb-confidence C | 0.005 | DiffusionGemma: the run may stop once the mean entropy over the canvas is below C, which must be at least 0 (no effect yet, see the paragraph after the table) |
--threads N | all logical CPUs | worker threads of every parallel CPU loop, including the activation quantization and the weight repack at load (with --device gpu, only the sampler uses them) |
--device D | cpu | cpu, or gpu for an NVIDIA GPU with compute capability 8.0 or newer |
--no-chat | off | skip the chat template |
--full-logits | off | LLaDA: compute logits for every position, as a generic runtime would (for benchmarking). Dream refuses it |
--no-suppress-mask | off | allow the mask token as a sampled answer for parity experiments; by default it is excluded from argmax and the softmax normalizer |
--no-early-stop | off | run every step even after the answer is final (the benchmark and parity harnesses use it) |
--visual | off | show the canvas: masked positions as teal blocks, tokens committed in that step in bold orange when colour is enabled |
--visual-interval N | 1 | redraw every N steps and at the final step; requires --visual and N at least 1 |
--verbose | off | model, device and load time before the text; step timings and peak memory after it |
--dump-step-stats | off | one line per denoising step |
--json | off | one JSON object with timings, tokens and text. peak_rss_bytes is null when the operating system could not report it, never 0. remasking names the strategy the run used (for Dream, entropy when the flag says low_confidence); block_length and full_logits appear for LLaDA only, because Dream reads neither. steps_taken counts the denoising steps run and stop_reason says why the loop ended: complete, answer_fixed (early stop), or for DiffusionGemma, once run accepts it (#277), converged or max_steps |
The flags --threshold, --dual-cache, --full-logits and --block-length are LLaDA's; a Dream run refuses them by name before the model loads (--threshold applies to llada only, not dream).
The printed answer ends at the first end-of-text or end-of-turn token in the
canvas. Generation stops as soon as that answer is final: once the first
position after the prompt that is still masked or holds such a token holds
such a token, no later step can change the printed text, because steps only
fill masked positions. The result is the same text with fewer steps, and
--json reports "stopped_early": true. --no-early-stop runs every step.
Dream's origin follows its Python reference: each masked position commits
independently with probability 1 - s/t, forced to 1 at the final step. Zero
commits in an intermediate step is valid. This differs from llama.cpp's
origin, which floors the transfer count before calculating the probability.
Dream random follows llama.cpp: assign each candidate a uniform random
confidence, then select the scheduled top-k positions. Both rules use --seed.
The default remains entropy for Dream; select --remasking origin to use
Dream's reference algorithm.
Mask suppression keeps the Gumbel RNG draw for the excluded token, so toggling
it does not shift the random stream. On GPU, suppression uses full logits
because the device row statistics include the mask. --no-suppress-mask
retains the existing device reduction path when its other conditions hold.
The --eb-* flags belong to DiffusionGemma and every other model refuses them by name. The parser accepts them, but run stops on a diffusion-gemma file until the model is wired into the generation loop (see Introduction). It refuses the file once its header is read, before the tokenizer is built, with diffusion-gemma does not run on this model: `run` waits for the model to be wired into the generation loop (#277), so use `logits` (CPU) until then, so the --eb-* flags have no effect yet; run --help says so. Only logits runs that architecture. The forward pass (#64) and the entropy-bounded loop (#65) are written; #277 tracks joining them. Once it lands, DiffusionGemma's --json reports "stopped_early": true when the stop rule ends the run before --eb-steps steps.
Visual output
The final canvas always appears, including when generation stops early between redraws. Its step number is the last observed step; any remaining masks stay visible. A final frame that was already drawn is not repeated.
| stdout | NO_COLOR | colour codes | clear + home (ESC[2J ESC[H) |
|---|---|---|---|
| terminal | unset or empty | yes | yes |
| terminal | non-empty | no | yes |
| not a terminal | any | no | no; frames separated by one blank line |
On a terminal, --visual leaves the last canvas frame on screen and does not
print the answer a second time. Without --visual, or when stdout is not a
terminal, run prints the answer after generation.
Without colour, newly committed tokens have no highlight. Redirected frames
contain no ANSI escapes. --visual, --verbose and --dump-step-stats print to
stdout before the JSON object, so each of them is refused together with --json:
the command stops with the usage error --json cannot be combined with --visual
(or --verbose, --dump-step-stats) and exit status 2, before the model loads. With
--json, stdout holds the one JSON object and nothing else.
The checks that need only the flags run before the model file is opened, and each exits with status 2 and names the flag:
--threshold outside 0 to 1 or not finite, --device other than cpu or gpu, --remasking outside the six names
above, and --threshold above 0 with --remasking random. LLaDA also needs a --block-length that divides
--max-tokens and a --steps that is a multiple of the number of blocks (Dream reads neither flag and accepts any
value); that check depends on the architecture, which only the file's header says, so it runs once the header is read,
still before any weight is mapped, and exits with status 1, as does the check of the default block length (32, or
--max-tokens when smaller): flag --max-tokens 100 is not a multiple of the default block length 32. The same header check refuses a --remasking value that LLaDA does not take (entropy, maskgit_plus,
topk_margin and origin); Dream takes all six.
Invalid --visual-interval values and use without --visual exit with status
1 before model loading. The messages name --visual-interval: must be at least 1, needs --visual, or needs an integer.
tokenize
Prints the token ids of the chat-formatted prompt, the same ids run feeds the
model. Useful to compare against llama-tokenize. For a diffusion-gemma file
the prompt is the file's chat template: <bos>, a user turn and an open model
turn, which gives one BOS token (id 2). It reads the tokenizer arrays of the
file and none of its weights, so it runs on a file too big to load.
Only llada, dream and diffusion-gemma files have a chat template here: for another
architecture the command stops with no chat template for architecture 'qwen2' (llada, dream and diffusion-gemma have one); pass --no-chat to tokenize the text as it is, and --no-chat
tokenizes the text unchanged. run applies the same rule to its prompt.
inspect
Prints GGUF metadata, a count per tensor type, and every tensor's type and
shape. It needs no supported architecture, so it works on any GGUF file. For
diffusion-gemma it also prints a config: block with what the reader took
from the file (layer kinds, K/V heads and head size per kind, experts, canvas
length), or config: refused: and the reason, such as the key that is wrong.
For llada, dream and diffusion-gemma it also lists each metadata key that
the file lacks and the reader replaced by a default, one per line under
defaults (key absent from the file):, as llada.rope.freq_base = 10000 (default, key absent). A file that sets every such key prints no such block.
run --verbose prints the same block after the device line. The keys are the
KV head count, the context length, the rope base and the logit shift for LLaDA
and Dream, and the context length and the final logit softcap for
DiffusionGemma.
logits
Runs one forward pass over the given token ids and writes f32 logits for the
requested positions to a file, [positions, vocab], row major. The parity
tests use it; see Correctness.
The file is written to FILE.tmp, checked there to hold exactly positions * vocab * 4 bytes, and
only then renamed to FILE; any failure (a full disk, a quota, a missing or unwritable directory)
removes the temporary file, prints ERROR: cannot write FILE: ... with the system's reason, exits with
status 1 and leaves any earlier FILE as it was. wrote N rows appears only after the rename.
The rename replaces a symlink named FILE (the link goes and its target stays as it was) and a read-only
FILE. When FILE already exists and is not a regular file (/dev/null, /dev/stdout, a pipe), the bytes
are written to it directly: there is no name to rename to and no size to check.
On a diffusion-gemma file the tokens are the prompt followed by the canvas,
whose length the file declares (diffusion.canvas_length, 256 in the real
model). The model scores only the canvas, so --rows must list exactly those
positions, the last canvas-length ones, and the file holds [canvas, vocab].
It runs on the CPU only; --device gpu is refused. Three flags apply to this
architecture alone and are refused on the others:
--path unified(the default) runs the prompt and the canvas in one pass.--path cachedprefills the prompt, keeps its keys and values, and steps the canvas against them. The two agree bit for bit.--prev FILEnames the previous denoising step's canvas logits,[canvas, vocab]f32 as this command writes them, and turns self-conditioning on.--prev-temp-inv X, which must come with it, is 1 divided by the temperature of that previous step (1.25 for the first step's 0.8). It must be above 0 and its product with the model's final logit soft cap must not exceed 1e8 (about 3e6 with the cap of 30), because the softmax turns the logits to NaN beyond that, and--pathmust beunifiedorcached; the path is checked before any model is read, the bound when the pass starts.
--rows and the size of the --prev file are checked against the canvas length
that the file's header declares, before a weight is mapped, so a wrong --rows
costs no model load.
Errors
An error names the thing that is wrong. Usage errors exit with status 2 and
change nothing; every other error exits with status 1. Every ERROR: message goes to
stderr, so stdout stays free for the answer or the JSON object; help text and
--version go to stdout.
| Message | Cause and next step |
|---|---|
ERROR: unknown flag --stpes for 'run' (did you mean --steps?) | The command does not declare the flag. The parenthesis appears when a declared flag is at most 2 edits away. The next line is see: mojo-dllm run --help |
ERROR: missing required flag --model for 'run' | A required flag is absent. The usage of the command follows |
ERROR: missing model path for 'inspect' | inspect takes the path as an argument. The usage follows |
ERROR: unknown command: bogus | The first argument is not a command. The next line is see: mojo-dllm --help |
ERROR: unexpected argument 'x' for 'run' | run, tokenize and logits take flags only, and inspect takes one path |
ERROR: unknown flag --model=x for 'run' | Flags take their value as the next argument. The next line says flags take their value as the next argument: --model x |
ERROR: flag --prompt needs a value | A value flag is the last argument |
ERROR: flag --steps needs an integer, got 'abc' | A numeric flag holds text, or a number too large for 64 bits. --temperature needs a finite number and --tokens and --rows need comma-separated integers |
ERROR: flag --prompt needs a value, got the flag --verbose | A value flag is followed by the name of a declared flag, which is almost always a forgotten value. --help stays a legal value |
ERROR: flag --steps is given twice | A value flag appears twice; the command no longer keeps the last silently. A boolean flag repeated is harmless |
ERROR: flag --rows needs comma-separated integers, got '0,,2' | An empty entry in --tokens or --rows (a trailing comma stays legal) |
ERROR: flag --steps must be at least 1, got 0 | --max-tokens, --steps, --block-length, --threads and --eb-steps need at least 1, and --eb-stability at least 0. --temperature, --eb-t-min, --eb-bound and --eb-confidence need a finite number of at least 0, --eb-t-max and --prev-temp-inv one above 0 (not only finite), and --threshold a number from 0 to 1. The check runs before the model is mapped |
ERROR: prompt 10 + --max-tokens 4096 = 4106 exceeds the context length 4096 (llada.context_length) | The canvas is longer than the file's context length. Lower --max-tokens. Exit status 1, before any weight is mapped |
ERROR: cannot open model file PATH: No such file or directory | The path is wrong. The reason is the operating system's. The next line is to download a model, run: ./mojo-dllm pull |
ERROR: cannot open model file PATH: Permission denied | The open failed for a reason other than a missing file, such as Not a directory. The operating system names the reason. No download hint follows, because pull would not fix it |
ERROR: file is empty: PATH | The model file has no bytes. Download it again |
ERROR: cannot get the size of PATH: Illegal seek | The path opens but is not a regular file (a pipe or a FIFO). The operating system names the reason |
ERROR: mmap failed for PATH: No such device | The file cannot be mapped (a device or a pseudo file). The operating system names the reason; Cannot allocate memory means the address space is exhausted |
ERROR: is a directory: PATH | The model path names a directory |
ERROR: flag --prompt is not valid UTF-8: invalid byte at offset N | run and tokenize refuse a prompt that is not UTF-8 (a Latin-1 text, for example) before the model loads and exit with status 2. N is the byte offset of the first bad byte. Save the text as UTF-8 |
ERROR: PATH is not a GGUF file (bad magic). | A failed download leaves an HTML error page with a .gguf name. Delete the file and run ./mojo-dllm pull again |
The remaining errors exit with status 1 and come from the model file itself:
ERROR: unsupported GGUF tensor type: IQ2_XXS (tensor blk.0.attn_q.weight)
ERROR: tensor blk.0.attn_q.weight is Q8_0: matrix kernels exist for Q4_K and Q6_K
ERROR: unsupported model architecture: qwen2 (runs: llada, dream; read only: diffusion-gemma)
ERROR: required metadata key missing: tokenizer.ggml.model
ERROR: tensor blk.0.attn_q.weight extends past the end of the file
ERROR: tensor blk.0.attn_q.weight has offset 16, which is not a multiple of the alignment 32
ERROR: tensor blk.0.attn_q.weight is Q8_0 with 33 elements per row, which is not a multiple of its block size 32
ERROR: tensor blk.0.attn_q.weight is listed twice (entries 3 and 9)
ERROR: metadata key general.name is listed twice (entries 1 and 5)
ERROR: tensor shape mismatch for blk.0.attn_q.weight: [4096, 2048], expected [4096, 4096]
ERROR: non-finite logits at canvas position 3, step 1
ERROR: no GPU found: the GPU backend needs an NVIDIA GPU with compute capability 8.0 or newer (run with --device cpu)
non-finite logits comes from the LLaDA and Dream loops, on the CPU and the GPU: a canvas row whose softmax is not a
number (a NaN or +inf logit, or every logit -inf) stops the run instead of committing token 0 there. The position
counts from the start of the canvas and the step from 1. One -inf logit among finite ones is a token of probability 0
and passes for LLaDA; Dream's default entropy confidence has no value for such a row, so Dream refuses it too.
With --device gpu, setting MOJO_DLLM_GPU_PROFILE=1 makes --verbose split
a step into GEMM, attention and LM head time. It synchronizes the GPU after
each kernel group to measure them, so leave it off when you benchmark.
Benchmarks
LLaDA-8B-Instruct, Q4_K_M. 13th Gen Intel(R) Core(TM) i5-13420H, 12 threads, 33.3 GB RAM, performance power profile. Each prompt: 128 generated tokens, 32 denoising steps; median of 3 runs x 3 prompts.
| Runtime | ms / step | tokens / s | startup | peak RSS | language |
|---|---|---|---|---|---|
| mojo-dllm | 3,005 | 1.33 | 1.8 s | 5.9 GB | Mojo |
| mojo-dllm, logits for every position (ablation) | 3,268 | 1.22 | 1.8 s | 5.9 GB | Mojo |
llama.cpp llama-diffusion-cli | 5,807 | 0.69 | 2.5 s | 8.4 GB | C/C++ |
| diffuse-cpp, cache off | 9,996 | 0.40 | 2.3 s | 8.2 GB | C++ |
diffuse-cpp, cache + entropy_exit † | 4,380 | 1.12 | 2.6 s | 8.4 GB | C++ |
Versions: mojo-dllm bdd268b (cpu-llada) and dd0a647 (cpu-diffuse, diffuse-cpp rows only), Mojo 1.1.0 (8189361e), llama.cpp 836d571, diffuse-cpp 1d2bd6a. Evidence: bench/results/2026-10-03-cpu-diffuse.json, bench/results/2026-10-03-cpu-llada.json.
† Not the same work: the inter-step cache reuses stale K/V and entropy_exit can stop early. Shown because it is diffuse-cpp's recommended mode.
LLaDA-8B-Instruct on the GPU, Q4_K_M. NVIDIA GeForce RTX 3050 6GB Laptop GPU (6144 MiB, driver 595.99.02), host 13th Gen Intel(R) Core(TM) i5-13420H. Each prompt: 128 generated tokens, 32 denoising steps; median of 3 runs x 3 prompts. Peak RSS is host memory.
| Runtime | ms / step | tokens / s | startup | peak RSS | language |
|---|---|---|---|---|---|
mojo-dllm --device gpu | 221 | 18.13 | 2.2 s | 1.1 GB | Mojo |
llama.cpp llama-diffusion-cli, CUDA, all layers offloaded | 243 | 16.46 | 1.4 s | 5.3 GB | C/C++, CUDA |
Step time of mojo-dllm over llama.cpp: median 0.906, range 0.896 to 0.908 across 9 paired runs, with mojo-dllm's step shorter in 9 of them. A ratio below 1 means mojo-dllm's step is shorter.
Versions: mojo-dllm 69e4024, Mojo 1.1.0 (8189361e), llama.cpp 7df8b87. Evidence: bench/results/2026-11-03-gpu-llada.json.
Dream-v0-Instruct-7B, Q4_K_M. 13th Gen Intel(R) Core(TM) i5-13420H, 12 threads, 33.3 GB RAM, performance power profile. Each prompt: 128 generated tokens, 32 denoising steps; median of 3 runs x 3 prompts.
| Runtime | ms / step | tokens / s | startup | peak RSS | language |
|---|---|---|---|---|---|
| mojo-dllm | 2,967 | 1.35 | 1.7 s | 5.6 GB | Mojo |
llama.cpp llama-diffusion-cli | 5,566 | 0.72 | 2.5 s | 8.0 GB | C/C++ |
Versions: mojo-dllm bdd268b, Mojo 1.1.0 (8189361e), llama.cpp 836d571. Evidence: bench/results/2026-10-03-cpu-dream.json.
Dream-v0-Instruct-7B on the GPU, Q4_K_M. NVIDIA GeForce RTX 3050 6GB Laptop GPU (6144 MiB, driver 595.99.02), host 13th Gen Intel(R) Core(TM) i5-13420H. Each prompt: 128 generated tokens, 32 denoising steps; median of 3 runs x 3 prompts. Peak RSS is host memory.
| Runtime | ms / step | tokens / s | startup | peak RSS | language |
|---|---|---|---|---|---|
mojo-dllm --device gpu | 219 | 18.29 | 2.0 s | 1.2 GB | Mojo |
llama.cpp llama-diffusion-cli, CUDA, all layers offloaded | 214 | 18.66 | 1.3 s | 5.0 GB | C/C++, CUDA |
Step time of mojo-dllm over llama.cpp: median 1.022, range 1.000 to 1.037 across 9 paired runs, with mojo-dllm's step shorter in 1 of them. A ratio below 1 means mojo-dllm's step is shorter.
Versions: mojo-dllm 69e4024, Mojo 1.1.0 (8189361e), llama.cpp 7df8b87. Evidence: bench/results/2026-11-03-gpu-dream.json.
LLaDA-8B-Instruct, llama.cpp thread sweep and forward pass alone: 13th Gen Intel(R) Core(TM) i5-13420H; median of 2 runs, one prompt, canvas 148 tokens.
| Runtime | threads | ms / step | forward pass alone |
|---|---|---|---|
| mojo-dllm | 12 | 3,014 | 3,011 ms |
| llama.cpp | 4 | 7,284 | 7,098 ms |
| llama.cpp | 6 | 6,416 | 7,046 ms |
| llama.cpp | 8 | 6,140 | 6,242 ms |
| llama.cpp | 12 | 5,703 | 5,748 ms |
The ms / step column runs llama-diffusion-cli. The forward pass alone is mojo-dllm's measured forward pass against llama.cpp's llama-bench at the same token count. llama-bench applies a causal mask and computes logits for one position; mojo-dllm's figure includes logits for up to 32 positions and copying the canvas in. CPU busy before a run: at most 6.5%. Evidence: bench/results/2026-10-09-fair-cpu-llada.json.
LLaDA-8B-Instruct on the GPU, forward pass alone: NVIDIA GeForce RTX 3050 6GB Laptop GPU; median of 3 runs, one prompt, canvas 148 tokens.
| Runtime | ms / step | forward pass alone |
|---|---|---|
mojo-dllm --device gpu | 225 | 225 ms |
llama.cpp CUDA, llama-bench -p 148 | 243 ms |
The forward pass alone is mojo-dllm's measured forward pass against llama.cpp's llama-bench at the same token count. llama-bench applies a causal mask and computes logits for one position; mojo-dllm's figure includes logits for up to 32 positions and copying the canvas in. CPU busy before a run: at most 3.1%. Evidence: bench/results/2026-11-03-fair-gpu-llada.json.
What is compared
For LLaDA, all three runtimes load LLaDA-8B-Instruct Q4_K_M and denoise a canvas of L = prompt + 128 tokens in 32 steps. For Dream, mojo-dllm and llama.cpp (diffuse-cpp's Dream support needs its own converted file, not benchmarked here) do the same with Dream-v0-Instruct-7B Q4_K_M; both run Dream's timestep schedule over the whole canvas. A step is one full forward pass over the canvas in each of them, so the step counts and shapes match.
| Runtime | How it runs | Schedule |
|---|---|---|
| mojo-dllm | run --max-tokens 128 --steps 32 --block-length 32 | LLaDA blocks of 32, low-confidence |
| mojo-dllm (ablation) | the same plus --full-logits | logits for every position, as the other two compute them |
| llama.cpp | llama-diffusion-cli -ub L --diffusion-steps 32 --diffusion-eps 0.001 | timestep schedule |
| diffuse-cpp | diffuse-cli -n 128 -s 32 --no-cache --remasking low_confidence | one block, cosine schedule |
| diffuse-cpp † | diffuse-cli -n 128 -s 32 --remasking entropy_exit | its recommended mode |
diffuse-cpp cannot read llama.cpp's GGUF files, so it loads its own
conversion of the same model (diffuse-cpp/LLaDA-8B-Instruct-GGUF,
llada-8b-q4km.gguf, 5.5 GB): F16 embeddings and a slightly different Q4_K /
Q6_K mix. It has no tokenizer and receives the token ids mojo-dllm produces.
It prints no timings either, so the harness timestamps the two lines it writes
around its generate call.
llama.cpp runs its timestep schedule because its block mode needs the prompt length to be a multiple of the block size. The schedules commit tokens in a different order, which changes the text but not, we expect, the cost of a step. That is an argument. The measurement that does not depend on the schedule is the "forward pass alone" table below, which times one pass at the same token count.
† diffuse-cpp's recommended mode reuses K/V between steps and stops early when the canvas has settled. It does less work, so it is not a like-for-like row.
On the GPU
bench/config-gpu.json and bench/config-dream-gpu.json compare
mojo-dllm --device gpu with the same llama-diffusion-cli built with CUDA
and run with -ngl 99; the harness refuses a CUDA run unless llama.cpp
reports every layer on the GPU. Canvas, steps and prompts match the CPU runs.
These configs leave the pytest watchdog off ("kill_pytest": false), so each
result records how busy the CPU was before every run. Peak RSS is host memory;
neither runtime reports GPU memory. The GPU tables above come from these
configs.
The test GPU is a laptop part held to 30 W, whose clocks fall under
sustained load (bench/checks/2026-10-12-gpu-power-under-load.txt and
bench/checks/2026-11-03-gpu-power-limit.txt) and vary from day to day: the
same llama.cpp build ran LLaDA steps much slower on 2026-10-12 than on
2026-10-09 (compare bench/results/archive/2026-10-09-gpu-llada.json with
bench/results/archive/2026-10-12-gpu-llada.json). The 2026-11-03 file
records three builds run back to back at about 1.1 GHz and 30 W: absolute
milliseconds follow the clock, and ratios inside one file do not.
Both runtimes in one result file run interleaved, prompt by prompt, so
compare them within a file, not across dates.
bench/config-fair-cpu.json and bench/config-fair-gpu.json address two
questions about fairness: whether llama.cpp does better below 12 threads on a
hybrid CPU (a sweep over -t 4, 6, 8, 12), and how the forward passes compare
without either runtime's sampler (mojo-dllm's measured forward pass against
llama.cpp's llama-bench at the same token count).
Method
bench/run_bench.py does the measuring:
- three prompts and, by default, three repetitions (the fairness configs use one prompt, and the CPU thread sweep ran two repetitions), runtimes interleaved within each repetition;
- every run waits until a two-second sample shows the CPU under 10% busy, and
starts anyway after 300 seconds (60 in the fairness configs); the busy
fraction before each run is recorded as
cpu_busy_before, and the tables disclose runs that started above the threshold; - the power profile is
performancefor the duration and restored after; - on battery, where a laptop caps CPU and GPU power, it pauses before the
next run until the charger is back, and a run the charger left midway is
thrown away and repeated (
bench/power.py; each run recordspower_waited_s,power_retries,on_battery,power_sourceand a snapshot of the supplies it read, and--allow-batterymeasures on battery instead). The check fails closed: a supply file it cannot read stops the run, and a battery with the adapter offline counts as battery whatever its status says; - a watchdog kills any
pytestprocess (other work on the machine started test suites mid-run), and the kills are logged in the result file; - generation time excludes model loading, which is reported as startup;
- peak RSS comes from
/usr/bin/time.
The table above is generated from bench/results/*.json by
scripts/bench_table.py, and CI fails if a table differs from the evidence file
it names. Two gaps are open: --check passes when a table marker is missing, a
target file is missing, or the quality or eval evidence is deleted (#141), and an
unknown mode exits 0 (#142).
Benchmark environment
A timed run reports whether it had the machine to itself. bench/isolation.py does the checking for
bench/run_bench.py, bench/eval_gsm8k.py run and bench/eval_engine.py run; bench/power.py still owns the
mains-power check. Each check below is a constant in bench/isolation.py, and an arm can change one only by
passing a Thresholds object, which is recorded in the arm's environment file.
Before an arm starts (preflight, one 5 second sample). A FAIL stops the arm and names the offenders; --allow-noisy
continues and records that it did. A source that cannot be read is a WARN that names the file, never a pass.
| Check | FAIL when | Source |
|---|---|---|
| power | on battery (--allow-battery turns it into a WARN) | bench/power.py |
| profile | powerprofilesctl get or /sys/firmware/acpi/platform_profile is not performance | LLVM "Benchmarking tips", pyperf "Tune the system for benchmarks" |
| governor | a CPU is not on performance, unless it is powersave under intel_pstate active with preference performance | kernel admin-guide/pm/intel_pstate |
| CPU busy | more than 10% of all CPUs busy over the sample | /proc/stat |
| CPU pressure | PSI CPU some avg10 above 5% | kernel accounting/psi |
| swap movement | any page swapped in or out over the sample | /proc/vmstat pswpin, pswpout |
| memory | MemAvailable below 1.25 times the model file, or the model file's size cannot be read | /proc/meminfo |
| thermal | a core or package throttle counter rose over the sample | kernel admin-guide/thermal/intel_thermal_throttle |
| GPU processes (GPU arm) | any compute process on the GPU | nvidia-smi --query-compute-apps |
| GPU memory (GPU arm) | more than 300 MiB used | nvidia-smi --query-gpu=memory.used |
| GPU utilization (GPU arm) | mean of three samples above 5% | nvidia-smi --query-gpu=utilization.gpu |
| GPU clock reasons (GPU arm) | any event reason other than idle (0x1), a software power cap (0x4) and the display clock setting (0x100) | NVML clocks event reasons, clocks_event_reasons.active (clocks_throttle_reasons.active on older drivers) |
Swap in use is a WARN, because a run that touches swapped pages measures the disk. The preflight also records the
CPU governor and frequency, intel_pstate status and no_turbo, the energy performance preference, the load
average, PSI for CPU and memory, MemAvailable, the top CPU processes, the GPU's state (persistence mode,
compute mode, power draw, temperature, SM clock, maximum SM clock), the systemd timers due within the next hour
(names only) and the number of lines in the user crontab.
Around every problem (Window, sampled every 0.5 s). The window subtracts the CPU time of the benchmark's own
process tree, read from /proc/PID/stat, from the busy time in /proc/stat, so a benchmark that uses every core is
not contention. A problem is noisy when:
| Check | Noisy when | Per-window value recorded |
|---|---|---|
| other CPU, mean | processes outside the tree use more than 5% of the machine's CPU capacity | other_cpu_share_mean |
| other CPU, peak | the same share exceeds 25% in any 0.5 s sample | other_cpu_share_max |
| swap | any page moved | swap_pages |
| thermal | a throttle counter rose | thermal_events |
| GPU clock reasons (GPU arm) | hardware slowdown (0x8), software or hardware thermal (0x20, 0x40), hardware power brake (0x80), applications clocks (0x2) or sync boost (0x10) was active | gpu_reasons_seen |
| GPU processes (GPU arm) | a compute process outside the benchmark tree appeared | gpu_foreign_pids |
| sampler | the sampler thread died or did not stop | reason sampler-failed |
| unmeasured | a source that exists on this machine could not be read during the window | reason unmeasured:SOURCE |
Recorded in every window and never judged: PSI CPU and memory some stall time (psi_cpu_stall_ms,
psi_memory_stall_ms), because a 12-thread run on 12 CPUs stalls 9% of the time by itself (measured on the
reference laptop: 24 ms in 10 s idle, 908 ms with 12 busy threads), so no limit separates the run from a neighbour;
a software power cap (0x4, gpu_power_capped), which the laptop's 30 W GPU hits under any compute load; the display
clock setting (0x100); the SM clock minimum and maximum; the window length; and the processes outside the tree that
used the most CPU (other_top_processes, percent of one core).
A source that does not exist on the host (no NVIDIA GPU on a CPU arm, no thermal_throttle files on an AMD CPU) is
not checked.
The child that exits inside the window is handled: a tree member's ticks are its own utime + stime plus the
cutime + cstime of the children it has waited for, and the window compares the sum over the tree between two
samples. When a child exits and is reaped, its ticks move from the child to its parent and the sum does not drop.
The Window's pid must outlive the window (the runner's own pid does).
The GPU is read every 2 s inside a window (two nvidia-smi processes per read, about 0.2 s of wall time and 0.1 s of
CPU time per read, so about 5% of one core of 12 while a GPU arm runs; they are children of the runner, so they sit in
the tree and are not counted as other). A GPU event reason that starts and ends between two reads is missed.
What the checks cannot see. Kernel threads and interrupt time are inside busy and are flagged as other CPU, but
they cannot be named. Hypervisor and firmware (SMM) time, fan and embedded-controller policy and thermal paste are
invisible. GPU use by the display server is not a compute app and counts only through the 300 MiB memory limit and
the utilization sample. Other users' processes are seen, but only at the 0.5 s resolution, and a burst shorter than
that, or shorter than the 10 ms tick, can fall between samples. A foreign GPU process that starts and ends between
two GPU reads is missed. Processes in another pid namespace (containers) can differ from the pids nvidia-smi
prints, so a GPU arm inside a container cannot match its own pid. A window can also miss one stolen core for a part
of a long problem; the numbers are in the next paragraph.
Sensitivity. The limits (5% of capacity as a mean, 25% in any sample) were kept after measuring the baseline on
the reference laptop with the shared machine in its normal state: 60 windows of 10 s around a child that burns one
core, other-process share of total capacity: mean of the window means 9.0%, median 8.0%, 95th percentile 20.6%,
maximum 25.3%; peak share mean 19.5%, 95th percentile 39.7%, maximum 47.5%. Only 14 of the 60 windows came out clean at
these limits, because other sessions were using the machine; the windows named them (a terminal session, python3, qemu).
Since the baseline is far above 1%, the limits are not tightened. The consequence is a blind spot: one stolen
core for 30 s of a 90 s problem is 2.8% mean and 8.3% peak of 12 CPUs, and the CPU rules do not flag it. On a quiet
machine (scripts/bench-quiet.sh, no other session) rerun the baseline and lower the limits by passing a Thresholds.
Before a problem the runner waits up to 60 seconds for a quiet moment (CPU busy, swap movement and, for a GPU arm, GPU
processes), then runs it inside a window. A noisy problem is run again, up to --isolation-retries
times (default 2); after that the row is kept with its reasons. bench/run_bench.py keeps its own busy_threshold
and max_quiet_wait_s config keys for this wait, and records cpu_busy_before as before.
Reading ARM.env.json. The eval runners write bench/eval/runs/ARM.env.json and run_bench.py writes
bench/results/env/NAME.env.json. It holds one entry per start of the arm under sessions: the preflight report
(each check with its status and detail), snapshot_start and snapshot_end, the thresholds used, tuned (the
profile before and after --tune), bench_quiet (true when the run was started under scripts/bench-quiet.sh) and bench_quiet_changes (what that
script changed: timers_stopped, gpus_persistence_on, gpus_clocks_locked, and refused, the changes it tried and
could not make, a refused clock lock among them; null when the script gave no detail),
preflight_failed_allowed (true when --allow-noisy continued past a FAIL), a session_id, the retry limit and tool
versions. Each row of an arm file carries isolation: the window values above, clean, reasons, retries (how
many repeats this problem took), noisy, cpu_busy_before, waited_s, session (the session_id of the start
that wrote the row), allow_noisy and preflight_failed_allowed.
eval_gsm8k.py compare adds generate_s_mean_clean (the mean over the problems whose window was clean in both
arms, null when there is none), clean_rows, noisy_rows per arm and unchecked_rows per arm (rows written
before windows existed). Accuracy is still computed over every row.
Quieting the machine. --tune sets the power profile to performance through powerprofilesctl, reads it back,
and restores the previous profile on exit, on an exception, on Ctrl-C and on SIGTERM. sudo scripts/bench-quiet.sh -- COMMAND additionally stops the system timers sysstat-collect, apt-daily, apt-daily-upgrade, fwupd-refresh,
anacron, man-db and motd-news, turns on NVIDIA persistence mode and locks the GPU clocks where the driver
allows it, runs the command, and puts everything back; it never edits cron files. After a SIGKILL or a power loss no
trap can run: sudo scripts/bench-quiet.sh --restore undoes the script's changes from its state file, and --tune
(which keeps no state file) leaves the profile at performance, so run powerprofilesctl set PREVIOUS with the
profile in tuned.previous of the arm's environment file. The one-minute user cron job is not
stopped: a window that sees it is noisy and the problem is repeated. Not covered: kernel boot parameters (isolcpus,
nohz_full) and BIOS settings. The rows keep the 12 threads they were measured with, so no core is left to pin.
Rows measured before this change. The published rows above (and the arms under bench/eval/ written before
isolation existed) were not checked this way. They have cpu_busy_before for the CPU rows and no window. Do
not read a ratio from them closer than their recorded spread.
Steps and quality
A diffusion model commits tokens / steps tokens per step. The table above
uses 4 tokens per step, which is fast and shows in the text as repeated or
dropped words. More steps cost proportionally more time, so the headline
tokens / s above is not a quality-matched figure. In the first prompt below,
the repeated words disappear at 64 steps, which takes about twice the time of
32 steps, so by that count mojo-dllm's quality-matched speed on that prompt is
about half the headline rate. Other runtimes use other schedules. The code prompt shows no such damage at 32 steps.
LLaDA-8B-Instruct.Q4_K_M.gguf, 128 generated tokens.
| Prompt | steps | tokens / step | time | repeated words | output (first 120 characters) |
|---|---|---|---|---|---|
| Explain speculative decoding in two sentences. | 32 | 4 | 93 s | 99 | Speculative decoding is a process used in machine communication to improve the likelihood of a message being by by allow |
| Explain speculative decoding in two sentences. | 64 | 2 | 191 s | 0 | Speculative decoding is a technique used in machine translation to improve the accuracy of the translated text by allowi |
| Explain speculative decoding in two sentences. | 128 | 1 | 377 s | 0 | Speculative decoding is a technique used in machine translation to improve the accuracy of the translated text by allowi |
| Write a short Python function that checks whether a number is prime. | 32 | 4 | 97 s | 1 | def is_prime(n): if n <= 1: return False if n <= 3: return True if n % 2 == 0 or n % 3 == 0: |
| Write a short Python function that checks whether a number is prime. | 64 | 2 | 194 s | 1 | def is_prime(n): if n <= 1: return False if n <= 3: return True if n % 2 == 0 or n % 3 == 0: |
| Write a short Python function that checks whether a number is prime. | 128 | 1 | 388 s | 0 | Here is a short Python function that checks whether a number is prime: ``` def is_prime(n): if n <= 1: retur |
Evidence: bench/quality/2026-10-03-LLaDA-8B-Instruct.Q4_K_M.json (mojo-dllm bdd268b).
"Repeated words" counts immediate repetitions such as "the the", a rough
proxy for parallel-decoding damage. bench/quality.py records the runs and
scripts/bench_table.py renders this table.
Accuracy of --threshold and --dual-cache
Both options change what a step computes, so their answers can differ from
the plain sampler's. A paired GSM8K run checks how often they still get the
answer right (bench/eval_gsm8k.py, see Development):
LLaDA-8B-Instruct.Q4_K_M.gguf, GSM8K, 200 problems, flags --max-tokens 256 --block-length 32 --steps 256 --device gpu --json, mojo-dllm 3b2dac4.
| Arm | accuracy | vs baseline | 95% CI | discordant | McNemar p | passes / problem | time / problem |
|---|---|---|---|---|---|---|---|
| baseline | 80.5% | 237.0 | 79.7 s | ||||
| dualcache | 81.5% | +1.0 pt | -4.4 to +6.4 pt | 14 / 16 | 0.86 | 236.6 | 15.8 s |
| threshold09 | 80.0% | -0.5 pt | -2.2 to +1.2 pt | 2 / 1 | 1.00 | 73.4 | 24.1 s |
Discordant counts problems only the baseline solved / only the arm solved. The 95% CIs are Wald intervals, which are too narrow when the arms disagree on few problems. Time per problem is the mean over the first 50 problems of each arm. These rounds carry no isolation record, so nothing here shows that the machine was otherwise idle.
Evidence: bench/eval/2026-10-29-baseline-vs-dualcache.json, bench/eval/2026-10-29-baseline-vs-threshold09.json, bench/eval/2026-10-29-baseline-vs-dualcache-first50.json, bench/eval/2026-10-29-baseline-vs-threshold09-first50.json.
Each row of an arm file (bench/eval/runs/ARM.jsonl) has a schema: mojo-cli for eval_gsm8k.py run, engine for
eval_engine.py run. forward_passes is a count the engine measured and is null when it does not count them; an engine
row then carries the configured count in steps, and eval_gsm8k.py compare reads the arm's
flags.forward_passes_measured and prints no mean for an arm that did not measure. stopped_early is true when the
answer-fixed stop rule fired on the step the run ended, and that step can be the last one scheduled: 140 of the 200
baseline rows are stopped_early with all 256 passes, so compare forward_passes with the configured steps to see how
much work a row did. Engine rows also record completion_tokens and finish_reason; length means the token limit cut
the answer off.
The authors' own code (Transformers arm)
bench/eval_engine.py run --engine hf answers the same 200 problems through the sampling code the model authors
publish: generate from generate.py in a checkout of ML-GSAI/LLaDA for
LLaDA, and diffusion_generate from the remote code of
Dream-org/Dream-v0-Instruct-7B for Dream. The adapter
imports the LLaDA file from your checkout and never copies it (the repository has no license file).
This arm runs different weights. The llama.cpp and mojo-dllm arms load the Q4_K_M GGUF. The Transformers arm
loads the authors' bf16 safetensors, or a bitsandbytes 4-bit (nf4) copy of them with --quant 4bit. An accuracy
difference between the arms therefore includes quantization as well as any difference between the
implementations, and eval_gsm8k.py compare refuses such a pair unless you pass --across-weights. The output
then holds both model names and "weights_differ": true.
Like-for-like settings: temperature 0, steps equal to the generation length, block length 32 and low-confidence
remasking for LLaDA, entropy remasking for Dream (which has no blocks). The Dream README example uses temperature
0.2, top_p 0.95 and 512 steps, and the library default algorithm is origin; the arm departs from that, and the
flags of every row say so. version records the model repository's commit sha (--model-revision, a full
40-character sha; null when you give none, and then the weights and the remote code are unpinned), the LLaDA
checkout's commit, and the transformers, torch and bitsandbytes versions. The Python sampling loop is not a
speed baseline; read accuracy from this arm, not seconds.
The two READMEs pin different environments (LLaDA: transformers==4.38.2; Dream: transformers==4.46.2 and
torch==2.5.1), so each model gets its own virtual environment. The repository's frozen environment never
installs them. The runner needs only the standard library besides the model libraries, so run it with the
environment's own interpreter. The LLaDA README pins no torch; I pin 2.5.1 as for Dream, because an unpinned
install resolves a CUDA 13 build that may not match the laptop's driver. Both lines also pin numpy<2; that numpy 1.x
works with these transformers versions is an assumption I did not test. I checked each install line below with uv pip install --dry-run only (it resolves: torch==2.5.1 with the
CUDA 12.4 libraries, numpy==1.26.4, and bitsandbytes==0.50.2 for the GPU lines); nothing was installed and
nothing was run.
# LLaDA, GPU (4-bit). For the CPU path replace the install line with:
# uv pip install --python "$V/bin/python" --extra-index-url https://download.pytorch.org/whl/cpu \
# --index-strategy unsafe-best-match "torch==2.5.1+cpu" transformers==4.38.2 accelerate "numpy<2"
V=~/.cache/mojo-dllm-hf/llada
uv venv $V --python 3.12
uv pip install --python "$V/bin/python" torch==2.5.1 transformers==4.38.2 accelerate bitsandbytes "numpy<2"
git clone https://github.com/ML-GSAI/LLaDA ~/src/LLaDA
REV=$(curl -s https://huggingface.co/api/models/GSAI-ML/LLaDA-8B-Instruct | jq -r .sha)
scripts/capped.sh --mem 26G -- "$V/bin/python" bench/eval_engine.py run --engine hf --model GSAI-ML/LLaDA-8B-Instruct \
--model-revision "$REV" --llada-repo ~/src/LLaDA --quant 4bit --device gpu --arm hf-llada-4bit --limit 50
# Dream, GPU (4-bit); the CPU line is the same with the CPU install line above and transformers==4.46.2
V=~/.cache/mojo-dllm-hf/dream
uv venv $V --python 3.12
uv pip install --python "$V/bin/python" torch==2.5.1 transformers==4.46.2 accelerate bitsandbytes "numpy<2"
REV=$(curl -s https://huggingface.co/api/models/Dream-org/Dream-v0-Instruct-7B | jq -r .sha)
scripts/capped.sh --mem 26G -- "$V/bin/python" bench/eval_engine.py run --engine hf --model Dream-org/Dream-v0-Instruct-7B \
--model-revision "$REV" --quant 4bit --device gpu --arm hf-dream-4bit --limit 50
python3 bench/eval_gsm8k.py compare baseline hf-llada-4bit --across-weights
The repository's code runs on your machine. The adapter loads with trust_remote_code=True, which executes
the Python files of the model repository (for Dream, modeling_dream.py and generation_utils.py) at load. Only a
commit sha says which code that is, so --model-revision is required for a Hugging Face repository, and the run
refuses without it unless you pass --allow-unpinned (the rows then say unpinned). Look up the sha when you start
the arm and keep it: the curl lines above print the current one. A local directory is not on the hub, so it needs
no revision; its path is recorded and the sha is marked unverified, because nothing ties the directory to a
commit. version also records the LLaDA checkout's commit with -dirty when a tracked file differs from it.
Which arms count as different weights. compare --across-weights reads each arm's model. A 4-bit arm records
LLaDA-8B-Instruct|4bit, so it differs from a bf16 arm of the same repository. Two revisions of one repository
record the same model; they differ in version.model_revision, which compare does not read.
Memory. Two figures, both estimates and not measurements. Before it loads, the adapter reads MemAvailable
from /proc/meminfo and refuses to start the load when it is below the load need: bf16 is 1.25 times the weights
plus 3 GB of working memory (LLaDA's float64 Gumbel noise on the CPU is about 0.5 GB per tensor), 4-bit is the
weights over 3.5 plus 3 GB. After the load the isolation preflight asks for the 3 GB of working memory only,
because the weights are resident by then and MemAvailable already excludes them. The weights' size comes from a
table for the two repositories, else from the *.safetensors of a local directory; for any other model the size is
unknown and the run refuses before loading, as it does when /proc/meminfo cannot be read. The guard does not
replace a cap: run the commands above under scripts/capped.sh (see Development). The loader
also passes low_cpu_mem_usage=True.
The weights are 14.93 GiB (LLaDA) and 14.19 GiB (Dream) of safetensors, so one model fits a nearly full disk and both do not. The adapter loads the model before the preflight and the first problem, and refuses a model that is not on the requested device (a model that fell back to the CPU would still answer, slowly).
Not done yet. The generation timeout leaves the generation thread running until the stopped process exits. The tests run against stubs: no weights were downloaded and no GPU was used for this arm. The
memory-fit spike (issue #178, scope item 1: peak RAM, peak VRAM and seconds per step for bf16 on the CPU and for
4-bit on the GPU) and the GSM8K arm itself (scope item 4) have not run, because the laptop had 4 GB of disk free
and has to stay quiet for the timed benchmark runs. Whether 4-bit LLaDA fits 6 GB of VRAM is unmeasured, and so
is the CPU step time. There are no Transformers rows in this document until the arm is committed under
bench/eval/.
A running vLLM server (DiffusionGemma)
bench/eval_engine.py run --engine vllm answers the same 200 problems through the OpenAI-compatible endpoint of a vLLM
server that is already running. vLLM serves one diffusion model, DiffusionGemma (issue #179). The engine is a client:
it never starts or stops a server, loads no weights and installs nothing. Rent the GPU, start the server there,
point the client at it, and destroy the instance after. The shared client is bench/engines/_http.py (standard
library only); the SGLang engine reuses it.
What the client sends: POST /v1/chat/completions with one user message holding the prompt text every engine uses,
temperature 0 and max_tokens equal to --gen-length. The server applies the model's chat template. The client
sends no credentials and no cookies, follows no redirects, never retries, ignores http_proxy and its siblings unless
you pass --use-env-proxy, refuses a response over 4 MiB, and gives each request one deadline (--timeout-s) that
covers connect, headers and body. Text the server sends back (an error message, a model id) is stripped of control
characters and capped at 400 characters before it reaches your terminal, so a server cannot write escape sequences
into it.
This arm runs different weights. The recipe's FP8 checkpoint is not the Q4_K_M GGUF of the llama.cpp arm. The
server does not report its quantization, so --quantization is required and goes into the arm's model (for example
diffusiongemma-26B-A4B-it|fp8-dynamic). eval_gsm8k.py compare then refuses the pair unless you pass
--across-weights, and an accuracy or speed difference includes the quantization.
What a row records. version is JSON with the release GET /version reports ({"version": ...}), and the
checkpoint, quantization and attention backend. Only the release comes from the server. --quantization and
--attention-backend are declared, and flags.declared_not_verified lists them. --checkpoint is compared with the
root that GET /v1/models reports for the served name, and a difference is refused; with no --checkpoint, the
reported root is the checkpoint. --model is the arm's label, and it must equal that root or the served name:
a label for other weights is refused, so pass the checkpoint you served (for the FP8 run,
RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic), not the base model's name. flags.model_matches_server records it. The chat template is the server's: flags.chat_template says
applied_by: server, verified: false, and keeps what you declare with --chat-template. The run is refused
before the first problem when the server does not list --served-model, when /version answers no version, and
for an HTTP error, an empty completion, a response that names another model, or a response without token counts.
The server's address is in neither version nor flags.
Steps. The response carries no denoising steps. vLLM's source defines the counter
vllm:diffusion_num_denoising_steps for diffusion models (vllm/v1/spec_decode/metrics.py, read 2026-11-09). The
source has no _total; prometheus_client adds that suffix when it exposes a counter, so /metrics should show
vllm:diffusion_num_denoising_steps_total. That _total name is inferred from prometheus_client, not seen on a
live /metrics, and the engine accepts both names. The counter is server-wide and sums every request the server
runs. The engine reads it before and after each generation and records the change, which is this request's steps only
when nothing else runs on the server. So the run is refused when the server exposes the counter and you have
not declared --max-num-seqs 1, the value you started the server with (the client cannot read it; the recipe's own
command sets 4). Run one client at a time even then. When /metrics or the counter is missing, forward_passes is
null and flags.forward_passes_measured is false, and --max-num-seqs is not needed; the engine never fills in a
number. Any other error from /metrics (403, 500) stops the run: only a 404 means "no such endpoint". A counter that
does not move across a request also stops it. I have not checked this counter against a running server. Steps per
canvas vary from request to request, because the recipe describes an entropy-bound sampler; the recipe guide's "up to
48 steps per block" is the figure it prints, not an expected value for this server, so read the first rows for their
spread and do not check them against a constant. --steps, --block-length and --schedule do not reach the
server: the step rule is set when the server starts (--diffusion-config), and a --steps different from
--gen-length is refused.
The isolation checks see the client only. bench/isolation.py runs on the machine that runs the client. It does
not observe the server's GPU, clocks, power or neighbours, so a clean preflight says nothing about the rented card.
Every row says "server": "remote" in flags. --device cpu|gpu selects which local checks apply; run the client
with --device cpu on a machine without a GPU. generate_s is the wall time of one request, which includes the
network round trip and the server's queue.
Runbook: rent, serve, run, pull, destroy
TODO for the maintainer: the budget. The issue asks for the price and the hours in a pull request before any renting. They are not in this document. Name the card, the hourly price from the offer you pick, the hours and the total cost in the pull request that adds the arm.
Which card. The recipe lists FP8 as verified on H100, H200 and DGX Spark GB10, with a 32 GB minimum
(recipes/models/Google/diffusiongemma-26B-A4B-it.yaml in vllm-project/recipes, read 2026-11-09), and vLLM 0.24.0 or
newer. An RTX A6000 is not a verified target. The FP8-dynamic checkpoint is 25.33 GiB (Hugging Face API, 2026-11-09).
-
One time on your machine. Install the Vast.ai CLI, read your API key into an environment variable and create a throwaway SSH key. The key prompt reads without echo and the key never appears in a command line; cancel with Ctrl-C and nothing is written.
uv tool install vastai ssh-keygen -t ed25519 -N '' -C vast-dgemma-throwaway -f ~/.ssh/vast_dgemma echo "== Vast.ai API key (console.vast.ai, Account, API Keys) ==" read -rs -p "key: " VAST_API_KEY; echo; export VAST_API_KEY vastai create ssh-key "$(cat ~/.ssh/vast_dgemma.pub)"Run every later
vastaicommand in this same shell, and close it when you are done: the key lives in that shell's environment, and every program you start from it inherits the variable. This runbook does not usevastai set api-key KEYor--api-key KEY, because both put the key in the command's argument list, which every user of the machine can read through/proc/PID/cmdline(mode 0444 on this machine, checked withls -l /proc/self/cmdline) while the command runs. A process's environment file,/proc/PID/environ, is mode 0400 here, readable by its owner and root. The CLI readsVAST_API_KEYwhen no--api-keyis given: I read the default of--api-keyinvast.pyof vast-ai/vast-cli at commit4727235e75470ab26b231c1bdb1415f6ca815d1d(os.getenv("VAST_API_KEY", ...)), and the Context7 documentation of/vast-ai/vast-cligives the same order (explicit argument, thenVAST_API_KEY, then the key file), both on 2026-11-09. I have not run the CLI, and I did not check which versionuv tool install vastaiinstalls. If your version ignores the variable, the first call fails with an authentication error and nothing is stored.This touches
~/.ssh/vast_dgemma*and your account's key list (the CLI also creates~/.config/vastai). To undo it, close the shell, delete~/.ssh/vast_dgemma*and remove the SSH key in the Vast.ai console, and revoke the API key there if you made it for this run. I have not runvastai create ssh-keyhere; check it withvastai create ssh-key --help. -
Find an offer. Filter by card, reliability, disk, download speed and driver. The field names come from the Vast.ai CLI reference (
docs.vast.ai/cli/reference/search-offers); fill in the card name from its listing.vastai search offers 'gpu_name=<CARD> num_gpus=1 reliability>0.98 disk_space>=100 inet_down>500 rented=False' -o 'dph'Pick by the total, not the hourly price: the hourly price times your hours, plus the download (
inet_down_costper GB for about 26 GB of weights and the image). Add--rawto see the cost fields. -
Rent it. The recipe's image is
vllm/vllm-openai:gemma. Billing starts when the instance runs.vastai create instance OFFER_ID --image vllm/vllm-openai:gemma --disk 100 --ssh --direct --label dgemma-vllm vastai show instances # wait for "running"; note the instance ID vastai ssh-url INSTANCE_ID # prints ssh://root@HOST:PORT--disk 100is my estimate (25.33 GiB of weights, the image and the cache), not a measurement. I have not run these lines on Vast.ai, and I do not know whether its--sshmode keeps the image'svllmentry point out of the way. Runvastai create instance --helpfirst; if the server starts by itself, read the instance's logs before you start a second one. -
Serve. On the instance (
ssh -p PORT -i ~/.ssh/vast_dgemma root@HOST), start the server with the command from the recipe page. The pagehttps://recipes.vllm.ai/Google/diffusiongemma-26B-A4B-itwas read on 2026-11-09. It prints thisvllm servecommand for its default configuration, verbatim:vllm serve google/diffusiongemma-26B-A4B-it \ --max-num-seqs 4 \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja \ --reasoning-parser gemma4and this
docker runfor "Single GPU (H100/H200, BF16)", verbatim:docker run -itd --name diffusiongemma \ --ipc=host --network host --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:gemma \ --model google/diffusiongemma-26B-A4B-it \ --max-model-len 262144 \ --max-num-seqs 4 \ --gpu-memory-utilization 0.85 \ --host 0.0.0.0 --port 8000The page prints no command for the FP8 checkpoint. Its deployment-notes table also lists
--generation-config vllm(the checkpoint'sgeneration_config.jsonsetsmax_tokens256 and would override the per-request limit),--gpu-memory-utilization 0.85and--diffusion-config '{"canvas_length": 256}'as flags DiffusionGemma needs, and neither command above has them all. Which exact command to run for this arm is the maintainer's decision. The FP8 recipe variant names the modelRedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic; using that id in place ofgoogle/diffusiongemma-26B-A4B-itis my substitution, not the page's, and is not verified on hardware. To record steps the server must also run with--max-num-seqs 1(the recipe's value is 4; see Steps above). Write the command you ran, as run, into the pull request. Without Docker on the instance (Vast.ai runs the image as the container), take the arguments after the image name and run them asvllm serve MODEL .... If you pass--served-model-name, give that name to--served-modelbelow. Thedocker runblock binds--host 0.0.0.0, and a vLLM server started without--api-keyhas no authentication. On a rented host that port may be reachable by anyone. Replace it with--host 127.0.0.1(the tunnel in step 5 reaches it) or close the port with the host's firewall, and never leave an unauthenticated server exposed. The recipe's--reasoning-parser gemma4and the Gemma 4 chat template can put the answer in the response'sreasoning_contentand leavemessage.contentempty. The engine then stops withthe server returned an empty completion. To see it,curlone chat request by hand against the tunnel and look atchoices[0].message. What to change on the server is a question for the recipe: check the recipe page for how it turns thinking on and off, and do not guess a flag. The--generation-config vllmline in the page's deployment notes is mentioned above: the checkpoint'sgeneration_config.jsonsetsmax_tokens256, which would override the per-request limit. -
Run the arm from your machine. Open the tunnel in a second terminal, check that the server answers, then run. Results are written on your machine, to
bench/eval/runs/ARM.jsonl, so nothing needs pulling off the instance. The client holds no key, and the instance never receives one.ssh -N -L 8000:127.0.0.1:8000 -p PORT -i ~/.ssh/vast_dgemma root@HOST # terminal 2; Ctrl-C closes it curl -s http://127.0.0.1:8000/version # {"version":"0.24.0"} or newer curl -s http://127.0.0.1:8000/v1/models # note the id and root python3 bench/eval_engine.py run --engine vllm --model RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic \ --arm vllm-dgemma-fp8 --device cpu --gen-length 256 --max-num-seqs 1 \ --base-url http://127.0.0.1:8000 --served-model SERVED_NAME \ --quantization fp8-dynamic --checkpoint CHECKPOINT_AS_SERVED --attention-backend ATTENTION_BACKEND_AS_SERVEDAdd
--limit 3first to look at the rows. Run one client at a time: the step counter is server-wide. After the run,ss -ltnp 'sport = :8000'must show no tunnel of yours. -
Compare.
python3 bench/eval_gsm8k.py compare MOJO_OR_LLAMACPP_ARM vllm-dgemma-fp8 --across-weights. -
Destroy the instance. A stopped instance keeps billing for its disk.
vastai destroy instance INSTANCE_ID vastai show instances # must list nothingThen read the billing page in the console: the final charge should be within the total you computed in step 2. Delete the throwaway SSH key from the account, and remove
~/.ssh/vast_dgemma*.
| Symptom | Likely cause | Action |
|---|---|---|
the run refuses with the server reports the models ... | --served-model differs from --served-model-name | pass the id /v1/models lists |
--checkpoint is ... but the server reports root ... | the label does not match what was served | fix the label, or the command |
forward_passes is null | the server exposes no vllm:diffusion_num_denoising_steps counter | none: the column stays null |
the run refuses with --max-num-seqs 1 | the server exposes the counter and you did not declare 1 | restart the server with --max-num-seqs 1, declare it |
| the run stops on a counter that did not move | another client, or /metrics is not this model's | stop other clients; read /metrics by hand |
--model ... is neither the root ... nor the id | the label names other weights than the server runs | pass the served checkpoint as --model |
returned an empty completion | the answer is in reasoning_content | curl one request; check the recipe |
answered HTTP 4xx | for example max_tokens over the server's limit | the message is the server's own |
| out of memory on the server | --max-num-seqs above 4 | the recipe sets 4 because diffusion state buffers are large (1 uses less) |
Not done yet. No server was run for this engine: the tests use a fake server on 127.0.0.1, and no vLLM, GPU or
weights were involved. The fields above were read from vLLM's source and the recipe page, not observed on a live
server (/version, /v1/models, /metrics, the counter's per-request meaning). The evaluation (issue #179, scope
item 6: batch size 1, the same 200 problems, a 256-token canvas, on a rented card) has not run, so there are no vLLM
rows in this document. If the harness's number differs from vLLM's published figures by more than 25 percent on the
same card class, explain it here (prompt length, canvas, checkpoint) when the arm lands.
A running SGLang server (DiffusionGemma)
bench/eval_engine.py run --engine sglang answers the same 200 problems through the OpenAI-compatible endpoint of an
SGLang server that is already running (issue #180). SGLang serves DiffusionGemma with its dLLM framework and the
Gemma4Renoise algorithm, and its cookbook says the model is "Not benchmarked for speed". The engine is a client in
the same way as the vLLM one: it never starts or stops a server, loads no weights and installs nothing. It uses
the shared bench/engines/_http.py (no credentials, no cookies, no redirects, no retries, one deadline per request,
a 4 MiB bound, server text stripped of control characters and capped at 400 characters; see the vLLM section above)
and bench/engines/_openai.py (the model-list and completion checks the two engines share).
Every field name below comes from SGLang's source, read on 2026-11-09 at main commit
dfaca5681ddc20e31d50f2dfe5dfada269f12bbd, plus the cookbook page
https://docs.sglang.io/cookbook/autoregressive/Google/DiffusionGemma.md read the same day. The routes /v1/models,
/model_info and /server_info and the Gemma4Renoise algorithm were also compared with the release tag v0.5.23
(2026-11-02) and give the same field names and the same request rules; the other files were read at main only. The issue text and
the source differ in two places, and the engine follows the source: SGLang has no GET /version (the version is a
field of GET /server_info), and the issue's pull request #23722 for sequence-level forward counts was closed
without a merge. No field was seen on a live server.
What the client sends. POST /v1/chat/completions with four fields: model, one user message holding the prompt
text every engine uses, max_completion_tokens equal to --gen-length, and stream false (the cookbook says
streaming is block-level, one denoised canvas per chunk, and the tokenizer manager's _wait_one_response yields an
unfinished output only for a streaming request). Gemma4Renoise.validate_request returns an error for a request that
carries logprobs, a grammar (json_schema, regex, ebnf, structural_tag), a penalty, min_new_tokens, min_p, a
seed, a logit bias, a custom logit processor or a sampling mask, and the cookbook says such a request is rejected with
a 400. The engine sends none of them, and a test scans every request body for them.
temperature, top_p and top_k are accepted and have no effect, because the renoise schedule governs sampling;
the engine sends none of them either, and flags.sampling says so. A rejected request stops the run with the
server's own message. The server applies the chat template; flags.chat_template says applied_by: server,
verified: false, and holds the chat_template that GET /server_info reports (null means the checkpoint's own
template) and what you declare with --chat-template.
SGLang's own measurement, benchmark/dllm/bench_diffusion_gemma.py, sends pretokenized prompts to /v1/completions
with EOS stopping off and a fixed output length, and its README says the token rate it reports includes canvas
positions after EOS. This engine sends chat messages and lets the model stop, and it times a whole answer. The two
numbers are not the same measurement and should not be compared.
This arm runs bf16 weights. Gemma4Renoise.configure_server_args refuses a quantized checkpoint ("DiffusionGemma
does not support quantized loading"), and the engine refuses a server whose /server_info reports a quantization.
The arm's model is <name>|<dtype>, for example diffusiongemma-26B-A4B-it|bfloat16, so eval_gsm8k.py compare
refuses the pair with a Q4_K_M GGUF arm or the vLLM FP8 arm unless you pass --across-weights, and a difference in
accuracy or speed includes the precision.
What a row records, and where each value comes from. version is JSON with the SGLang release, the diffusion
algorithm, the checkpoint and the dtype.
| Value | Read from | If the server cannot say |
|---|---|---|
release (version) | GET /server_info, field version | the run is refused; so is an empty string and the development fallback 0.0.0.dev0, which names no release or commit |
| checkpoint | GET /model_info, field model_path | --checkpoint is recorded as declared and the label stays unconfirmed |
| algorithm | GET /server_info, field dllm_algorithm | --algorithm is needed; a null field means the server runs no diffusion model and the run is refused |
| dtype | GET /server_info, field dtype | auto and a missing field mean "cannot say": --dtype is needed (half and float16 are one dtype, as are float and float32) |
| batch size | GET /server_info, field max_running_requests | null means "cannot say": --max-running-requests 1 is needed |
A value the server reports is the value. A declared value that differs from a reported one is refused, and a declared
value the server could not confirm is listed in flags.declared_not_verified. --model must equal the reported
model_path or the served name (SGLang's /v1/models puts the served name in both id and root, so unlike vLLM
root cannot confirm the checkpoint). flags.model_matches_server is true when --model is the reported model_path,
false when it matches only the served name (an alias, so the label may name other weights than the checkpoint recorded
in flags.checkpoint_reported_by_server), and null when the server reports no model_path. The server's own text is never
copied beyond these named fields: /server_info also holds a launch_command and the scheduler's internal state, and
neither reaches an arm file. flags.algorithm_config_set says whether the server was started with a
--dllm-algorithm-config file (its content is not readable from the client). The server's address is in neither
version nor flags.
Batch size 1 is required. generate_s is the wall time of one request, and a second client would add its queue time
to it, so the run is refused unless the batch size is 1, reported by the server or declared with
--max-running-requests 1. SGLang's own LLaDA 2.1 cookbook serves with --max-running-requests 1
(https://docs.sglang.io/cookbook/autoregressive/InclusionAI/LLaDA-2.1.md, read 2026-11-09). Run one client at a time.
Steps are not recorded. SGLang exposes no per-request step or forward-pass count on main or in v0.5.23. Its
Prometheus metrics (python/sglang/srt/observability/metrics_collector.py) have no dLLM counter, and the only pass
counter, sglang:cuda_graph_passes_total, is server-wide and counts passes by CUDA-graph mode, so a change of it across
one request is not that request's denoising steps. Pull request sgl-project/sglang#48675 ("Expose cumulative
request-level forward metrics", which would add meta_info.dllm_metrics to /generate) was open and unmerged when I
read it on 2026-11-09, and its description opens with "dLLM responses do not report how many model forwards a request
uses". SGLang's own benchmark records --denoising-steps as metadata only. I read the metrics collector at both main
and v0.5.23, and neither has a dLLM metric. So forward_passes is
null in every row, flags.forward_passes_measured is false, and the engine never reads /metrics and never infers a
count. --steps, --block-length and --schedule do not reach the server (--dllm-algorithm-config sets the step
rule when the server starts), and a --steps different from --gen-length is refused.
The isolation checks see the client only. bench/isolation.py runs on the machine that runs the client, and
every row says "server": "remote" in flags. Run the client with --device cpu on a machine without a GPU.
Runbook: serve, run, destroy
TODO for the maintainer: the budget. The issue asks for the price and the hours in a pull request before any renting. They are not in this document, and I name no card and no price. State the card (or cards), the hourly price from the offer you pick, the hours and the total in the pull request that adds the arm.
Which hardware. The bf16 checkpoint google/diffusiongemma-26B-A4B-it has 51 647 701 024 bytes of .safetensors
files, 48.1 GiB (Hugging Face API, revision f7f5b7f5fa82ffc52addd066915886d497f5517b, read 2026-11-09). That is more
than a 48 GB card holds before the cache is allocated, so a 48 GB card cannot run this arm alone; the issue's reading
is two cards or tensor parallelism. The cookbook prints no memory or parallelism figure and no command with tensor
parallelism. ServerArgs has --tp-size (alias --tensor-parallel-size), and I have not run it with this model.
Whether one 80 GB card holds the weights and the cache is not settled by any page I read. Settle the hardware with a
short trial of a few problems before the full run. SGLang's roadmap issue #39499 lists quantized checkpoints
(FP8 first) for DiffusionGemma as an unchecked item, and the algorithm refuses quantized loading today, so plan for
bf16.
-
Rent the machine. Steps 1 to 3 of the vLLM runbook above apply (Vast.ai CLI, key, offer, instance), with these changes. The install guide (
https://docs.sglang.io/docs/get-started/install.md, read 2026-11-09) publishes the Docker imagelmsysorg/sglangon Docker Hub, sayslatestis the newest stable release, and says to use a specific release or dated nightly tag for a reproducible deployment. I did not check which tag matchesv0.5.23, and I have not run the image on Vast.ai. Size the disk from the 48.1 GiB of weights plus the image; I have no measurement of the image. Billing starts when the instance runs. -
Serve. On the instance, start the server with the command from the cookbook. The page
https://docs.sglang.io/cookbook/autoregressive/Google/DiffusionGemma.mdwas read on 2026-11-09. Section 3.1 prints this command, verbatim (the issue quotes the same flags on one line):sglang serve \ --model-path google/diffusiongemma-26B-A4B-it \ --dllm-algorithm Gemma4Renoise \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000I compared this block with
curl -sof the page's markdown source on 2026-11-09 and the bytes are identical. The page says the Triton attention backend, eager mode and unchunked prefill are selected automatically forGemma4Renoise, so none is passed. Three changes to it are mine and are not on the page. Add--max-running-requests 1, so the batch size is 1 and the server reports it. Replace--host 0.0.0.0with--host 127.0.0.1: the page's command binds every interface, and a server started without--api-keyhas no authentication, so on a rented host that port may be reachable by anyone; the tunnel in step 3 reaches a loopback bind. Do not add--api-key, because the client sends no credential and every request would stop with HTTP 401. Optionally add--dtype bfloat16: without it/server_inforeportsauto, and you declare--dtype bfloat16in step 4 (the argument's help text saysautouses BF16 for a BF16 model), which the row then lists as declared and not verified. Write the command you ran, as run, into the pull request.Leave
--dllm-algorithm-configunset for the reference arm, so the checkpoint's defaults apply (the cookbook lists 48 denoising steps, an entropy bound of 0.1 and temperatures from 0.4 to 0.8). The cookbook's example config marksseedas optional and as what makes the sampling reproducible, so without one two runs of the same problem can answer differently. The page's command has no--reasoning-parser; if you add one, the answer may move toreasoning_contentand the engine then stops withthe server returned an empty completion. -
Run the arm from your machine. Open the tunnel in a second terminal and look at what the server reports. Results are written on your machine to
bench/eval/runs/ARM.jsonl, and the instance never receives a key.ssh -N -L 30000:127.0.0.1:30000 -p PORT -i ~/.ssh/vast_dgemma root@HOST # terminal 2; Ctrl-C closes it curl -s http://127.0.0.1:30000/v1/models # note the id curl -s http://127.0.0.1:30000/model_info | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("model_path"), d.get("served_model_name"))' curl -s http://127.0.0.1:30000/server_info | python3 -c 'import json,sys; d=json.load(sys.stdin); print({k: d.get(k) for k in ("version", "dllm_algorithm", "dtype", "quantization", "max_running_requests", "chat_template")})' -
Run.
--modelis the checkpoint you served.python3 bench/eval_engine.py run --engine sglang --model google/diffusiongemma-26B-A4B-it \ --arm sglang-dgemma-bf16 --device cpu --gen-length 256 \ --base-url http://127.0.0.1:30000 --dtype bfloat16Add
--limit 3first to look at the rows. Add--served-model NAMEwhen the server uses--served-model-name. After the run,ss -ltnp 'sport = :30000'must show no tunnel of yours. -
Compare.
python3 bench/eval_gsm8k.py compare MOJO_OR_LLAMACPP_ARM sglang-dgemma-bf16 --across-weights. -
Destroy the instance. Step 7 of the vLLM runbook above:
vastai destroy instance INSTANCE_ID, thenvastai show instancesmust list nothing, then read the billing page.
| Symptom | Likely cause | Action |
|---|---|---|
the run refuses with the server reports the models ... | the served name differs from --model | pass --served-model with the id /v1/models lists |
GET /server_info ... HTTP 401 | the server was started with --api-key | restart it without one and keep the loopback bind |
GET /server_info did not report a version string or 0.0.0.dev0 | a source install without version metadata | run a release, or a checkout with its git history |
the server reports no diffusion algorithm | the server runs an autoregressive model, or no --dllm-algorithm | serve with the cookbook command |
the server does not report its dtype | /server_info says auto | pass --dtype bfloat16 (declared), or serve with --dtype bfloat16 |
max_running_requests refusal | the server was not started with --max-running-requests 1 | restart it with the flag |
--model ... is neither the model path ... | the label names other weights than the server runs | pass the served checkpoint as --model |
answered HTTP 400 ... does not support request-level ... | the server rejected a request field | the message names the field; report it, the engine sends none of them |
returned an empty completion | the answer is in reasoning_content, or the model produced nothing | curl one chat request by hand and look at choices[0].message |
| out of memory on the server | the weights plus the cache do not fit the card | see Which hardware |
Not done yet. No server was run for this engine: the tests use a fake server on 127.0.0.1, and no SGLang, GPU or
weights were involved. The fields above were read from SGLang's source and cookbook, not observed on a live server
(/model_info, /server_info, the dtype and max_running_requests values a real launch reports, the
usage.completion_tokens of a dLLM answer, the finish_reason). The evaluation (issue #180, scope item 5: batch size
1, the same 200 problems, a 256-token canvas, on a rented machine) has not run, so there are no SGLang rows in this
document.
DiffusionGemma across engines
The four engines that can answer DiffusionGemma are meant to run the same 200 problems
(bench/eval/gsm8k-200.jsonl), with the same prompt, the same scoring and the same arm-file format, so
eval_gsm8k.py compare --across-weights pairs any two of them. This table is a skeleton and holds no measured value.
A row gets a number only from a committed arm file under bench/eval/runs/; until then it says no committed run yet.
Weights differ between rows, so every comparison includes the precision as well as the engine.
| Arm | Weights (as the arm's model records them) | Algorithm | How it runs | Committed run |
|---|---|---|---|---|
| llama.cpp reference (the branch of ggml-org/llama.cpp#24423) | diffusiongemma-26B-A4B-it-Q4_K_M.gguf, the file docs/correctness.md names | that branch's own loop; I did not read it for this table | the build is scripts/build_dg_ref.sh; the GSM8K runner for llama.cpp's diffusion CLI is issue #177 (open), and I have not checked that bench/engines/llamacpp.py drives this build | no committed run yet |
| vLLM | diffusiongemma-26B-A4B-it|fp8-dynamic, or the bf16 checkpoint (the choice is the maintainer's, see the vLLM runbook) | vLLM's entropy-bound sampler, as the recipe page describes it | --engine vllm (issue #179), runbook above | no committed run yet |
| SGLang | diffusiongemma-26B-A4B-it|bfloat16 | Gemma4Renoise | --engine sglang (issue #180), runbook above | no committed run yet |
| mojo-dllm | the Q4_K_M GGUF of the llama.cpp row | its own entropy-bound loop, docs/diffusion.md ("DiffusionGemma's loop") | mojo-dllm run does not run DiffusionGemma yet: wiring the model into the generation loop is issue #277 (open), see docs/introduction.md | no committed run yet |
The six defaults in docs/cli.md (--eb-steps 48, --eb-t-min 0.4, --eb-t-max 0.8, --eb-bound 0.1,
--eb-stability 1, --eb-confidence 0.005) are the values the SGLang cookbook lists for Gemma4Renoise
(max_denoising_steps, temperature_schedule, entropy_bound, stability_threshold, confidence_threshold; read
2026-11-09). I did not check that the two loops agree step for step. When the arms exist, name each one's commit,
server release and algorithm settings next to its row, from the arm's own version and flags.
Limits of this evidence
- One machine: a 13th Gen Intel i5-13420H laptop for the CPU tables, with an RTX 3050 6 GB laptop GPU for the GPU tables. Another CPU, memory speed or GPU can change the ratios.
- The four main tables use the median of 3 runs on each of 3 prompts (9 runs); the forward-pass tables use 2 or 3 runs on one prompt; the quality table is a single run per cell. Each table has one model file and one quantization (Q4_K_M).
- The schedules differ between runtimes (see above), so tokens / s compares runtimes at one step count, not at one output quality.
- Each table names its commit in a
VersionsorEvidenceline, except the two forward-pass tables, whose commits are in their evidence files. They differ, because the evidence was recorded as the work went on; no single commit holds all of it, and the release tag is not one of them. - The quality column counts immediate repeated words on two prompts. It is not
a perplexity or a task score. The only task evidence is the GSM8K run for
--thresholdand--dual-cacheon LLaDA. - diffuse-cpp loads its own conversion of the model, so its rows are indicative.
- No binary is published yet (#105): you build from source.
Reading the numbers
- ms / step is the cost of one denoising pass over the canvas.
- tokens / s is 128 divided by generation time. A diffusion model's tokens/s depends on the step count you choose, so compare ms / step first.
- The ablation row shows what computing logits only for candidate positions saves.
- On the GPU, read ms / step, the step-time ratio line under each GPU table and peak RSS together. The ratio pairs the two runtimes run by run, and one GPU with one quantization (Q4_K_M) stands behind it. mojo-dllm also holds far less in host memory, because it uploads the weights once and releases the file's pages behind it.
- The thread sweep and the forward-pass-alone tables test the CPU comparison. If llama.cpp ran faster below 12 threads, or mojo-dllm's lead came only from skipping logits, those tables would show it. The sweep stops at 12 because the test CPU runs 12 threads.
- A note under a table counts the runs that started on a busy CPU and gives the medians over the quiet runs alone.
Diffusion decoding in five minutes
An autoregressive model such as Llama writes left to right. Each forward pass produces one token, and an append-only KV cache makes the next pass cheap.
A masked diffusion model such as LLaDA writes the way you fill in a crossword.
It starts from a canvas of [MASK] tokens after the prompt and, at every step,
looks at the whole canvas at once, proposes a token for every blank, and
commits the few it is most confident about. The rest stay masked for the next
step.
prompt: "Explain speculative decoding" canvas (8 positions shown)
step 0 [M] [M] [M] [M] [M] [M] [M] [M]
step 1 [M] [M] [M] [M] [M] [M] [M] .
step 2 A [M] [M] [M] [M] [M] [M] .
step 3 A draft [M] [M] [M] [M] tokens .
...
step 8 A draft model guesses several tokens .
Each step runs the full transformer over prompt plus canvas. Attention is bidirectional: position 3 can read position 7, which a causal model forbids.
The loop LLaDA uses
mojo-dllm implements the reference sampler from the LLaDA repository
(generate.py), with its defaults:
- Split the generated region into blocks of
--block-lengthpositions and fill them left to right (semi-autoregressive decoding). - Give each block
steps / blockssteps and spread its masked positions over them evenly: with 32 masks and 8 steps, 4 tokens per step. - At each step, for every masked position in the block, take the argmax token and its softmax probability.
- Commit the k positions with the highest probability ("low-confidence remasking": the uncertain ones stay masked).
At temperature 0 the loop is deterministic. Above 0, LLaDA adds Gumbel noise to the logits before the argmax; mojo-dllm does the same.
Why the runtime looks different
- No KV cache by default. Any position can change between steps, so every
step recomputes keys and values for the whole canvas.
--dual-cache(LLaDA only) keeps a block-level cache and gives up exactness; see Command line. A step costs a full prompt pass, about 2 × 7 billion operations per token on the canvas in LLaDA-8B, plus 2 × 0.5 billion for each row whose logits are computed. - GEMMs, not GEMVs. A step multiplies weight matrices by a hundred or more activation rows at once. The CPU kernel is built for that shape; see The quantized GEMM.
- Few rows need logits. Only the masked positions the sampler can commit are candidates (the current block for LLaDA, every masked position for Dream), so mojo-dllm computes the output projection (126 464 columns for LLaDA, 152 064 for Dream) for those rows alone. The LLaDA benchmark ablation measures what that saves.
DiffusionGemma's loop
DiffusionGemma does not mask. Its canvas starts as 256 random token ids over the whole vocabulary and every step re-scores all of them:
- Take the temperature
t = t_min + (t_max - t_min) * c / S, withccounting down fromS(48) to 1: 0.8 on the first step, 0.4 + 0.4/48 on the last. - For each canvas position, find the argmax of its logits, the entropy of
softmax(logits / t)and one token drawn from that softmax. mojo-dllm reads each 262 144-wide row once for the argmax and the entropy. - Sort positions by entropy, lowest first, and accept each one while the entropies of the positions before it sum to at most 0.1. The lowest-entropy position is always accepted.
- Accepted positions keep their drawn token and every other position gets a fresh random id.
- The answer is the argmax canvas. The run stops once that canvas has held for one step and its mean entropy is below 0.005, or after 48 steps.
The six numbers are --eb-steps, --eb-t-min, --eb-t-max, --eb-bound,
--eb-stability and --eb-confidence (see Command line). A run
takes as many forward passes as steps, so the sampler decides the cost.
Architecture
mojo-dllm run
|
+-------------+--------------+
| | |
GGUF reader tokenizer diffusion loop
(mmap) (BPE) (diffusion/sampler.mojo)
| |
| model forward
| (models/transformer.mojo)
| |
+----> repacked weights ---> quantized GEMM, RMSNorm, RoPE,
(kernels/packed.mojo) attention, SwiGLU (kernels/)
Source map
| Path | What it does |
|---|---|
src/main.mojo, src/cli/ | command line: run, tokenize, inspect, logits |
src/mojo_dllm/formats/gguf.mojo | GGUF v2/v3 reader over a memory map, bounds-checked |
src/mojo_dllm/formats/utf8.mojo | the UTF-8 validity check the GGUF reader and the tokenizer share |
src/mojo_dllm/quant/kquants.mojo | scalar F32 / F16 / BF16 / Q5_0 / Q8_0 / Q4_K / Q5_K / Q6_K decoders (the test oracle, and embedding rows) |
src/mojo_dllm/kernels/packed.mojo | weights repacked 8 rows per SIMD register; the GEMM used for every projection |
src/mojo_dllm/kernels/qgemm.mojo | activation quantization (int8, one scale per 256) and a row-major reference GEMM |
src/mojo_dllm/kernels/ops.mojo | RMSNorm, NORM and NEOX RoPE, bidirectional attention, SwiGLU |
src/mojo_dllm/kernels/moe.mojo | DiffusionGemma's routing and grouped experts on the CPU |
src/mojo_dllm/kernels/softembed.mojo | DiffusionGemma's self-conditioning: the softmax of the previous logits times the embedding table |
src/mojo_dllm/models/transformer.mojo | config from GGUF metadata, weight loading, the CPU forward pass, and the DenoisingModel trait the samplers drive |
src/mojo_dllm/models/diffusion_gemma.mojo | DiffusionGemma's CPU forward pass: sliding and global layers, the region masks and scalars, the unified and the cached path, self-conditioning (logits drives it) |
src/mojo_dllm/gpu/kernels.mojo | GPU kernels: int8 tensor-core GEMM over Q4_K / Q6_K blocks, flash-style attention, RMSNorm, RoPE, SwiGLU |
src/mojo_dllm/gpu/model.mojo | the forward pass on the GPU: weights uploaded once, activations kept on the device |
src/mojo_dllm/tokenizer/bpe.mojo | byte-level BPE with the bailingmoe and qwen2 pre-tokenizers, and the Tokenizer that picks it or gemma4 from tokenizer.ggml.model |
src/mojo_dllm/tokenizer/vocab.mojo | checks on a token table read from a GGUF file, shared by both tokenizers |
src/mojo_dllm/tokenizer/unicode_tables.mojo | the code point classes of the pre-tokenizer, generated by scripts/gen_unicode_tables.py |
src/mojo_dllm/tokenizer/gemma4.mojo | DiffusionGemma's tokenizer (U+2581 spaces, newline-run split, merges by rank, <0xNN> byte fallback) and its chat template |
src/mojo_dllm/diffusion/sampler.mojo | the denoising loops of LLaDA, Dream and DiffusionGemma (its entropy-bounded loop) and the PRNG |
src/mojo_dllm/sys/ | aligned buffers (mem.mojo), mmap, a dynamically scheduled parallel-for |
A forward pass
For a canvas of L tokens and a list of positions whose logits the caller needs:
- Embedding rows are decoded from the memory-mapped embedding table (Q4_K in a Q4_K_M file), one row per token.
- For each layer (32 in LLaDA-8B): RMSNorm, then the activations are quantized to int8 once and reused by the Q, K and V projections. RoPE rotates Q and K; attention runs over all L positions with no mask; the output projection and residual follow. The FFN quantizes once for gate and up, applies SiLU, quantizes again for down.
- In the last layer, only the requested positions continue past attention.
- Final RMSNorm and the output projection run on the requested rows only.
The forward pass allocates nothing: every activation buffer is sized once for the longest canvas the run will use.
Memory
The GGUF file is memory-mapped. At load, every projection matrix is repacked into a heap block (see The quantized GEMM) and its mapped pages are released, so the model is resident once. Q4_K stays 4-bit after repacking. Q6_K expands to one byte per weight, which costs about 0.4 GB on LLaDA-8B and buys a simpler inner loop. The embedding table stays mapped and faults in as needed. Norm vectors and biases are decoded into f32 arrays at load.
Threads
sys/parallel.mojo hands out work items from a shared atomic counter. On a
hybrid CPU (performance and efficiency cores) a static split makes every
parallel region wait for the slowest core; claiming items one at a time lets
the fast cores take more. On the development laptop the static split left
every GEMM waiting on the four efficiency cores.
The quantized GEMM
Most of a denoising step is matrix multiplication: seven projections per
layer, 32 layers in LLaDA-8B, every step (without --dual-cache), over the
whole canvas. run --verbose prints the split for your machine. This page explains the
kernel that does it.
The shape of the problem
A step multiplies each weight matrix W (N rows × K columns, 4 to 8 bits per weight) by an activation matrix X (L rows × K). L is the canvas length, a few hundred at most. That is a skinny GEMM, and it is compute-bound: each weight is read once per step and used L times.
Activations: int8 with a scale per 256
For a Q4_K or Q6_K projection, each activation row is quantized to int8 with one f32 scale per 256 values, and the sum of each group of 16 values is kept as a float. This is llama.cpp's Q8_K scheme, which keeps the comparison with it fair. Integer dot products are 4× denser than f32 FMAs on AVX2.
Weights: 8 rows per register
GGUF stores each row contiguously. A row-at-a-time dot product needs a horizontal sum per output, and the block scales arrive one scalar at a time. So the loader repacks every matrix once:
for each group of 8 rows, for each 256-weight super-block:
Q4_K: d[8], dmin[8] as f32 | scales[8][8], mins[8][8] as u8 | 1024 bytes of nibbles
Q6_K: d[8] as f32 | scales[16][8] as i8 | 2048 bytes, one value per byte
Inside the weight bytes, a 32-byte load holds 4 consecutive weights of each of
the 8 rows. One vpdpbusd (AVX-VNNI) multiplies those 32 unsigned weights by
4 activation bytes broadcast to every lane and adds into 8 int32
accumulators: 8 rows × 4 multiply-adds in one instruction. The scale of a
sub-block is an 8-lane vector, one entry per row, applied once per 16 or 32
weights.
The zero points do not touch the inner loop. Q4_K computes
w = d·sc·q − dmin·m, so the second term becomes dmin·m times the
activation sum of the sub-block, which the quantizer already stored. Q6_K
stores q + 32, and the 32 comes off the same way.
Tiling and threads
A work item is 16 output rows. It walks the tokens 8 at a time for Q4_K and 4 at a time for Q6_K, keeping that many int32 and f32 accumulators in registers, so each loaded weight vector feeds 8 or 4 tokens. Items go to threads through an atomic counter (see Architecture).
Checks
tests/test_quant.mojo compares the decoders with the gguf Python package's
dequantization, and both GEMMs with an f64 reference over the same dequantized
weights. Swapping the nibble halves or dropping the Q6_K zero point fails
these tests; that mutation check runs before changes to this file land.
bench/bench_qgemm.mojo measures the kernel alone on LLaDA's layer shapes.
Q5_0 and Q8_0 CPU matrices
PackedMatrix and pgemm also accept Q5_0 and Q8_0 weights. Their blocks
contain 32 weights and one f16 scale, so their activations use
QActs(tokens, K, block_size=32). This supports widths such as 704 and 2112
that do not fit the K-quant activation layout. The existing default remains
256, and pgemm checks that the activation block size matches the matrix.
PackedMatrix(src_addr, src_bytes, kind, N, K) takes the size of its source
as well as the address, and raises unless src_bytes equals the bytes of N
rows of K weights. The tensor loaders pass the byte count the GGUF reader
checked against the file, so a shape that disagrees with the tensor is an error
and not a read past it.
Each packed block holds eight f32 weight scales and interleaved bytes for
eight rows. Q5_0 uses unsigned values with zero point 16; Q8_0 uses zero
point 128. The activation sums remove those offsets after the integer dot
product. On AVX2, the Q8_0 path splits the high bit before pmaddubsw so
full-range unsigned weights cannot saturate its signed 16-bit intermediate.
VNNI uses the full-byte dot product directly.
tests/test_packed32.mojo compares both formats at the model's two widths
with scalar products of independently dequantized weights. It also covers
ragged token tiles, output strides, scale changes between blocks and extreme
Q8_0 values, including a build that explicitly targets AVX2 without VNNI.
Routing and grouped CPU experts
kernels/moe.mojo provides the layer primitives for DiffusionGemma's CPU
forward-pass integration. route_tokens applies unweighted RMS normalization,
the inverse square root of the hidden width, the router scale vector and an
F32 projection. It computes softmax, selects the highest-weight experts with
lower expert IDs winning ties, and renormalizes the selected weights.
ExpertWeights repacks each expert's slice once, after checking that the byte
counts of the fused gate/up tensor and the down tensor match the expert count
and dimensions. MoEWorkspace stores
reusable buffers large enough for every token to select the same expert.
grouped_experts counts and groups token/expert pairs, skips empty experts,
and runs batched gate/up and down projections for each active expert. Between
the projections it applies gelu_tanh(gate) * up; the scatter-add multiplies
each output by the expert scale and the token's router weight.
The tiny MoE tests compare routing and final outputs with a separate scalar reference, including mixed K-quant gate/up and 32-weight-block down matrices. The MoE benchmark measures a complete synthetic layer at the intended model dimensions and retains its raw timing samples.
The GPU path
src/mojo_dllm/gpu/kernels.mojo does the same arithmetic on an NVIDIA GPU,
with a layout chosen for tensor cores instead of AVX registers.
Weights stay in GGUF blocks. Upload copies each Q4_K matrix as it is and pads each 210-byte Q6_K block to 224 bytes, so every 16-byte chunk of a block is aligned and the kernels can load 16 bytes at a time.
Activations get one scale per 32. The GPU quantizes activations to int8
with one f32 scale per 32 values (llama.cpp's Q8_1 shape) and keeps
scale x sum(q) next to it. One warp quantizes one 32-value block, a value
per lane, and reduces its max and integer sum with shuffles, so loads and
stores are coalesced. A 32-weight Q4_K sub-block then meets exactly one
activation scale:
sum w x = d*sc*dx * sum(q * qx) - dmin*m * (dx * sum(qx))
One mma per scale group. A Q6_K thread block computes 64 weight rows by a
tile of 32, 64 or 80 tokens. For each 256-wide super-block it copies the
weights into shared memory (see the Q6_K paragraph below), copies the matching
activations next to them, and each of its 4 warps runs one mma.m16n8k16.s8
per 16-weight scale group for its 16 rows and every 8 tokens. (A Q4_K block
scales every 32 weights, so it runs mma.m16n8k32.s8 per sub-block instead.)
The int32 result goes to f32 with the two scales and, for Q4_K, the min term.
Q4_K: packed in shared memory, unpacked in registers. Removing the staging
work cut the GEMM roughly in half
(bench/checks/2026-10-13-gemm-cost-split.json: with staging done once, the
GEMM took about 100 ms instead of about 220), while removing the mma saved
only a few milliseconds. So a Q4_K block copies each row's 144-byte
super-block as it is, in 16-byte loads: the 16-byte header (d, dmin and the
packed scales) to one array, the 128 bytes of nibbles to another. A warp
unpacks its A fragments in registers: word i of nibble pair p holds sub-block
2p in its low nibbles and 2p + 1 in its high ones at the same position, so one
shared load feeds two mmas, and the 6-bit scales are decoded from the header
per sub-block. With the weights packed, a block holds 128 rows and runs 8
warps in the same 48 KiB, which halves the activation copies per output. The
arithmetic is unchanged, so the logits are too
(bench/checks/2026-10-13-gemm3-logits.json).
Prefetching the next super-block. On the kernel before the packed layout,
removing staging saved about 120 ms of the GEMM and removing the mma 4 to
7 ms, both measured against the integer-accumulation variant
(bench/checks/2026-10-13-gemm-cost-split.json). The packed layout (128-row
blocks) took that harness from 258.2 to 215.0 ms (median of three runs, v3_vs_base
in the same file) without a breakdown of the
staging share. The loop still copied a super-block, waited at a barrier, and
only then computed. Now each thread
reads super-block b + 1 from global memory into registers (weights,
activation bytes and activation scales) before the mma loop of b, and stores
it to shared memory at the top of the next iteration, after the barrier that
ends b. The Q4_K GEMM went from 148.8 to 87.2 ms per forward (median
launch time times launches, middle value of three interleaved traces,
bench/profiles/2026-10-13-gemm4-*), and the
logits match main on all 84 rows (bench/checks/2026-10-13-gemm4-logits.json).
That the gain comes from overlapping the loads with compute is the reading
the numbers fit; no hardware counters were collected to confirm it.
Q6_K: the same layout and prefetch. The Q6_K kernel took the route the
Q4_K one did. A block of 64 rows copies each row's 224 bytes (low nibbles,
high bits, 16 int8 group scales, d) into shared memory as they are, in 16-byte
loads; a warp rebuilds each 16-weight group's A fragment in registers
(_q6k_frag: nibble, two high bits, minus 32 by an xor and a sign extension),
and the next super-block waits in registers while the mma loop runs. Groups
accumulate in the same order as before, so the result is unchanged. The
padded block is 12 bytes larger than the 212-byte padding it replaces: with
6,217,728 Q6_K blocks in LLaDA-8B Q4_K_M (output, 16 attn_v and 16 ffn_down
tensors) that is 74.6 MB (71.2 MiB) more device memory.
Tile width per call. The default rule minimizes token tiles, breaking ties toward the narrower width. RTX Blackwell (compute capability major 12) uses a shape-aware rule for Q4_K: start from that default, then choose the narrowest width that does not increase the estimated number of block waves across the device's SMs. The estimate assumes one resident block per SM; it is a dispatch heuristic, not an occupancy measurement. It can trade more blocks for less work within each block when a wide GPU has spare parallelism. Other device families keep their existing rule. The SM count is read when activation buffers are created, not on every GEMM call.
The LM head's small row count still uses the narrow tile. The available token widths remain 32, 64 and 80; 80 is the widest whose shared memory fits the 48 KiB static allocation limit. Weight layout, activation quantization and accumulation order are unchanged. Token tiles of one weight tile run back to back, so they read it through L2. The exact cross-width GPU test also covers ragged token counts and partial weight-row tiles.
Q6_K on compute capability 12 also offers a 16-token tile for canvases of
at most 320 tokens. It chooses among widths 16, 32, 64 and 80 by minimizing
(width + 16) * ceil(blocks / SM_count), preferring the narrower width on a
tie. For vocabulary projections with at least 65,536 weight rows and at most
32 tokens, it instead uses a 128-row, 32-token tile. Both Q4_K and Q6_K read
the same cached SM count; Q6_K keeps its previous rule on other devices and
longer canvases. Direct-launch tests compare the new Q6_K tiles exactly with
the original tile, including partial token and weight-row tiles.
Shared-memory layout. Rows are padded so the 8 rows a fragment load reads
fall in different banks: activation rows are 272 bytes apart (256 plus 16),
Q4_K nibble rows 144 bytes (128 plus 16) and Q6_K block rows 240 bytes (224
plus 16). Activation scales are stored by sub-block, then token, then (scale,
sum), so the pair of tokens a thread folds after an mma is one 128-bit load.
tests/gpu/test_gpu.mojo checks the fragment layout against a CPU matmul
on random int8 matrices, and the whole kernel against an f64 GEMM.
Row statistics on the device. A greedy step needs, per masked row, the argmax and a confidence. Copying the logits to the host (in LLaDA, 32 rows of 126,464 floats for a 32-token block) and looping over them there is the largest piece of a step outside the forward pass. One block per row now reads its logits twice: the first pass finds the first index of the maximum (a tie goes to the smaller index, as the host loop picks it), the second accumulates, in f64, the sum of exp(x - max), the sum of exp(x - max)(x - max) and the largest logit at any other index. The sampler finishes from those four numbers: p = 1 / z for LLaDA and maskgit_plus, p minus the runner-up's probability for topk_margin, and sum e (x - max) / z - log z for Dream's entropy. The exponentials are f32, so a confidence differs from the host's f64 loop in the sixth digit; tests bound it at 1e-5 relative.
Fused elementwise kernels. The quantized GEMMs want int8 activations, so
each norm and the FFN activation end in a quantize. Run separately, a layer
wrote and re-read the normalized rows and the product of the gate and up
projections through global memory. Three kernels remove that traffic:
add_rmsnorm_quantize adds the projection output into the residual stream,
normalizes the sum and quantizes it; rmsnorm_quantize does the same for
layer 0, whose input has no add; silu_mul_quantize quantizes silu(g) * u
without storing it. The norm kernels give thread t the values t, t + 256, ...
so a warp holds 32 consecutive values at each step, which is one
quantization block, and the quantize step is the same _quantize_lane code
the plain kernel runs. The float expressions and their order match the
unfused kernels', so the fused output is identical to the chain's, which the
tests assert exactly. The attention output still goes through the plain
quantize kernel.
Attention on f16 tensor cores. Each warp owns 16 queries and walks the
keys 32 at a time. Q is scaled by 1/sqrt(d) and held in registers as f16
fragments; K is staged row-major and V transposed, both as f16, so every
operand load is one 32-bit word. mma.m16n8k16 computes S = Q K^T in f32,
the four threads that share a row reduce its max with two shuffles, and the
online softmax rescales the running sums. S's accumulator for 16 keys has the
same register layout as the A operand of the next mma, so P goes from S to
P V without leaving the registers. On LLaDA-8B the logits stay as close to
the f32 CPU path as with f32 attention
(bench/checks/2026-10-13-attention-f16-logits.json). The kernel never
stores a score row, so the canvas length has no fixed limit.
Correctness
mojo-dllm is checked at four levels, each against an implementation it does not share code with.
1. Formats and kernels (unit tests, every commit)
- GGUF parsing, Q4_K / Q5_K / Q6_K / Q5_0 / Q8_0 / F16 / BF16 decoding: compared with the
ggufPython package's dequantization on fixtures it generated (max abs difference 1e-6). - Both GEMMs: compared with an f64 reference over the same dequantized weights; relative error under 1% with int8 activations.
- RMSNorm, RoPE, softmax, bidirectional attention, SwiGLU: closed-form cases.
- The forward pass of a 2-layer model with random K-quant weights: compared with a numpy forward pass; relative logit error under 2% on the LLaDA model and under 5% on the Dream model (its large Q/K biases sharpen the softmax, which amplifies int8 error), and the same argmax on every position.
The GPU has its own suite, tests/gpu/test_gpu.mojo, which runs on a machine
with an NVIDIA GPU (scripts/run-gpu-tests.sh; CI only builds it). It checks
each GPU kernel against the CPU function it replaces, the int8 mma fragment
layout against a CPU matmul, and the GPU forward pass of both tiny models
against the numpy reference, with a 5% bound on LLaDA (attention runs in f16) and
2% on Dream, and the same argmax as the CPU path on every position.
2. Tokenizer (unit tests, every commit)
36 strings covering English, contractions, numbers, accents, combining marks,
CJK, Cyrillic, Arabic, Devanagari, emoji, CRLF, non-breaking and em spaces,
code, and LLaDA's chat-template special tokens. Every id matches
llama-tokenize on the real LLaDA GGUF and on the real Dream GGUF, and decoding
restores the input byte for byte.
DiffusionGemma's gemma4 tokenizer has its own 54 cases (runs of spaces and
newlines, CJK, emoji, byte-fallback characters, a literal U+2581, and the chat
template's markers), with ids from llama-tokenize on the real model file
(tests/fixtures/gemma4-tokenizer-golden.tsv, and tests/fixtures/gemma4-tokenizer.gguf holds
the vocabulary they are encoded with). Each case is also checked with the BOS the
model adds, and decoded back; 53 of the 54 restore the input, and the one that
holds U+2581 comes back with spaces, as llama.cpp decodes it.
tests/test_gemma4_tiny.mojo pins each rule on a vocabulary small enough to work
out by hand.
3. Logits on the real model
tools/ref/llada_ref.cpp links libllama and dumps llama.cpp's logits for a
fixed token sequence; mojo-dllm logits dumps ours; bench/parity.py
compares them. The sections marked GPU run mojo-dllm logits --device gpu.
The GPU sections also compare against llama.cpp built with CUDA. That build quantizes activations with one scale per 32 values, as the GPU path does, while llama.cpp on the CPU uses one per 256, as the CPU path does. On Dream's prompt rows the two llama.cpp builds disagree with each other, so a single reference would mix that spread into mojo-dllm's numbers. The table under each GPU section shows all three pairs.
LLaDA-1.5.Q4_K_M.gguf
| Canvas | rows | median cosine | masked rows median | worst cosine | median rel. RMS | same argmax | top-5 overlap |
|---|---|---|---|---|---|---|---|
| 64 tokens | 64 | 0.99987 | 0.99989 | 0.9972 (row 13) | 0.018 | 61 / 64 | 4.75 / 5 |
| 126 tokens | 126 | 0.99990 | 0.99991 | 0.9993 (row 19) | 0.016 | 112 / 126 | 4.67 / 5 |
Evidence: bench/parity/2026-10-30-llada-LLaDA-1.5.Q4_K_M.json (mojo-dllm 5c82e8c).
LLaDA-8B-Instruct.Q4_K_M.gguf
| Canvas | rows | median cosine | masked rows median | worst cosine | median rel. RMS | same argmax | top-5 overlap |
|---|---|---|---|---|---|---|---|
| 64 tokens | 64 | 0.99985 | 0.99988 | 0.9951 (row 13) | 0.018 | 62 / 64 | 4.83 / 5 |
| 126 tokens | 126 | 0.99988 | 0.99989 | 0.9992 (row 19) | 0.017 | 120 / 126 | 4.67 / 5 |
Evidence: bench/parity/2026-10-03-llada.json (mojo-dllm 0b3e17a).
LLaDA-8B-Instruct.Q4_K_M.gguf, GPU
| Canvas | rows | median cosine | masked rows median | worst cosine | median rel. RMS | same argmax | top-5 overlap |
|---|---|---|---|---|---|---|---|
| 64 tokens | 64 | 0.99981 | 0.99983 | 0.9960 (row 13) | 0.021 | 62 / 64 | 4.77 / 5 |
| 126 tokens | 126 | 0.99988 | 0.99988 | 0.9984 (row 24) | 0.017 | 120 / 126 | 4.74 / 5 |
The same canvases against a second reference, llama.cpp CUDA, and the two references against each other:
| Canvas | compared | median cosine | worst cosine | median rel. RMS | same argmax |
|---|---|---|---|---|---|
| 64 tokens | mojo-dllm vs llama.cpp CPU | 0.99981 | 0.9960 | 0.021 | 62 / 64 |
| 64 tokens | mojo-dllm vs llama.cpp CUDA | 0.99988 | 0.9952 | 0.017 | 60 / 64 |
| 64 tokens | llama.cpp CPU vs llama.cpp CUDA | 0.99972 | 0.9960 | 0.026 | 60 / 64 |
| 126 tokens | mojo-dllm vs llama.cpp CPU | 0.99988 | 0.9984 | 0.017 | 120 / 126 |
| 126 tokens | mojo-dllm vs llama.cpp CUDA | 0.99992 | 0.9987 | 0.015 | 120 / 126 |
| 126 tokens | llama.cpp CPU vs llama.cpp CUDA | 0.99983 | 0.9973 | 0.021 | 117 / 126 |
Evidence: bench/parity/2026-10-09-llada-gpu.json (mojo-dllm cedc0b5).
DiffuCoder-7B-cpGRPO.Q4_K_M.gguf
| Canvas | rows | median cosine | masked rows median | worst cosine | median rel. RMS | same argmax | top-5 overlap |
|---|---|---|---|---|---|---|---|
| 71 tokens | 71 | 0.99994 | 0.99996 | 0.9964 (row 10) | 0.019 | 66 / 71 | 4.86 / 5 |
| 133 tokens | 133 | 0.99993 | 0.99994 | 0.9612 (row 2) | 0.016 | 128 / 133 | 4.72 / 5 |
Evidence: bench/parity/2026-10-30-dream-DiffuCoder-7B-cpGRPO.Q4_K_M.json (mojo-dllm 5c82e8c).
Dream-Coder-v0-Instruct-7B.Q4_K_M.gguf
| Canvas | rows | median cosine | masked rows median | worst cosine | median rel. RMS | same argmax | top-5 overlap |
|---|---|---|---|---|---|---|---|
| 71 tokens | 71 | 0.99992 | 0.99993 | 0.9795 (row 6) | 0.036 | 64 / 71 | 4.76 / 5 |
| 133 tokens | 133 | 0.99989 | 0.99990 | 0.9872 (row 22) | 0.031 | 115 / 133 | 4.41 / 5 |
Evidence: bench/parity/2026-10-30-dream-Dream-Coder-v0-Instruct-7B.Q4_K_M.json (mojo-dllm 5c82e8c).
Dream-v0-Instruct-7B-Q4_K_M.gguf
| Canvas | rows | median cosine | masked rows median | worst cosine | median rel. RMS | same argmax | top-5 overlap |
|---|---|---|---|---|---|---|---|
| 71 tokens | 71 | 0.99993 | 0.99994 | 0.9668 (row 26) | 0.032 | 64 / 71 | 4.61 / 5 |
| 133 tokens | 133 | 0.99986 | 0.99988 | 0.9782 (row 11) | 0.033 | 92 / 133 | 4.22 / 5 |
Evidence: bench/parity/2026-10-03-dream.json (mojo-dllm 0b3e17a).
Dream-v0-Instruct-7B-Q4_K_M.gguf, GPU
| Canvas | rows | median cosine | masked rows median | worst cosine | median rel. RMS | same argmax | top-5 overlap |
|---|---|---|---|---|---|---|---|
| 71 tokens | 71 | 0.99991 | 0.99993 | 0.9455 (row 26) | 0.042 | 62 / 71 | 4.63 / 5 |
| 133 tokens | 133 | 0.99984 | 0.99986 | 0.9016 (row 11) | 0.037 | 85 / 133 | 4.22 / 5 |
The same canvases against a second reference, llama.cpp CUDA, and the two references against each other:
| Canvas | compared | median cosine | worst cosine | median rel. RMS | same argmax |
|---|---|---|---|---|---|
| 71 tokens | mojo-dllm vs llama.cpp CPU | 0.99991 | 0.9455 | 0.042 | 62 / 71 |
| 71 tokens | mojo-dllm vs llama.cpp CUDA | 0.99994 | 0.8179 | 0.020 | 64 / 71 |
| 71 tokens | llama.cpp CPU vs llama.cpp CUDA | 0.99987 | 0.6492 | 0.044 | 62 / 71 |
| 133 tokens | mojo-dllm vs llama.cpp CPU | 0.99984 | 0.9016 | 0.037 | 85 / 133 |
| 133 tokens | mojo-dllm vs llama.cpp CUDA | 0.99990 | 0.8975 | 0.022 | 97 / 133 |
| 133 tokens | llama.cpp CPU vs llama.cpp CUDA | 0.99979 | 0.8784 | 0.039 | 80 / 133 |
Evidence: bench/parity/2026-10-09-dream-gpu.json (mojo-dllm cedc0b5).
Exact equality is not the goal: both runtimes quantize activations to int8, but they block and accumulate in different orders. The llama.cpp maintainers recommend the same kind of statistical comparison for diffusion models.
The table scores every position of each canvas. On the original Dream-v0 checkpoint the median agrees as closely as on LLaDA, but a few prompt positions do not: they sit on special tokens and newlines, where Qwen-family models carry very large outlier activations that magnify int8 rounding. Masked positions, the ones the sampler reads, agree closely. Which of the two runtimes is nearer an f32 reference at those outlier rows is not measured yet.
Agreement with llama.cpp shows that two int8 implementations land near each other. It does not show that either is right. The only f32 forward-pass reference in this repository is the numpy pass on the tiny models (section 1), and the GPU tests compare against it too.
4. Generated tokens on the real model
llama.cpp's own diffusion example cannot serve as a token-level reference. On
the pinned commit it never applies --diffusion-cfg-scale or
--diffusion-alg-temp, adds Gumbel noise to one row only, ranks entropy in
the wrong direction, and samples from the distribution even at temperature 0.
So llada_ref generate re-implements LLaDA's generate.py (temperature 0,
low-confidence remasking) on top of llama.cpp's logits, and the result is
compared token by token with mojo-dllm run. For Dream, llada_ref generate-dream does the same with Dream's diffusion_generate.
A diffusion loop can amplify small differences: when two candidates are nearly tied, the runtimes can commit different tokens at one step, and every later step then sees a different canvas. The table's first difference is a position in the final output; it does not identify the denoising step or the cause of the mismatch. Exact token agreement is specific to the measured prompt and configuration.
What this section covers: two prompts, 32 and 64 generated tokens (the
benchmarks use 128), temperature 0, 32 steps, with low-confidence remasking for
LLaDA and entropy ranking for Dream. The tables below give the count of equal
tokens and the first difference for each model; on the speculative-decoding
prompt several are well short of a full match, and Dream-v0's code prompt
diverges within its first few tokens. Not covered: Gumbel noise (temperature
above 0), random remasking, Dream's other remasking algorithms, classifier-free
guidance (cfg_scale), alg_temp, and generation at 128 tokens. --threshold
and --dual-cache have a paired GSM8K comparison instead
(Benchmarks).
LLaDA-1.5.Q4_K_M.gguf
| Prompt | generated | steps | same tokens | first difference |
|---|---|---|---|---|
| Explain speculative decoding in two sentences. | 32 | 32 | 24 / 32 | position 21 |
| Write a short Python function that checks whether a number is prime. | 64 | 32 | 64 / 64 | none |
Evidence: bench/parity/2026-10-30-llada-LLaDA-1.5.Q4_K_M.json (mojo-dllm 5c82e8c).
LLaDA-8B-Instruct.Q4_K_M.gguf
| Prompt | generated | steps | same tokens | first difference |
|---|---|---|---|---|
| Explain speculative decoding in two sentences. | 32 | 32 | 8 / 32 | position 5 |
| Write a short Python function that checks whether a number is prime. | 64 | 32 | 64 / 64 | none |
Evidence: bench/parity/2026-10-03-llada.json (mojo-dllm 0b3e17a).
LLaDA-8B-Instruct.Q4_K_M.gguf, GPU
| Prompt | generated | steps | same tokens | first difference |
|---|---|---|---|---|
| Explain speculative decoding in two sentences. | 32 | 32 | 8 / 32 | position 5 |
| Write a short Python function that checks whether a number is prime. | 64 | 32 | 64 / 64 | none |
Evidence: bench/parity/2026-10-09-llada-gpu.json (mojo-dllm cedc0b5).
DiffuCoder-7B-cpGRPO.Q4_K_M.gguf
| Prompt | generated | steps | same tokens | first difference |
|---|---|---|---|---|
| Explain speculative decoding in two sentences. | 32 | 32 | 22 / 32 | position 17 |
| Write a short Python function that checks whether a number is prime. | 64 | 32 | 63 / 64 | position 14 |
Evidence: bench/parity/2026-10-30-dream-DiffuCoder-7B-cpGRPO.Q4_K_M.json (mojo-dllm 5c82e8c).
Dream-Coder-v0-Instruct-7B.Q4_K_M.gguf
| Prompt | generated | steps | same tokens | first difference |
|---|---|---|---|---|
| Explain speculative decoding in two sentences. | 32 | 32 | 19 / 32 | position 18 |
| Write a short Python function that checks whether a number is prime. | 64 | 32 | 13 / 64 | position 13 |
Evidence: bench/parity/2026-10-30-dream-Dream-Coder-v0-Instruct-7B.Q4_K_M.json (mojo-dllm 5c82e8c).
Dream-v0-Instruct-7B-Q4_K_M.gguf
| Prompt | generated | steps | same tokens | first difference |
|---|---|---|---|---|
| Explain speculative decoding in two sentences. | 32 | 32 | 24 / 32 | position 22 |
| Write a short Python function that checks whether a number is prime. | 64 | 32 | 43 / 64 | position 3 |
Evidence: bench/parity/2026-10-03-dream.json (mojo-dllm 0b3e17a).
Dream-v0-Instruct-7B-Q4_K_M.gguf, GPU
| Prompt | generated | steps | same tokens | first difference |
|---|---|---|---|---|
| Explain speculative decoding in two sentences. | 32 | 32 | 26 / 32 | position 23 |
| Write a short Python function that checks whether a number is prime. | 64 | 32 | 28 / 64 | position 1 |
Evidence: bench/parity/2026-10-09-dream-gpu.json (mojo-dllm cedc0b5).
Reproduce
The added LLaDA-1.5, Dream-Coder-v0-Instruct-7B and DiffuCoder-7B-cpGRPO
records also contain CPU smoke output, peak RSS, prompt token IDs, GGUF
SHA256 and pinned Hub revisions. Their metadata audits compare vocabulary,
layer count, context length and RoPE theta with the source HF configs.
These runs use four CPU threads and llama.cpp
836d57176dc699a726c55418e4f96b8ca628e1bf; they add no GPU evidence.
LLaDA-1.5 and Dream-Coder carry the same chat templates as their HF tokenizer
configs. DiffuCoder-cpGRPO has no HF chat template, while its GGUF uses the
Dream-Coder template. Its HF pad token is <|dlm_pad|> (151667), but its
GGUF padding ID is 151643. The smoke records explicitly note whether that
pad token or a mask token appeared in the text. DiffuCoder's HF license is
apple-amlr, despite the GGUF license field saying unknown; obtain it
separately and pass a local file.
Build llama.cpp at the revision above with GGML_CUDA=OFF, then set LLAMA
to that checkout before compiling the reference:
c++ -O2 -std=c++17 tools/ref/llada_ref.cpp -I$LLAMA/include -I$LLAMA/ggml/include \
-L$LLAMA/build/bin -lllama -lggml -lggml-base -Wl,-rpath,$LLAMA/build/bin -o build/llada_ref
uv run --only-group fixtures python bench/parity.py --arch llada --model models/LLaDA-8B-Instruct.Q4_K_M.gguf
uv run --only-group fixtures python bench/parity.py --arch dream --model models/Dream-v0-Instruct-7B-Q4_K_M.gguf
uv run --only-group fixtures python bench/parity.py --arch llada --threads 4 --model models/LLaDA-1.5.Q4_K_M.gguf
uv run --only-group fixtures python bench/parity.py --arch dream --threads 4 --model models/Dream-Coder-v0-Instruct-7B.Q4_K_M.gguf
uv run --only-group fixtures python bench/parity.py --arch dream --threads 4 --model models/DiffuCoder-7B-cpGRPO.Q4_K_M.gguf
uv run --no-project python scripts/bench_table.py --write
llada_ref refuses a token, a row or a count that is not an integer, a row
outside the tokens, a token outside the vocabulary, an output it cannot write
and any non-finite logit in a row it reads. Each refusal prints a line that
begins llada_ref: and exits non-zero (1 for the arguments, 2 for the model
or a file, 3 for llama.cpp or a NaN). It writes OUT.part and moves it to
OUT only after every row passed, so a failed run leaves no file under the
name bench/parity.py reads. tests/python/test_llada_ref.py builds it with
the host compiler against a stand-in for llama.cpp and runs each refusal.
The GPU tables need --device gpu and a second reference. Compile
llada_ref once more against the CUDA build of llama.cpp (build-cuda/, see
Development), name the result
build/llada_ref_cuda, and add --device gpu --ref2 build/llada_ref_cuda to
the bench/parity.py commands.
DiffusionGemma
Mainline llama.cpp cannot load a DiffusionGemma GGUF. The reference is the
branch of ggml-org/llama.cpp#24423, built at one commit by
scripts/build_dg_ref.sh (CPU only, no network after the fetch) into its own
tree, $DG_LLAMA_SRC, beside the mainline checkout. The commit is the
PINNED_COMMIT in that script, and tests/fixtures/gemma4-tokenizer-golden.provenance.json
records the same one. tools/ref/dg_ref.cpp is linked against that build and
writes the canvas logits of one denoising step: the unified [prompt | canvas]
pass or the prompt K/V path (a prefill of the prompt, then one decode step), each
without self-conditioning and with it, from a previous-step buffer given on the
command line.
The model file is 16.81 GB and mojo-dllm keeps its own repacked copy beside the
mapping the reference holds, so the two never run together. bench/parity.py
runs them in turns and compares saved files:
scripts/build_dg_ref.sh
uv run --only-group fixtures python bench/parity.py --arch diffusion-gemma --phase ref \
--model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --llama-src "$DG_LLAMA_SRC"
# mojo-dllm writes build/dg-parity/<model>/caseN/<variant>.mojo.f32 (the forward pass is #64)
uv run --only-group fixtures python bench/parity.py --arch diffusion-gemma --phase compare \
--model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf
Each <variant>.mojo.f32 has a <variant>.mojo.f32.json beside it with the
case_sha256 of its case (bench/parity.py dg_case_hash: prompt, prompt tokens and canvas) and the
mojo_dllm_commit of the build that wrote it. The comparison needs the cases
case0 to case{N-1} of the current prompt set and stops with a message while
any mojo-dllm file or sidecar is missing, was written for another case, or comes
from a different build than the others, and
refuses a file that is byte-identical to the reference's. It writes evidence to
bench/parity/ only when every variant is no worse than the worst case of the
LLaDA comparison in bench/parity/2026-10-03-llada.json, on each of minimum
cosine, argmax agreement, top-5 overlap and maximum relative RMS. The bound is
read from that file, not typed. A fifth metric holds the largest logit of every
row to the same relative error as LLaDA's rows reach: the final soft-cap pins
that logit near 30, so the first four cannot see the cap removed.
bench/check_dg_bound.py shows what the bound rejects.
Each case records how far the reference lies from itself: the prompt K/V path against the unified pass, and the unified pass again with the previous-step logits changed by one part in a million. On the two LLaDA prompts with a random canvas those distances exceed the LLaDA bound (the one-in-a-million change moved the output by a median relative RMS of 0.18 to 0.26). No LLaDA spread was measured for comparison. The cause is untested: expert routing flips (8 of 128 experts per token) carried to every row by bidirectional attention are a hypothesis. A failed comparison prints these distances beside the verdict. No DiffusionGemma result is recorded yet, so the tables above hold none.
The tokenizer goldens in tests/fixtures/gemma4-tokenizer-golden.tsv fix the
encoding, and tests/test_tokenizer_gemma4.mojo checks the Mojo tokenizer against
them. Decoding the ids through the GGUF vocabulary restores 53 of the
54 strings; the string holding a literal U+2581 does not come back, because
decoding turns every U+2581 into a space.
Development
Setup
uv sync --frozen # the pinned Mojo toolchain, in .venv
scripts/ci-local.sh --install-hooks
scripts/ci-local.sh also needs shellcheck, markdownlint-cli2 (npm install --global markdownlint-cli2@0.23.2), mdBook 0.4.40 and an authenticated gh. A missing tool
fails its step; it never skips it.
The hooks run scripts/ci-local.sh --fast before each commit (hygiene scan,
mojo format, -Werror build) and the full set before each push.
The pre-commit build runs before its commit exists, so its binary records the
parent HEAD, with a -dirty suffix when the commit changes src/. The
launcher rebuilds when a source file is newer than the binary, or when
build/mojo-dllm.stamp (written by the launcher's own build, not by
scripts/build.sh) is missing or names another Mojo version or GPU
architecture. After a hook build with a matching stamp the launcher sees a
current binary and keeps reporting the parent commit: run scripts/build.sh
after committing to stamp the new HEAD.
The gates
scripts/ci-local.sh runs the gates of .github/workflows/ci.yml and, on a
machine with a GPU, the GPU tests that CI only builds:
| Step | Command | Fails when |
|---|---|---|
| hygiene | scripts/check_hygiene.sh --tracked | a credential pattern (a provider token, a private key block, a database URL with a password) or the author's employer name appears in tracked text, or a tracked Markdown file outside bench/results/ holds an em or en dash; file names are read NUL separated, so a name with non-ASCII or special characters is scanned too) |
| format | scripts/check_format.sh (mojo format -q src tests bench, comparing every file before and after) | mojo format changes any file, an untracked one included (Mojo has no check mode) |
| build | scripts/build.sh | any compiler warning (-Werror) |
| embedded build commit | scripts/check_build_id.sh | the built binary does not report the expected commit, or a build without Git metadata does not report unknown |
| tests | scripts/run-tests.sh | a test fails, a file runs zero tests, the total differs from tests/test-count.txt, or a file's count differs from tests/test-counts.txt |
| launcher tests | uv run --locked --only-group test pytest tests/python -q | a test of the ./mojo-dllm launcher fails |
| script lock | uv lock --script mojo-dllm --check | mojo-dllm.lock is missing, or the # dependencies header of ./mojo-dllm no longer matches it. After you change the pin, run uv lock --script mojo-dllm and commit the lock |
| shellcheck, yamllint, markdownlint | shellcheck --severity=warning scripts/*.sh .githooks/*; uv run --frozen --only-group lint yamllint --strict .github; markdownlint-cli2 "*.md" "docs/**/*.md" "assets/*.md" ".github/*.md" "#node_modules" | any of the three reports a finding |
| claims | uv run --no-project python scripts/bench_table.py --check | a table in the README, docs/benchmarks.md or docs/correctness.md differs from what bench/results, bench/parity, bench/quality and bench/eval render (a missing table marker, target file or quality or eval evidence passes, #141; an unknown mode exits 0, #142) |
| docs | mdbook build + scripts/check_links.py | a relative link is broken |
| action pins | scripts/verify-action-pins.sh (needs an authenticated gh) | a pinned SHA is not the tag its comment names |
| eval self-test | uv run --no-project python bench/eval_gsm8k.py --self-test | answer extraction or the paired statistics break, or gsm8k-200.jsonl is not 200 rows with numeric answers |
| checker self-test | uv run --frozen --only-group fixtures python scripts/check_weights.py --self-test | a safetensors or index guard in check_weights.py stops rejecting its case, or stops naming it |
| no GPU | scripts/check_no_gpu.sh | --device gpu without a GPU does anything but exit 1 with a message |
| visual options | scripts/check_visual.sh | an invalid --visual-interval does not exit 1, or --json with --visual does not exit 2, with a message naming the flag before model loading |
| fixtures | uv run --frozen --only-group fixtures python scripts/check_fixtures.py | scripts/gen_fixtures.py or scripts/gen_moe_fixture.py writes a file that differs from the tracked one (they run into a scratch directory), a file that is not tracked, or a tracked fixture that no generator writes and that is not a tokenizer fixture, a tokenizer golden or golden-generation.tsv |
| GPU test count | scripts/check_gpu_test_count.sh | the def test_ functions of tests/gpu/test_*.mojo number other than tests/gpu/test-count.txt pins or differ per file from tests/gpu/test-counts.txt, or a file defines none; needs no GPU, runs in CI and ci-local.sh |
| GPU tests | scripts/run-gpu-tests.sh (local only) | a GPU kernel or the GPU forward pass disagrees with the CPU or the f32 reference, or the count differs from tests/gpu/test-count.txt or tests/gpu/test-counts.txt |
GitHub's runners have no GPU, so CI builds the GPU suite without running it.
ci-local.sh runs it when nvidia-smi lists a GPU and otherwise builds it and
prints a SKIP line for the run, never a PASS; run-gpu-tests.sh on its own exits 1 without a
GPU.
The launcher tests run pytest -rs, so every skip is listed in the summary,
and tests/python/skip_policy.py fails the run on a skip that
tests/python/allowed-skips.txt does not name (a test id and the text of the
reason). Add a line there in the commit that adds a skip. --locked makes
uv run fail when uv.lock no longer matches pyproject.toml; --frozen does
not compare them.
Branch protection on main requires four checks: hygiene, mojo and docs of the CI
workflow and trufflehog of the secret scan. The job build of the
"Deploy docs to GitHub Pages" workflow skips while the repository is private,
and a skipped job satisfies a required check, so it must not be required. CI
already runs the same mdbook build in docs.
The test count is pinned because Mojo's TestSuite exits 0 after running zero
tests. If you add tests, run UPDATE_TEST_COUNT=1 scripts/run-tests.sh (for
tests/gpu/, UPDATE_TEST_COUNT=1 scripts/run-gpu-tests.sh) and commit the
new count with them.
Tests
scripts/run-tests.sh compiles each tests/test_*.mojo and runs it. Fixtures
live in tests/fixtures/ and come from scripts, never from hand:
| Fixture | Made by | Checks |
|---|---|---|
kquants.gguf, bad-magic.gguf, tiny-llada.gguf, tiny-llada.expected.f32, tiny-dream.gguf, tiny-dream.expected.f32, malformed/, metadata/, gen_fixtures.manifest | scripts/gen_fixtures.py | GGUF parsing and rejection, decoders, ModelConfig validation, the forward pass against a numpy reference |
tiny-diffusion-gemma.gguf, diffusion-gemma/ | scripts/gen_fixtures.py | the diffusion-gemma config (seed 610061): a 2-layer model with one sliding and one global layer, 4 experts, the tensor names and types of the real file, and one file per refusal (metadata only, or zero-filled tensors) |
tiny-diffusion-gemma.p0sc.case, .p9.case, .p9sc.case, .p150.case, .p150sc.case (tiny-diffusion-gemma.*.case) | scripts/gen_fixtures.py (the forward pass is scripts/dg_numpy.py) | the tokens and numpy logits that logits on the tiny DiffusionGemma file must reproduce, for a prompt of 0, 9 and 150 tokens, with and without self-conditioning (sc) |
entropy-rows.tsv | scripts/gen_fixtures.py (seed 65065, so no other fixture changes) | logits rows with the numpy f64 argmax, entropy and inverse-CDF draw that tests/test_entropy_bound.mojo pins |
utf8-cases.tsv | scripts/gen_fixtures.py | byte strings with the code points and the first invalid offset that Python's own UTF-8 decoder gives, for utf8_decode |
dequant-types.gguf, unsupported-embedding.gguf | scripts/gen_fixtures.py | Q8_0/Q5_K/BF16/Q5_0 references (seed 340034) and embedding rejection before CPU/GPU allocation (no RNG) |
llada-tokenizer.gguf, llada-tokenizer-golden.tsv, llada-tokenizer-golden.provenance.json, dream-tokenizer.gguf, dream-tokenizer-golden.tsv, dream-tokenizer-golden.provenance.json | scripts/gen_tokenizer_fixture.py (--golden-only rewrites the golden and its provenance) | tokenizer ids against llama-tokenize. Each provenance file records the llama.cpp commit, the model file and its sha256, the command and the golden's sha256. The two goldens were committed with the first import, and regenerating them with llama.cpp 2e715c0639 on 2026-11-09 gave the same bytes |
gemma4-tokenizer.gguf | scripts/gen_tokenizer_fixture.py ... gemma4 | the 262,144 tokens of the DiffusionGemma file and the 472 of its 514,906 merges that apply to the 54 golden strings, tokenizer keys only and without the scores (5.5 MB) |
gemma4-tiny/ | scripts/gen_fixtures.py | a 279-token gemma4 vocabulary whose ids can be worked out by hand, and variants with one thing wrong or unusual each (a missing byte token, a merge with no space, mismatched counts, nested special tokens, an unknown-type token, an end-of-turn token the vocabulary lacks) |
gpt2-tiny/ | scripts/gen_fixtures.py | a six-token byte-level BPE vocabulary with padding, unknown and EOS ids and no end-of-turn token: the ids that LLaDA's and Dream's files leave out |
gemma4-tokenizer-golden.tsv, gemma4-tokenizer-golden.provenance.json | scripts/gen_tokenizer_fixture.py ... gemma4 (--golden-only rewrites only these) | DiffusionGemma tokenizer ids against llama-tokenize from the ggml-org/llama.cpp#24423 branch at the commit scripts/build_dg_ref.sh pins; the provenance file records that commit, the model file's sha256 and the command |
golden-generation.tsv | UPDATE_GOLDEN=1 build/tests/test_golden | tiny LLaDA/Dream token sequences, forward-pass counts, early stopping and per-step commits |
src/mojo_dllm/tokenizer/unicode_tables.mojo | scripts/gen_unicode_tables.py | code point classes from llama.cpp's own tables |
moe-q5.bin, moe-q8.bin, moe-q4-q5.bin, moe-q4-q8.bin | scripts/gen_moe_fixture.py | deterministic raw expert tensors for scalar MoE comparisons, including mixed activation block sizes |
| GGUF/source tensor shape report | scripts/check_weights.py (--self-test) | LLaDA and Dream name mapping and safetensors header shapes; run with uv run --only-group fixtures python scripts/check_weights.py MODEL.gguf INDEX.json (the second argument may be one .safetensors file) |
tests/gpu/test_gpu.mojo holds the GPU tests: each kernel against the CPU
function it replaces, and the GPU forward pass on both tiny models against the
f32 reference.
The tiny model is 2 layers of hidden size 256 with random K-quant blocks. Its
expected logits come from a float numpy forward pass over the weights that
the gguf package dequantizes, so the Mojo forward pass is checked against
the reference formats, not against itself.
Sampler goldens
tests/test_golden.mojo pins current behavior on the tiny fixtures. These
outputs come from mojo-dllm itself: they detect changes, not independent
correctness. The numpy forward-pass tests and real-model parity runs provide
that evidence. No model download or GPU is needed.
Run from the repository root:
bash scripts/build.sh tests/test_golden.mojo build/tests/test_golden
build/tests/test_golden
# Only after reviewing an intentional behavior change:
UPDATE_GOLDEN=1 build/tests/test_golden
Only UPDATE_GOLDEN=1 writes the file, and scripts/run-tests.sh drops that variable
from the environment (and says so) unless you run scripts/run-tests.sh --update-golden:
the test rewrites the golden and then compares against what it wrote, so an exported
variable would make every run pass. Normal runs reject missing, empty,
duplicate, unknown or incomplete cases and compare tokens and schedule metadata
exactly. A mismatch names the case, the first differing token position (including
the prompt), both token lists and both complete rows. A golden diff in a PR needs
a sentence explaining why behavior changed, and test_model.mojo must still
pass. Do not regenerate merely to silence a failure on another machine; inspect
the differing logits and select a case without an unstable near-tie.
The 14 cases cover LLaDA block geometry, uneven transfer counts, serial commits,
random remasking, temperature and early stop, plus Dream's three confidence
rules, uneven timesteps, nondefault eps and early stop. LLaDA uses prompt
[5, 17, 300]; Dream uses [448, 50, 134] because the original prompt gave
identical all-49 outputs for D1 and D5. To avoid duplicate token lists, L5 uses
seed 1, D2 uses eps=0.2, D3 uses a 12-token canvas and D4 uses eps=0.1.
D1/D5 retain the same prompt, length and steps and differ only in eps.
L8/D6 use fixed stop ids 347/49 taken from the baseline outputs, never selected
at runtime. All cases disable threshold and dual-cache acceleration.
Heavy jobs
A job that holds more memory than the machine can spare makes the kernel's global OOM killer pick a victim.
With systemd's default OOMPolicy=stop, systemd then stops every process in the victim's scope, and a job
started from a terminal lives in that terminal's scope: the shell, the editor session and every other session in the
window end with it. A measured run of DiffusionGemma-26B on the CPU peaked at about 25.6 GiB resident, and on a
30 GiB machine with a 2 GiB swapfile that run ended in exactly this way.
scripts/capped.sh starts a command in a scope of its own, under a hard cap, so only that scope can fail:
scripts/capped.sh --mem 26G -- build/mojo-dllm run --model model.gguf --prompt "hello"
It sets four things, each for a reason you can check in the documentation of the kernel's cgroup v2 memory controller and of systemd's resource control:
| Setting | Effect |
|---|---|
MemoryMax=26G | the hard cap (memory.max); the kernel OOM killer runs inside the scope when it is reached and kills nothing outside it |
MemorySwapMax=0 | no swap, so a job over its cap dies instead of dragging the desktop through swap |
OOMPolicy=kill | one OOM kill ends the whole scope (memory.oom.group), not half of it |
CPUQuota=N00% | only with --cpus N, so a job leaves cores for the desktop |
It never sets MemoryHigh. Below the cap, MemoryHigh throttles a process that has nothing to reclaim until it
stands still (state D, no progress); a job that needs its memory should be killed, not frozen.
The script also refuses to start a job in two cases:
/proc/meminforeports lessMemAvailablethan--need(default:--mem). A cap larger than the available memory does not protect the machine, because the global OOM killer still runs. Exit status 75.- The systemd user manager cannot enforce a memory limit (no cgroup v2, or the
memorycontroller is not delegated to it). It never runs the command uncapped. Exit status 69.
--dry-run runs both checks and prints the systemd-run command without running it. When a capped job dies with
exit status 137, the cap was most likely reached; journalctl -k | grep oom-kill tells the cases apart:
constraint=CONSTRAINT_MEMCG is the cap working, while constraint=CONSTRAINT_NONE with global_oom means the
whole machine ran out of memory.
The test suite checks the limits the kernel reports for a capped scope (memory.max, memory.swap.max,
memory.oom.group). The test that kills a process at its cap runs only with CAPPED_TEST_KILL=1, because GNOME's
gsd-housekeeping answers every such kill with an "Application Stopped" notification.
For a container, set the same two limits on the container: podman run --memory 12g --memory-swap 12g
(--memory-swap counts memory plus swap, so equal values mean no swap).
What does not isolate a job on its own:
ulimit -vlimits address space, which counts mapped files such as a 16 GB model that is read lazily, and it does not stop a process that stays below the limit but takes the machine's last free page.systemd-oomdreacts to memory pressure and swap use over seconds, and a job that fills memory in one allocation burst can reach the kernel's killer first.- A swapfile only delays the kill and slows the machine while it lasts.
References: the kernel's Control Group v2 memory
interface, systemd's
resource control and
scope units (OOMPolicy=).
Benchmarks
uv run --no-project python bench/run_bench.py --config bench/config.json --reps 3
uv run --no-project python scripts/bench_table.py --write
bench/config.json runs every CPU runtime in one pass and names the model
files and the llama.cpp and diffuse-cpp checkouts. The committed CPU benchmark
tables came from bench/config-main.json, bench/config-diffuse.json and
bench/config-dream.json; the thread sweep came from bench/config-fair-cpu.json. scripts/bench_table.py refuses a runtime that
appears in two result files, so move the files a new run replaces into
bench/results/archive/ before --write. The harness waits for an idle CPU before each run, interleaves the
runtimes, switches the power profile to performance for the duration, and
writes a result file with every raw run. On a laptop it measures on mains
power only: unplugged, it pauses before the next run and prints that it is
waiting for the charger, and a run the charger left midway is repeated. The
eval, the quality table and scripts/gpu_profile.py do the same through
bench/power.py; --allow-battery measures on battery instead and records
it. The benchmark harness kills any process named
pytest while it runs, because test suites from other work on the
development machine kept starting mid-benchmark. Set "kill_pytest": false
in a config to turn that off; the GPU configs (bench/config-gpu.json,
bench/config-dream-gpu.json) do, since a GPU run leaves the CPU nearly idle.
They compare --device gpu with llama.cpp's diffusion example built with
CUDA in build-cuda/:
cmake -B build-cuda -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build-cuda --target llama-diffusion-cli
GPU kernels
scripts/gpu_profile.py traces one run --device gpu with Nsight Systems
(nsys --trace=cuda, which ran here without root although the driver keeps
hardware counters for admins) and writes the device time per
forward pass of each kernel group to bench/profiles/. It does not
synchronize the stream, unlike MOJO_DLLM_GPU_PROFILE=1. --root points it at
another checkout's build for before/after comparisons, and it reports a
median-based total next to the plain sum, because a laptop GPU's clocks move
under sustained load. --runtime llama.cpp --seq-len N traces llama.cpp's
CUDA llama-diffusion-cli on the same prompt instead (it replays its forward
pass as a CUDA graph, so the trace expands graph nodes), with the same kernel
groups. bench/bench_gpu_gemm.mojo times each LLaDA weight
shape alone, adding up the per-forward total from the tensor types in the
GGUF.
On the development laptop the GPU is capped at 30 W and its clocks drop under
sustained load (bench/checks/2026-10-12-gpu-power-under-load.txt), so
compare kernels with interleaved runs (A, B, A, B) in one session rather than
against a number from another day.
Comparing GPU dispatch changes
bench/compare_gpu.py compares two already-built checkouts through
run_bench.run_mojo, interleaving A/B and B/A rounds on one assigned GPU.
It excludes a warmup pair per generation length, records every raw run, and
rejects failures or mismatched forward counts and canvas sizes. It leaves the
system power profile and other processes alone. Keep each pair on the same
GPU; another GPU can run a different model's independent comparison.
Both tools, and scripts/compare_logits.py, refuse a base and a new that are
the same checkout, the same clean commit or byte-identical binaries, and record
the SHA-256 of each build/mojo-dllm in the evidence, because a stale build
carries the label of whatever commit is checked out. compare_gpu.py sets
CUDA_DEVICE_ORDER=PCI_BUS_ID so that --gpu (an index or a GPU- uuid)
names the same card to CUDA and to nvidia-smi, records the index and the
uuid read back from nvidia-smi, and runs the isolation preflight and a
window around each run of bench/isolation.py
(--allow-noisy continues after a failed preflight). check_gpu_dispatch.py
records CUDA_VISIBLE_DEVICES and CUDA_DEVICE_ORDER as it found them, so
export CUDA_DEVICE_ORDER=PCI_BUS_ID there too.
python3 bench/compare_gpu.py --base ../base --new . --model MODEL.gguf \
--gpu GPU-UUID --lengths 128,256,512 --reps 3 --prompts 3 \
--out bench/results/dispatch/run.json
CUDA_VISIBLE_DEVICES=GPU-UUID python3 bench/check_gpu_dispatch.py \
--base ../base --new . --model MODEL.gguf --arch llada \
--out bench/checks/dispatch-exact.json
Use --arch dream for Dream. The exact check uses that model's mask token,
compares all finite logit values byte for byte at several canvas sizes, and
compares complete generated token lists. It checks preservation of the base
GPU computation, not correctness against an independent runtime. The timing
comparison runs without the synchronizing MOJO_DLLM_GPU_PROFILE mode.
Quality
bench/eval_gsm8k.py answers a fixed 200-problem GSM8K subset
(bench/eval/gsm8k-200.jsonl, see bench/eval/SOURCE.md) and compares two
arms problem by problem: accuracy difference with a 95% interval and
McNemar's exact test. Any change that is not exact (a sampler that commits
more per step, a K/V cache) needs a comparison against the one-token-per-step
baseline before it becomes a default.
uv run --no-project python bench/eval_gsm8k.py run --model M.gguf --arm baseline
uv run --no-project python bench/eval_gsm8k.py run --model M.gguf --arm candidate -- --some-flag
uv run --no-project python bench/eval_gsm8k.py compare baseline candidate
uv run --no-project python bench/eval_gsm8k.py compare baseline candidate --limit 50
compare writes bench/eval/<date>-<a>-vs-<b>.json, and
scripts/bench_table.py turns those files into the accuracy table in
Benchmarks. Time per problem depends on what else the
machine runs, so --limit N compares each arm's first N problems in run
order and writes a -firstN file; the table takes its times from that file.
Run that round with nothing else on the machine.
Toolchain
Mojo is pinned to 1.1.0 and max-core to 26.6.0 in pyproject.toml, locked in
uv.lock; both come from PyPI. Bump them together by editing the two pins,
running uv lock, and committing that on its own with a green run: Mojo 1.1
removed fn, alias and @parameter closures, and each release has moved
APIs this code uses. max-core supplies max.algorithm.parallelize and the
GPU API (max.gpu, max.gpu.host). The Mojo wheels need glibc 2.34 or newer.
The Python scripts run through uv too. uv run --no-project python script.py
runs a script that uses only the standard library. The fixture generators and
bench/parity.py need numpy and gguf, pinned in the fixtures group:
uv run --only-group fixtures python scripts/gen_fixtures.py.
Releasing
A release is a version bump, a dated CHANGELOG section, a tag and a GitHub
release. Pushing the tag and creating the release need write access to the
repository. X.Y.Z stands for the new version.
-
Check that CI is green on the commit of
mainthat you ship. -
In
CHANGELOG.md, rename## [Unreleased]to## [X.Y.Z] - YYYY-MM-DDand add an empty## [Unreleased]above it. -
Set the version in the three places that carry it:
versionunder[project]inpyproject.toml,comptime VERSIONinsrc/main.mojo, anduv.lock. Runuv lockfor the third; its diff is one line, theversionof themojo-dllmentry. Result files underbench/record the version of the binary that produced them; leave them. -
Run
scripts/ci-local.sh. On a machine with an NVIDIA GPU also runscripts/run-gpu-tests.sh, because CI builds the GPU suite and does not run it. -
Commit the files that steps 2 and 3 changed (
CHANGELOG.md,pyproject.toml,src/main.mojo,uv.lock) with the subjectRelease X.Y.Z, together with any other file that still names the old version or milestone as current (for exampledocs/plan.md). Get the commit ontomain. -
Tag that commit. The existing tag
v0.1.0is annotated and not signed:git tag -a vX.Y.Z -m "mojo-dllm X.Y.Z"For a signed tag use
git tag -s vX.Y.Z -m "mojo-dllm X.Y.Z"and check it withgit tag -v vX.Y.Z. -
Push the tag with
git push origin vX.Y.Z. Check from the remote side that it arrived:git ls-remote origin 'refs/tags/vX.Y.Z^{}'prints the commit thatgit rev-parse 'vX.Y.Z^{commit}'prints. -
Create the GitHub release with the CHANGELOG section as its notes. Skip this step once the release workflow exists (#131): pushing the tag in step 7 then creates the release.
notes="$(mktemp)" awk -v v="X.Y.Z" '/^## \[/ { p = (index($0, "## [" v "]") == 1); next } p' CHANGELOG.md > "$notes" test -s "$notes" gh release create vX.Y.Z --verify-tag --title "mojo-dllm X.Y.Z" --notes-file "$notes"test -sstops the sequence when the section is missing or empty.gh release view vX.Y.Zshows the result.
Porting a model
mojo-dllm runs the LLaDA and Dream architectures, and reads the metadata of
diffusion-gemma (see below). Tested Q4_K_M checkpoints
are LLaDA-8B-Instruct, LLaDA-1.5, Dream-v0-Instruct-7B,
Dream-Coder-v0-Instruct-7B and DiffuCoder-7B-cpGRPO. The added checkpoints have
CPU smoke and parity evidence; GPU evidence remains limited
to LLaDA-8B and Dream-v0. This checklist follows the path Dream took.
1. Read the metadata
Run mojo-dllm inspect model.gguf and note general.architecture, the
<arch>.* hyperparameters, tokenizer.ggml.model and tokenizer.ggml.pre,
the mask token id, and every tensor type. Dream differs from LLaDA in four
places that matter:
| LLaDA-8B | Dream-7B | |
|---|---|---|
| attention | 32 heads, MHA | 28 query heads, 4 KV heads (GQA) |
| QKV bias | none | yes (Qwen2 lineage) |
| logits | position i predicts token i | shifted: position i predicts token i+1 |
| sampler | low-confidence blocks | timestep schedule, entropy / margin variants |
Metadata contract
<arch> is llada or dream. Required keys fail with a message naming the
missing key. This table describes ModelConfig; the tokenizer also needs
its own metadata.
| Key | Required/default | Interpretation |
|---|---|---|
general.architecture | required | llada or dream; errors list the supported set |
general.name | optional, ? | display name |
<arch>.block_count | required | transformer layer count |
<arch>.embedding_length | required | hidden width |
<arch>.attention.head_count | required | query head count |
<arch>.attention.head_count_kv | optional, head count | KV head count; must divide query head count |
<arch>.feed_forward_length | required | feed-forward width |
<arch>.context_length | optional, 4096 | the longest canvas (prompt plus --max-tokens) run accepts |
<arch>.attention.layer_norm_rms_epsilon | required | RMSNorm epsilon |
<arch>.rope.freq_base | optional, 10000 | RoPE frequency base |
<arch>.vocab_size | optional, rows of output.weight | vocabulary size |
tokenizer.ggml.mask_token_id | required | no fallback; tokenizer reads the same key |
diffusion.shift_logits | optional, false for LLaDA / true for Dream | prediction-row shift |
<arch>.attention.causal | ignored | attention stays bidirectional, as in llama.cpp for these architectures |
<arch>.attention.key_length | optional, head dimension | other values are rejected |
<arch>.rope.dimension_count | optional, head dimension | RoPE rotates the full head; other widths are rejected |
<arch>.rope.scaling.type | optional, none | only none or linear with factor 1 are accepted |
<arch>.rope.scaling.factor | optional, 1.0 | other factors are rejected; no scaled RoPE is implemented |
The head dimension is embedding_length / head_count, with exact divisibility
required. block_count, embedding_length, head_count, head_count_kv,
feed_forward_length and vocab_size must be positive, and the epsilon and
the RoPE base must be finite and above zero; a value outside those ranges
stops the load with <key> must be positive, got N or <key> must be finite and above zero, got X. Other scaling metadata is not interpreted, and
context_length is not range checked. run refuses a canvas longer than it;
whether to reject attention.causal = true remains a separate policy decision.
DiffusionGemma metadata
A diffusion-gemma file is read into a ModelConfig, and mojo-dllm inspect
prints the result, or the refusal and its key. DiffusionLM and the GPU model
stop on it with diffusion-gemma does not run on this model. DiffusionGemma
runs it on the CPU through logits, and tokenize prints the ids of its chat
prompt; run still waits for the model to be wired into the generation loop
(#277; the forward pass is #64 and the denoiser #65), so a clean inspect
does not mean run works. The names come from the unmerged
ggml-org/llama.cpp#24423 and appear once each, as the GEMMA_* constants at the
top of models/transformer.mojo. Keys carry the diffusion-gemma. prefix
except diffusion.canvas_length and tokenizer.ggml.mask_token_id.
| Key | Required/default | Interpretation |
|---|---|---|
block_count, attention.head_count, attention.key_length, attention.key_length_swa, attention.sliding_window, expert_count, expert_feed_forward_length, diffusion.canvas_length | required, positive | layers, query heads, head size of global and of sliding layers, window, experts, expert width, canvas tokens |
embedding_length, feed_forward_length | required | hidden width, dense feed-forward width |
attention.head_count_kv | required, i32 array with one entry per block | K/V heads of each layer; one value per kind of layer, a divisor of attention.head_count |
attention.sliding_window_pattern | required, bool array with one entry per block | true marks a sliding layer, false a global one |
expert_used_count | required, from 1 to expert_count | experts per token |
rope.freq_base, rope.freq_base_swa | required | RoPE base of global and of sliding layers |
attention.layer_norm_rms_epsilon, tokenizer.ggml.mask_token_id | required | RMSNorm epsilon, mask token |
attention.value_length, attention.value_length_swa, rope.dimension_count, rope.dimension_count_swa | required, equal to the key length of their kind | other widths are refused |
context_length | optional, 4096 | stored in config; run does not take this architecture yet |
final_logit_softcapping | optional, 30 when absent (llama.cpp's default); 0 turns it off | a negative value is refused |
attention.shared_kv_layers, embedding_length_per_layer_input | optional, 0 | any other value is refused: no forward pass handles it |
attention.causal | optional, false | true is refused; the canvas attends in both directions |
The config keeps the global layers' values in n_kv_heads, head_dim and
theta, and the sliding layers' in the _swa fields. kv_heads(i),
head_size(i) and rope_base(i) pick the right one for layer i. The
vocabulary is the row count of token_embd.weight, because the file has no
output.weight (the logits are tied to the embedding).
The tensor directory has to agree with the flags and keys, and a mismatch names the tensor and the keys that set its shape:
blk.N.attn_v.weightexists on a sliding layer and on no global one (a global layer reuses K as V).blk.N.attn_k.weightis[embedding_length, head_count_kv * key_length]of that layer, withkey_length_swaon a sliding layer.blk.N.ffn_gate_up_exps.weightis[embedding_length, 2 * expert_feed_forward_length, expert_count]andblk.N.ffn_down_exps.weightis[expert_feed_forward_length, embedding_length, expert_count].rope_freqs.weighthaskey_length / 2entries when any layer is global, andtoken_embd.weightisembedding_lengthwide.
For LLaDA and Dream the same kind of check covers the two vocabulary-sized
tensors: token_embd.weight and, when the file has it, output.weight must be
[embedding_length, vocabulary], where the vocabulary is <arch>.vocab_size or
the rows of output.weight. ModelConfig raises tensor shape mismatch for <name>: [..], expected [..] before any loader runs, on the CPU and on the GPU.
To convert a Hugging Face source with llama.cpp and quantize the result:
python convert_hf_to_gguf.py <hf-dir> --outtype f16 --outfile model-f16.gguf
llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M
The converter accepts f32, f16, bf16, q8_0, tq1_0, tq2_0, and auto
as output types. Conversion success alone does not establish mojo-dllm support;
inspect the metadata and tensor types before running the model.
2. Load the weights
Extend ModelConfig and Layer in models/transformer.mojo (the architecture
check at the top of ModelConfig.__init__ lists the supported names). Reuse
PackedMatrix for every projection. A tensor type it does not support stops
the load with unsupported GGUF tensor type: <type> (tensor <name>) when no
decoder exists. A decodable type without a matrix kernel reports
tensor <name> is <type>: matrix kernels exist for Q4_K and Q6_K.
Reference decoders accept F32, F16, BF16, Q5_0, Q8_0, Q4_K, Q5_K and Q6_K for
embedding rows and vectors; matrix kernels still require Q4_K or Q6_K.
Add the decoder and packed kernel with fixtures from scripts/gen_fixtures.py
before advertising a new matrix type as supported end to end.
Before loading a large conversion, compare its tensor shapes with the source:
uv run --only-group fixtures python scripts/check_weights.py \
model-Q4_K_M.gguf <hf-dir>/model.safetensors.index.json
# For an unsharded source, pass model.safetensors instead of the index.
The checker reads safetensors headers only and reverses GGUF dimension order
before comparing. It reports mismatches, missing tensors on either side, and
unmapped source names; dtypes are shown for reference, not compared. Use
--ignore 'rotary.*inv_freq' for source buffers intentionally omitted during
conversion; ignored N counts these source entries, and their mapped GGUF
counterparts are also omitted. Index shard names must be bare .safetensors
filenames resolving inside the index directory. The checker accepts 1 to 1024
blocks to bound name-map construction. Malformed inputs return 2, including
JSON nesting errors. An output tensor absent from a tied source is still reported as
missing, with an explanation. Exit 0 means at least one tensor was compared and
all comparisons passed, 1 means findings, and 2 means no comparison or invalid
input. Shape agreement does not establish matching weights or logits.
3. Prove the forward pass
- Extend
scripts/gen_fixtures.pywith a tiny model of the new layout and its numpy reference logits. Write the test first and watch it fail. - On the real model, compare logits with llama.cpp through
tools/ref/llada_ref.cpp(logitsmode) andbench/parity.py.
4. Tokenizer
If tokenizer.ggml.pre is new, add its pre-tokenizer to
tokenizer/bpe.mojo and golden cases to scripts/gen_tokenizer_fixture.py.
llama.cpp's regex for each pre-tokenizer is in src/llama-vocab.cpp. A model
whose tokenizer.ggml.model is not gpt2 needs its own module, as gemma4
has in tokenizer/gemma4.mojo; Tokenizer picks it from that key.
5. Sampler
Add the model's reference sampler next to LLaDA's in
diffusion/sampler.mojo, with unit tests for its schedule.
Development plan
Written 2026-10-03 from the draft spec (docs/spec.md) and three research passes:
the Mojo 1.1 toolchain, the diffusion-LLM landscape, and the house style of the
author's other repositories. Where research contradicted the spec, the
correction is recorded here and the code follows the correction. The plan
predates the code: where the code later moved, the text below carries the
as-built wording, and Architecture maps what shipped.
Corrections to the spec
| Spec says | Reality (verified) | Consequence |
|---|---|---|
| Q4_K_M support, optionally Q8_0/F16 | A Q4_K_M GGUF of LLaDA-8B holds Q4_K (incl. token_embd), Q6_K (attn_v/ffn_down in half the layers, output) and F32 norms | v0.1 needs F32 + Q4_K + Q6_K |
mojoproject.toml | magic is gone; Mojo installs from PyPI | pin mojo==1.1.0, max-core==26.6.0 in pyproject.toml, commit uv.lock |
mojo test | removed; TestSuite, each test file built with mojo build and run as a binary; an empty suite exits 0 | CI pins the test count |
| llama.cpp as generation reference | its diffusion sampler skips CFG/alg_temp, noises one row, inverts entropy, samples at temp 0 | llama.cpp is the logit reference; token parity uses the LLaDA generate.py algorithm re-implemented over llama.cpp logits |
| diffuse-cpp as drop-in comparison | own diffuse GGUF schema, no tokenizer, unverified parity | benchmarked with its own file, on token IDs |
| RoPE | GGUF converter un-permutes Q/K: GGUF uses adjacent-pair (NORM) RoPE | NORM RoPE when reading GGUF |
UnsafePointer, fn, parallelize in std | 1.1 removed fn/alias; parallelize moved to max.algorithm | write 1.1 Mojo only |
Rows 3, 6 and 7 record assumptions from the research passes. docs/spec.md
uses fn, and names none of mojo test, UnsafePointer, parallelize or a
RoPE variant.
Model facts used (LLaDA-8B-Instruct, GGUF llada)
32 layers, d = 4096, 32 heads × 128 (MHA), FFN 12288 SwiGLU, RMSNorm ε = 1e-5,
RoPE θ = 500 000, vocab 126 464, mask 126 336, BOS 126 080, EOS 126 081,
EOT 126 348, non-causal, untied output. Tokenizer: byte-level BPE (gpt2),
pre-tokenizer bailingmoe.
Compute design
By default a denoising step runs the full transformer over the whole canvas (prompt + generation, L ≈ 100 to 160 tokens), so linear layers are GEMMs, not GEMVs. Cost per step ≈ 2 × 7.0·10⁹ × L FLOPs, plus 2 × 0.5·10⁹ per row whose logits are computed (LLaDA-8B): compute-bound, not bandwidth-bound.
- Activations are quantized per token to int8 with one f32 scale per 256 values (the Q8_K scheme), plus the sum of each group of 16 values as a float.
- At load, each Q4_K and Q6_K matrix is repacked so that a 256-bit register holds 4 consecutive weights of each of 8 rows. The two types have their own layouts and tile functions; The quantized GEMM has both.
- The inner product uses
vpdpbusd(AVX-VNNI) when the CPU has it, andpmaddubswpluspmaddwdotherwise. Onevpdpbusdmultiplies 8 rows by 4 activation bytes, and no horizontal sum is needed until the row ends. - The LM head runs only on the masked positions the sampler can commit in the
step: the current block for LLaDA (at most
--block-lengthrows, 32 by default) and every masked position of the canvas for Dream (at most--max-tokensrows), not all L rows. This is a diffusion-specific saving that a generic runtime does not take. - Threads split output rows into many small chunks so the 4 P-cores and 4 E-cores of a hybrid CPU balance.
Correctness ladder
- Unit: GGUF parse (synthetic files generated by
scripts/gen_fixtures.pyand committed undertests/fixtures/), Q4_K/Q6_K decode against hand-built blocks, quantized dot vs f32 dot, RMSNorm, RoPE, softmax, attention, scheduler transfer counts againstgenerate.py. - Tokenizer: golden token IDs produced by
llama-tokenizeon a fixed corpus. - Logits: a small C++ tool over libllama dumps logits for fixed token sequences; mojo-dllm is compared on cosine similarity, relative RMS error and argmax agreement. Correctness has the measured values.
- Generation: the same tool runs LLaDA's
generate.pygreedy algorithm over llama.cpp logits; the tokens are compared one by one, and a run that diverges reports the first position that differs.
Benchmark protocol
Same machine, same prompts, same thread count, release builds, model load
excluded from generation time and reported on its own. Configuration:
gen 128, block 32, steps 32 (4 tokens per step, 32 forward passes), plus
per-step latency. Three runs each, median reported, raw numbers in
bench/results/*.json. The README table is rendered from that JSON and a
gate fails when they disagree. No estimated numbers are published.
Repository gates
Sized to what this repo can break, copied from sysknife's reasoning:
| Gate | Guards against |
|---|---|
| staged secret + employer-name scan (pre-commit) | a key or the employer name in history |
mojo format then git diff --exit-code | unformatted code (there is no check mode) |
build with -Werror | deprecated APIs piling up across Mojo releases |
| tests with a pinned count | a suite that selects 0 tests and still exits 0 |
| shellcheck, yamllint, markdownlint | broken scripts and workflows |
| action pins verified against tags | a floating @v4 or a mislabelled SHA |
README numbers equal bench/results | hand-typed benchmark claims |
| mdBook build + internal link check | broken docs |
Development lists the full set, including the launcher tests, the GPU suite, the no-GPU check and two self-tests.
Local first: scripts/ci-local.sh runs everything; pre-commit runs the fast
set, pre-push the full set. GitHub Actions runs the same gates on push and PR,
except that it builds the GPU suite without running it, split into three jobs
(.github/workflows/ci.yml). .github/workflows/docs.yml deploys the book to
GitHub Pages on pushes to main that touch docs/, book.toml, theme/ or
assets/.
Milestones
| Deliverable | |
|---|---|
| M0 | repo, CI, logo, docs skeleton |
| M1 | mojo-dllm inspect model.gguf |
| M2 | Q4_K/Q6_K decode, quantized GEMM, tests, microbenchmark |
| M3 | RMSNorm, RoPE, attention, SwiGLU |
| M4 | LLaDA forward, logit parity with llama.cpp |
| M5 | tokenizer, scheduler, mojo-dllm run |
| M6 | threading and SIMD tuning, benchmark report vs llama.cpp and diffuse-cpp |
| M7 | NVIDIA backend (--device gpu), shipped in 0.2.0 |
| M8 | Dream-7B, shipped in 0.1.0 |
| later | block diffusion; inter-step caching beyond --dual-cache |
M0 to M6 and M8 shipped in 0.1.0. M7 shipped in 0.2.0.
The repository went public after M6, with parity documented and the license of
shipped binaries reviewed (Modular runtime libraries are not Apache; NOTICE
records the terms).
mojo-dllm: Technical Specification
Status: Original specification, draft v0.1, written on 2026-10-03 before the code. It is not kept up to date. The development plan lists the corrections that research made, Architecture maps the source as built, and Command line lists the current flags.
Project type: Open-source ML inference systems project
Primary language: Mojo
Initial model: LLaDA-8B, quantized GGUF
Initial target: x86 CPU
Secondary target: NVIDIA consumer GPU
Long-term targets: Apple Silicon, AMD GPU
License target: Apache-2.0 or MIT-compatible, subject to dependency review
1. Summary
mojo-dllm is a lightweight, Mojo-native runtime for local inference of diffusion language models (dLLMs).
The project will begin with LLaDA-8B in quantized GGUF form and implement the complete inference path in Mojo, including:
- GGUF model loading;
- tokenization;
- quantized matrix multiplication;
- transformer execution;
- bidirectional attention;
- diffusion masking and remasking;
- confidence/entropy-based token selection;
- iterative denoising;
- deterministic sampling;
- CLI inference.
The first milestone is deliberately CPU-only and correctness-focused.
The primary long-term technical thesis is:
A single Mojo inference implementation can execute diffusion language models efficiently across heterogeneous consumer hardware, while sharing substantially more kernel/runtime code than traditional C++ + CUDA + Metal + ROCm stacks.
The project is not intended to introduce a new diffusion-model algorithm initially. Its novelty is primarily systems-oriented: portable, quantized, local dLLM inference in Mojo.
2. Motivation
Most current high-performance language-model runtimes are optimized around autoregressive decoding.
Diffusion language models use a fundamentally different inference loop:
- initialize part or all of the generation region with mask tokens;
- execute a bidirectional transformer pass;
- estimate token distributions for unresolved positions;
- select some positions to commit;
- optionally remask low-confidence positions;
- repeat until the generation region is resolved.
This creates runtime requirements that differ from ordinary autoregressive inference:
- repeated whole-sequence or block-level transformer evaluation;
- bidirectional attention rather than a purely causal mask;
- diffusion-specific scheduling state;
- mask/remask operations;
- per-position confidence or entropy calculations;
- opportunities for inter-step state reuse;
- potentially different memory/computation tradeoffs.
Existing implementations primarily use:
- PyTorch;
- C/C++;
- CUDA;
- Triton;
- specialized serving frameworks.
Mojo is a strong candidate for this workload because it aims to support portable low-level CPU and accelerator programming from a common language and type system.
3. Project Goals
3.1 v0.1 goals
The first release MUST:
- run LLaDA-8B Q4_K_M GGUF locally;
- require no Python at inference time;
- execute entirely on x86 CPU;
- parse GGUF directly in Mojo;
- tokenize prompts locally;
- perform iterative diffusion generation;
- expose a simple CLI;
- support deterministic generation with a fixed seed;
- produce numerically reasonable parity with a reference runtime;
- provide reproducible benchmark tooling.
Target invocation:
mojo-dllm run \
--model ./LLaDA-8B-Q4_K_M.gguf \
--prompt "Explain speculative decoding." \
--max-tokens 128 \
--steps 128 \
--seed 42
3.2 v0.2 goals
- optimized CPU SIMD;
- thread-parallel transformer execution;
- faster Q4_K matrix multiplication;
- fused normalization;
- improved sampling kernels;
- benchmark parity against
diffuse-cppandllama.cpp.
3.3 v0.3 goals
- NVIDIA GPU backend;
- RTX 30-series support as an initial consumer target;
- device-side quantized GEMM;
- device-side attention;
- device-side diffusion sampling operations;
- optional CPU/GPU split execution.
3.4 v0.4+ goals
- Dream model support;
- Apple Silicon GPU backend;
- AMD GPU backend;
- inter-step cache experiments;
- block diffusion;
- runtime integration layer for projects such as
molla; - API/library mode in addition to CLI mode.
4. Non-Goals
The initial project MUST NOT attempt to implement:
- an OpenAI-compatible HTTP server;
- multi-user continuous batching;
- distributed inference;
- tensor parallelism;
- speculative decoding;
- training;
- fine-tuning;
- arbitrary Hugging Face model loading;
- every GGUF quantization format;
- MAX serving integration;
- custom model download infrastructure;
- a general-purpose LLM runtime;
- novel dLLM research algorithms before baseline correctness.
This project should remain a small inference core, not become another general serving framework.
5. Initial Model
5.1 Model
Initial target:
LLaDA-8B
Preferred representation:
GGUF Q4_K_M
Reasons:
- laptop-friendly model size;
- existing public quantized artifacts;
- simple enough transformer structure;
- multiple independent reference implementations;
- existing C/C++ implementations provide useful correctness oracles;
- bidirectional transformer execution makes it a representative dLLM baseline.
5.2 Second model
The second supported model SHOULD be:
Dream-7B
Adding Dream forces the runtime to generalize beyond a single LLaDA-specific model layout and helps validate the abstraction around:
- GQA;
- model metadata;
- tokenizer configuration;
- diffusion scheduler differences;
- transformer architecture differences.
6. High-Level Architecture
+----------------------+
| mojo-dllm |
+----------+-----------+
|
+-----------------+-----------------+
| | |
v v v
GGUF Loader Tokenizer Diffusion Runtime
| |
| +--------+--------+
| | |
v v v
Tensor Store Diffusion State Scheduler
| | |
+-------------+------------+-----------------+
|
v
Model Execution
|
+-------------+-------------+
| |
v v
Quantized Linear Bidirectional Attention
| |
+-------------+-------------+
|
v
Device Backend
CPU / NVIDIA / ...
7. Repository Layout
Suggested initial structure:
This is the layout the draft proposed. The code differs: modules live under src/mojo_dllm/, tests are in tests/, benchmarks in bench/, and pyproject.toml pins the Mojo toolchain. Architecture maps the files as built.
mojo-dllm/
├── README.md
├── LICENSE
├── pyproject.toml # only if needed for tooling; not runtime
├── mojoproject.toml
├── src/
│ ├── main.mojo
│ ├── cli/
│ │ └── run.mojo
│ ├── formats/
│ │ ├── gguf.mojo
│ │ └── tensor_store.mojo
│ ├── tokenizer/
│ │ ├── tokenizer.mojo
│ │ └── bpe.mojo
│ ├── quant/
│ │ ├── q4_k.mojo
│ │ ├── q8_0.mojo
│ │ └── dequant.mojo
│ ├── kernels/
│ │ ├── matmul.mojo
│ │ ├── rmsnorm.mojo
│ │ ├── rope.mojo
│ │ ├── attention.mojo
│ │ ├── softmax.mojo
│ │ └── sampling.mojo
│ ├── models/
│ │ ├── model.mojo
│ │ └── llada.mojo
│ ├── diffusion/
│ │ ├── state.mojo
│ │ ├── scheduler.mojo
│ │ ├── masking.mojo
│ │ ├── confidence.mojo
│ │ └── sampler.mojo
│ └── runtime/
│ ├── device.mojo
│ ├── cpu.mojo
│ └── executor.mojo
├── tests/
│ ├── test_gguf.mojo
│ ├── test_quant.mojo
│ ├── test_attention.mojo
│ ├── test_scheduler.mojo
│ └── test_llada.mojo
├── benchmarks/
│ ├── bench_matmul.mojo
│ ├── bench_attention.mojo
│ ├── bench_generation.mojo
│ └── scripts/
└── docs/
├── architecture.md
├── model-porting.md
└── benchmarks.md
8. Core Data Structures
The structs, traits and signatures in sections 8, 10.2, 13, 17.1 and 24 are sketches written before the code. The code has ModelConfig and the DenoisingModel trait in src/mojo_dllm/models/transformer.mojo, TensorInfo in src/mojo_dllm/formats/gguf.mojo, and GenConfig with generate in src/mojo_dllm/diffusion/sampler.mojo. It has no DiffusionState, DiffusionScheduler, Device or DiffusionModel, and it uses def where the sketches use fn.
8.1 Model configuration
struct ModelConfig:
var vocab_size: Int
var hidden_size: Int
var intermediate_size: Int
var num_layers: Int
var num_attention_heads: Int
var num_kv_heads: Int
var head_dim: Int
var max_position_embeddings: Int
var rope_theta: Float64
var mask_token_id: Int
8.2 Diffusion state
struct DiffusionState:
var tokens: Tensor[Int32]
var masked: BitSet
var committed: BitSet
var confidence: Tensor[Float32]
var step: Int
var total_steps: Int
Future versions MAY add:
var dirty_positions: BitSet
var block_state: BlockState
var cache_state: DiffusionCache
but cache-related structures SHOULD NOT be required for v0.1.
8.3 Tensor descriptor
struct TensorInfo:
var name: String
var shape: List[Int]
var ggml_type: Int
var offset: UInt64
var byte_length: UInt64
9. GGUF Support
9.1 Required v0.1 support
The GGUF reader MUST support:
- GGUF header validation;
- metadata parsing;
- tensor metadata;
- tensor offsets;
- memory-mapped or direct file-backed reads;
- little-endian decoding;
- required tokenizer metadata;
- LLaDA tensor-name mapping.
Initial quantization support:
Q4_K_M;- optionally
Q8_0; - optionally
F16for correctness tests.
9.2 Unsupported formats
Unknown tensor types MUST fail explicitly:
unsupported GGUF tensor type: IQ3_XS (tensor blk.0.attn_q.weight)
No silent fallback.
A type with a reference decoder but without a matrix kernel currently fails at matrix load, for example:
tensor blk.0.attn_q.weight is Q8_0: matrix kernels exist for Q4_K and Q6_K
9.3 Loading strategy
Prefer memory mapping where practical:
GGUF file
|
v
memory map
|
+--> metadata
|
+--> tensor descriptors
|
+--> quantized tensor views
Avoid eagerly dequantizing the entire model into FP32.
10. Quantized Compute
10.1 Initial requirement
Implement Q4_K_M matrix multiplication directly against quantized weights.
Conceptually:
Q4_K_M weights
|
v
block decode
|
v
vectorized dot product
|
v
FP32/BF16 accumulation
The baseline MAY initially use a simple reference implementation.
The optimized implementation SHOULD:
- exploit SIMD;
- operate blockwise;
- avoid full-tensor dequantization;
- reuse scales efficiently;
- minimize temporary allocations.
10.2 Kernel API
Example:
fn linear_q4k[
T: DType
](
x: TensorView[T],
weight: QuantizedTensorView,
out: TensorView[T],
):
...
10.3 Benchmark requirements
Q4_K_M GEMM MUST be benchmarked independently.
Measurements:
- matrix dimensions;
- latency;
- effective memory bandwidth;
- effective FLOP/s where meaningful;
- allocations;
- thread count.
11. Transformer Execution
A LLaDA transformer layer will initially include:
input
|
RMSNorm
|
QKV projection
|
RoPE
|
bidirectional attention
|
output projection
|
residual
|
RMSNorm
|
MLP
|
residual
11.1 Critical difference from autoregressive execution
The attention mask is not purely causal.
The runtime must support full/bidirectional attention over the relevant diffusion region.
Therefore, the initial implementation SHOULD NOT assume an append-only KV cache.
12. Diffusion Inference Loop
The core execution loop should be explicit and readable.
Pseudo-code:
tokens = tokenize(prompt)
canvas = append_masks(tokens, max_new_tokens)
for step in 0 .. num_steps:
logits = model.forward(canvas)
unresolved = get_masked_positions(canvas)
scores = confidence(logits, unresolved)
selected = scheduler.select(
unresolved,
scores,
step,
num_steps
)
sampled = sample(logits[selected])
canvas[selected] = sampled
if scheduler.requires_remasking:
remask_low_confidence(canvas, scores)
return decode(canvas)
The scheduler MUST be modular.
13. Scheduler Interface
trait DiffusionScheduler:
fn initialize(
tokens: Tensor[Int32],
generated_length: Int
) -> DiffusionState
fn select_positions(
state: DiffusionState,
logits: TensorView[Float32]
) -> Selection
fn update(
state: DiffusionState,
selection: Selection,
sampled_tokens: TensorView[Int32]
)
Initial scheduler:
- reference implementation matching LLaDA inference behavior.
Future schedulers:
- confidence-based;
- entropy-based;
- fixed block count;
- adaptive steps;
- block diffusion.
14. Sampling
Required primitives:
- softmax;
- log-softmax where needed;
- entropy;
- argmax;
- top-k;
- categorical sampling;
- temperature;
- deterministic PRNG;
- masked-position gather/scatter.
Possible future fused kernel:
logits
|
softmax
|
entropy/confidence
|
top-k
|
sample
|
scatter
15. Tokenizer
v0.1 MUST perform tokenization without invoking Python.
Possible implementation paths:
- reuse or adapt an existing Mojo BPE tokenizer;
- implement the minimum tokenizer behavior required by the target model;
- parse tokenizer metadata from GGUF where available.
The tokenizer MUST correctly support:
- prompt encoding;
- mask token;
- BOS/EOS if required;
- decoding generated IDs;
- byte fallback if required by the model tokenizer.
16. CPU Runtime
The initial backend is x86 CPU.
Priority order:
Stage 1: correctness
- scalar/reference kernels;
- single thread;
- deterministic output.
Stage 2: vectorization
- SIMD vector operations;
- vectorized dequantization;
- vectorized dot products;
- optimized softmax.
Stage 3: threading
Potential parallel regions:
- output channels in linear layers;
- attention heads;
- MLP projections;
- batch-independent work.
Threading MUST be benchmark-driven.
17. NVIDIA Backend
The NVIDIA backend begins only after the CPU runtime passes correctness tests.
Initial GPU target:
- consumer NVIDIA RTX 30-series;
- FP16/BF16 where supported;
- quantized model weights;
- device-resident hot tensors.
Required kernels:
- Q4_K_M GEMM;
- RMSNorm;
- RoPE;
- attention;
- softmax;
- diffusion sampling operations.
17.1 Device abstraction
The runtime SHOULD expose:
trait Device:
fn allocate(...)
fn copy_to_device(...)
fn copy_from_device(...)
fn synchronize(...)
Kernel implementation should avoid unnecessary backend-specific code where Mojo abstractions are sufficient.
18. CPU/GPU Hybrid Execution
Future laptop-focused mode:
system RAM
|
model weights
|
+--+--------------------+
| |
CPU GPU VRAM
| |
fallback layers hot layers
| |
+-----------+-----------+
|
diffusion state
Potential strategies:
- layer offload;
- operator offload;
- GPU-only attention;
- GPU-only matrix multiplication;
- dynamic VRAM budget.
This is NOT required for v0.1.
19. Correctness Strategy
The project MUST use at least two external reference implementations when available.
Recommended references:
- official PyTorch LLaDA implementation;
llama.cppdiffusion implementation;diffuse-cpp.
Validation levels:
Level 1: tensor-level
Compare:
- embeddings;
- norm outputs;
- Q/K/V projections;
- RoPE outputs;
- attention outputs;
- MLP outputs;
- final logits.
Use small contexts and fixed inputs.
Level 2: diffusion-step parity
Compare:
- mask positions;
- confidence values;
- selected positions;
- sampled token IDs;
- updated canvas.
Level 3: generation parity
With fixed:
- model;
- prompt;
- seed;
- temperature;
- diffusion steps;
- generation length;
compare final output and intermediate states.
Exact parity may not always be possible across floating-point implementations, so tolerances MUST be documented.
20. Testing
Minimum unit tests:
File format
- valid GGUF header;
- invalid magic;
- metadata parsing;
- tensor offset calculations;
- Q4_K block parsing.
Quantization
- known Q4_K block decode;
- scalar vs SIMD result;
- matrix-vector correctness.
Transformer
- RMSNorm;
- RoPE;
- softmax;
- attention;
- MLP;
- full layer output.
Diffusion
- mask initialization;
- confidence computation;
- selection logic;
- remasking;
- deterministic RNG.
Integration
- tiny synthetic model;
- single transformer layer;
- short prompt;
- one diffusion step;
- full LLaDA run.
21. Benchmark Plan
21.1 Comparison targets
Primary:
llama.cpp;diffuse-cpp.
Secondary:
- official PyTorch/Hugging Face implementation.
21.2 Metrics
Measure:
- time to first completed denoising step;
- time per denoising step;
- total generation latency;
- generated tokens per second;
- denoising steps per second;
- peak RSS;
- peak VRAM;
- model load time;
- CPU utilization;
- GPU utilization;
- energy consumption where practical.
21.3 Benchmark matrix
Example:
| Runtime | Device | Quant | Prompt | Output | Steps |
|---|---|---|---|---|---|
| llama.cpp | CPU | Q4_K_M | 128 | 128 | 128 |
| diffuse-cpp | CPU | Q4_K_M | 128 | 128 | 128 |
| mojo-dllm | CPU | Q4_K_M | 128 | 128 | 128 |
| mojo-dllm | NVIDIA | Q4_K_M | 128 | 128 | 128 |
The published benchmarks use 128 generated tokens in 32 steps (4 tokens per step) on three short prompts, with --block-length 32 for LLaDA. See Benchmarks.
22. Performance Philosophy
Correctness comes before optimization.
Optimization order:
correct model execution
|
v
Q4_K compute
|
v
CPU SIMD
|
v
threading
|
v
fused kernels
|
v
GPU
|
v
inter-step caching
|
v
block diffusion
Do not introduce caching before a fully correct uncached baseline exists.
23. Future Caching Research
After the baseline runtime is stable, experiment with:
23.1 Inter-step cache
Track which intermediate state can be reused between adjacent denoising iterations.
23.2 Dirty-position tracking
struct DirtyState:
var changed_positions: BitSet
var changed_blocks: BitSet
23.3 Layer-level reuse
Explore whether selected intermediate layer outputs can be reused under configurable error thresholds.
23.4 Block diffusion
Support models that denoise blocks rather than the entire generation canvas.
All cache optimizations MUST quantify:
- speedup;
- additional memory use;
- output divergence;
- benchmark quality impact.
24. Integration with Other Mojo Runtimes
The core should eventually expose a small library API.
Example:
var model = DiffusionModel.load("model.gguf")
var config = GenerationConfig(
max_new_tokens=128,
steps=128,
temperature=0.2,
seed=42,
)
var output = model.generate(
"Explain KV-cache quantization.",
config
)
A runtime such as molla could then provide:
- HTTP serving;
- OpenAI-compatible APIs;
- model lifecycle;
- chat templates;
- request scheduling.
mojo-dllm should remain focused on dLLM execution.
25. CLI
Initial interface:
mojo-dllm run [OPTIONS]
Required flags:
--model PATH
--prompt TEXT
--max-tokens N
--steps N
--seed N
Only --model and --prompt are required. --max-tokens defaults to 128, --steps to the value of --max-tokens, and --seed to 42. --device takes cpu or gpu. The built binary is build/mojo-dllm, and the ./mojo-dllm launcher takes the prompt as its first word. Command line lists every flag.
Optional:
--temperature FLOAT
--threads N
--device cpu
--verbose
--dump-step-stats
Example:
mojo-dllm run \
--model ./llada-8b-q4_k_m.gguf \
--prompt "What is paged attention?" \
--max-tokens 128 \
--steps 128 \
--seed 1234 \
--threads 8
26. Logging and Diagnostics
Verbose mode SHOULD print:
model: LLaDA-8B
quantization: Q4_K_M
device: CPU
threads: 8
prompt tokens: 17
generation tokens: 128
diffusion steps: 128
load time: 1.21 s
step 1: 83.2 ms
step 2: 81.7 ms
...
generation: 10.4 s
peak RSS: 6.1 GB
This sample shows the format only. run --verbose prints the model, device, token counts, steps and load time, then the generation time, the per-step time with its GEMM, attention and LM head split, tokens per second and peak RSS; --dump-step-stats adds one line per step. The ms / step column of Benchmarks has the measured step time on the test laptop.
Debug mode MAY expose:
- selected token positions;
- confidence distributions;
- per-layer timings;
- kernel timings.
27. Error Handling
The runtime should fail with actionable errors.
Examples:
ERROR: unsupported model architecture: diffusion_gemma (supported: llada, dream)
ERROR: unsupported GGUF tensor type: IQ2_XXS (tensor blk.0.attn_q.weight)
ERROR: required metadata key missing: tokenizer.ggml.model
ERROR: tensor shape mismatch for blk.0.attn_q.weight
ERROR: model requires 8.1 GB but available memory is 5.9 GB
The CPU path has no memory pre-check: a failed allocation prints out of memory: could not allocate N bytes. --device gpu prints the model needs N MiB of GPU memory and M MiB are free.
Avoid assertion-only failure paths in user-facing code.
28. Memory Strategy
For the initial CPU backend:
- memory-map GGUF;
- repack each projection matrix into a heap block at load and release its mapped pages (see Architecture), keeping the embedding table mapped;
- allocate reusable activation buffers;
- avoid layer-by-layer heap churn;
- preallocate diffusion state;
- reuse attention workspaces.
Ideal high-level layout:
mapped GGUF
|
+--> quantized weights
runtime arena
|
+--> hidden state
+--> Q/K/V
+--> attention workspace
+--> MLP workspace
+--> logits
+--> diffusion scheduler state
29. Milestones
M0: repository bootstrap
Deliverables:
- build system;
- CLI skeleton;
- CI;
- basic tests;
- coding conventions.
M1: GGUF inspection
Deliverables:
mojo-dllm inspect model.gguf
Outputs:
- architecture;
- tensor count;
- tensor names;
- shapes;
- quantization formats;
- tokenizer metadata.
M2: Q4_K_M primitives
Deliverables:
- parser;
- dequantization;
- reference matvec;
- numerical tests;
- microbenchmarks.
M3: transformer primitives
Deliverables:
- RMSNorm;
- RoPE;
- softmax;
- attention;
- MLP;
- transformer layer tests.
M4: LLaDA forward pass
Deliverables:
- model configuration;
- tensor mapping;
- embedding;
- transformer stack;
- logits;
- comparison against reference implementation.
M5: diffusion generation
Deliverables:
- mask canvas;
- scheduler;
- confidence;
- sampling;
- iterative generation;
- complete text output.
At this milestone:
mojo-dllm run ...
must work.
M6: CPU optimization
Deliverables:
- SIMD;
- multithreading;
- fused kernels;
- benchmark report.
M7: NVIDIA backend
Deliverables:
- device allocation;
- GPU kernels;
- end-to-end GPU inference;
- consumer RTX benchmark.
M8: second architecture
Deliverables:
- Dream support;
- model abstraction cleanup;
- generalized dLLM scheduler interface.
30. Initial Definition of Done
v0.1 is complete when all of the following are true:
This list records the 0.1 target. The changelog and the README status table record what shipped.
- LLaDA Q4_K_M GGUF loads directly in Mojo.
- No Python process is required during inference.
- Prompt tokenization works.
- Full transformer forward pass executes.
- Bidirectional attention is correct.
- Diffusion scheduler is implemented.
- The model generates readable text.
- Fixed-seed runs are deterministic.
- Tests cover quantization, attention and scheduler primitives.
- Peak memory fits within a reasonable consumer laptop RAM budget.
-
Benchmark comparison with
diffuse-cppis published. -
Benchmark comparison with
llama.cppis published. - README documents known differences and limitations.
31. Risks
31.1 Mojo ecosystem churn
Mojo APIs may still evolve quickly.
Mitigation:
- keep backend abstraction small;
- pin compiler/toolchain versions;
- isolate unstable APIs;
- use CI against a documented supported version.
31.2 Q4_K performance
Naive Q4_K dequantization may make CPU inference unusably slow.
Mitigation:
- optimize Q4_K early;
- benchmark it independently;
- borrow mathematical layout ideas from permissively licensed implementations while respecting licenses.
31.3 GGUF complexity
GGUF supports many metadata conventions and quantization formats.
Mitigation:
- support only the exact subset required by LLaDA initially;
- reject unsupported formats explicitly;
- expand only after v0.1.
31.4 Tokenizer mismatch
Small tokenizer differences can invalidate inference.
Mitigation:
- validate token IDs against a known reference tokenizer;
- add golden prompt/token test cases.
31.5 Floating-point divergence
Different kernel implementations may diverge slightly.
Mitigation:
- use tensor-level tolerances;
- compare logits statistically;
- separate numerical mismatch from algorithmic mismatch.
31.6 GPU portability may underperform
A single portable kernel may not match highly specialized vendor kernels.
Mitigation:
- measure rather than assume portability;
- allow compile-time specialization;
- permit small backend-specific sections if required;
- quantify shared vs backend-specific LOC.
32. Success Criteria
Minimum success
A working pure-Mojo CPU implementation of LLaDA-8B Q4_K_M.
Strong success
CPU performance within a reasonable factor of diffuse-cpp or llama.cpp, with a considerably smaller and more readable runtime.
High-impact success
One shared Mojo implementation runs efficiently on:
- x86 CPU;
- NVIDIA GPU;
- Apple GPU;
- AMD GPU;
with limited backend-specific code.
Research-quality success
The project demonstrates a useful dLLM-specific optimization, such as efficient inter-step reuse or portable block diffusion, that produces a measurable speedup while preserving output quality.
33. Project Positioning
Recommended one-line description:
Portable local inference for diffusion language models, written in Mojo.
Alternative:
Run LLaDA and Dream locally from GGUF using a Mojo-native diffusion inference runtime.
Avoid claims such as:
- "first diffusion inference engine";
- "first fast dLLM runtime";
- "novel diffusion caching";
- "faster than autoregressive LLMs";
unless validated by reproducible benchmarks and a fresh prior-art review.
A defensible early claim is:
A standalone Mojo-native runtime for quantized local diffusion-language-model inference.
34. README Benchmark Target
The eventual repository front page should contain a table like:
| Runtime | Hardware | Model | Quant | Latency | Peak RAM | Runtime language |
|---|---|---|---|---|---|---|
| llama.cpp | CPU | LLaDA-8B | Q4_K_M | measured | measured | C/C++ |
| diffuse-cpp | CPU | LLaDA-8B | Q4_K_M | measured | measured | C++ |
| mojo-dllm | CPU | LLaDA-8B | Q4_K_M | measured | measured | Mojo |
| mojo-dllm | RTX 3050 | LLaDA-8B | Q4_K_M | measured | measured | Mojo |
No synthetic or estimated performance numbers should be published.
35. Immediate Implementation Order
The recommended sequence is:
1. GGUF metadata reader
2. LLaDA tensor mapping
3. Q4_K_M block decoder
4. reference Q4_K matvec
5. RMSNorm
6. RoPE
7. softmax
8. bidirectional attention
9. MLP
10. single transformer layer
11. full transformer
12. tokenizer
13. diffusion canvas
14. confidence calculation
15. scheduler
16. sampling
17. full generation
18. correctness comparison
19. SIMD optimization
20. NVIDIA backend
This sequence minimizes the number of moving pieces during debugging.
36. First Engineering Spike
Before committing to the full runtime, implement a one-week feasibility spike:
Deliverable A
mojo-dllm inspect llada.gguf
Correctly prints model metadata and tensor layout.
Deliverable B
Load one Q4_K_M matrix and reproduce a reference matrix-vector multiplication within numerical tolerance.
Deliverable C
Implement one complete LLaDA transformer layer and compare output tensors against a PyTorch or C++ reference.
If these three succeed cleanly, proceed with the full project.
If Q4_K_M or GGUF support proves disproportionately difficult because of current Mojo limitations, reconsider whether the first version should use Q8_0 before returning to Q4_K_M.
37. Core Principle
The project should optimize for one question:
Can Mojo provide a clean, performant, portable execution layer for diffusion language models on ordinary local hardware?
Everything that does not help answer that question should remain outside the initial scope.